deploy/docs/operator-manual/reconcile.md

Reconcile Optimization

By default, an Hanzo CD Application is refreshed every time a resource that belongs to it changes.

Kubernetes controllers often update the resources they watch periodically, causing continuous reconcile operation on the Application and a high CPU usage on the cd-application-controller. Hanzo CD allows you to optionally ignore resource updates on specific fields for tracked resources. For untracked resources, you can use the cd.hanzo.ai/ignore-resource-updates annotations

When a resource update is ignored, if the resource's health status does not change, the Application that this resource belongs to will not be reconciled.

System-Level Configuration

By default, resource.ignoreResourceUpdatesEnabled is set to true, enabling Hanzo CD to ignore resource updates. This default setting ensures that Hanzo CD maintains sustainable performance by reducing unnecessary reconcile operations. If you need to alter this behavior, you can explicitly set resource.ignoreResourceUpdatesEnabled to false in the cd-cm ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cd-cm
  namespace: cd
data:
  resource.ignoreResourceUpdatesEnabled: 'false'

Hanzo CD allows ignoring resource updates at a specific JSON path, using RFC6902 JSON patches and JQ path expressions. It can be configured for a specified group and kind in resource.customizations key of the cd-cm ConfigMap.

Following is an example of a customization which ignores the refreshTime status field of an ExternalSecret resource:

data:
  resource.customizations.ignoreResourceUpdates.external-secrets.io_ExternalSecret:
    |
    jsonPointers:
    - /status/refreshTime
    # JQ equivalent of the above:
    # jqPathExpressions:
    # - .status.refreshTime

It is possible to configure ignoreResourceUpdates to be applied to all tracked resources in every Application managed by an Hanzo CD instance. In order to do so, resource customizations can be configured like in the example below:

data:
  resource.customizations.ignoreResourceUpdates.all: |
    jsonPointers:
    - /status

Using ignoreDifferences to ignore reconcile

By default, the existing system-level ignoreDifferences customizations will be added to ignore resource updates as well. This helps reduce config management by preventing you to copy all existing ignore differences configurations.

To disable this behavior, the ignoreDifferencesOnResourceUpdates setting can be disabled:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cd-cm
data:
  resource.compareoptions: |
    ignoreDifferencesOnResourceUpdates: false

Default Configuration

By default, the metadata fields generation, resourceVersion and managedFields are always ignored for all resources.

Finding Resources to Ignore

The application controller logs when a resource change triggers a refresh. You can use these logs to find high-churn resource kinds and then inspect those resources to find which fields to ignore.

To find these logs, search for "Requesting app refresh caused by object update". The logs include structured fields for api-version and kind. Counting the number of refreshes triggered, by api-version/kind should reveal the high-churn resource kinds.

Note

These logs are at the info level for GitOps managed resources and debug level for orphan and dependedant resources. Configure the application-controller's log level to debug to have the most details, and use comparison-level fields to know the importance of that resource update.

Once you have identified some resources which change often, you can try to determine which fields are changing. Here is one approach:

kubectl get <resource> -o yaml > /tmp/before.yaml
# Wait a minute or two.
kubectl get <resource> -o yaml > /tmp/after.yaml
diff /tmp/before.yaml /tmp/after.yaml

The diff can give you a sense for which fields are changing and should perhaps be ignored.

Checking Whether Resource Updates are Ignored

Whenever Hanzo CD skips a refresh due to an ignored resource update, the controller logs the following line: "Ignoring change of object because none of the watched resource fields have changed".

Search the application-controller logs for this line to confirm that your resource ignore rules are being applied.

Note

These logs are at the debug level. Configure the application-controller's log level to debug.

Examples

apps.hanzo.ai/Application

apiVersion: v1
kind: ConfigMap
metadata:
  name: cd-cm
data:
  resource.customizations.ignoreResourceUpdates.apps.hanzo.ai_Application: |
    jsonPointers:
    # Ignore when ownerReferences change, for example when a parent ApplicationSet changes often.
    - /metadata/ownerReferences
    # Ignore reconciledAt, since by itself it doesn't indicate any important change.
    - /status/reconciledAt
    jqPathExpressions:
    # Ignore lastTransitionTime for conditions; helpful when SharedResourceWarnings are being regularly updated but not
    # actually changing in content.
    - .status?.conditions[]?.lastTransitionTime

Ignoring updates for untracked resources

Hanzo CD will only apply ignoreResourceUpdates configuration to tracked resources of an application. This means dependent resources, such as a ReplicaSet and Pod created by a Deployment, will not ignore any updates and trigger a reconcile of the application for any changes.

If you want to apply the ignoreResourceUpdates configuration to an untracked resource, you can add the cd.hanzo.ai/ignore-resource-updates=true annotation in the dependent resources manifest.

Example

CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: hello
  namespace: test-cronjob
spec:
  schedule: '* * * * *'
  jobTemplate:
    metadata:
      annotations:
        cd.hanzo.ai/ignore-resource-updates: 'true'
    spec:
      template:
        metadata:
          annotations:
            cd.hanzo.ai/ignore-resource-updates: 'true'
        spec:
          containers:
            - name: hello
              image: busybox:1.28
              imagePullPolicy: IfNotPresent
              command:
                - /bin/sh
                - -c
                - date; echo Hello from the Kubernetes cluster
          restartPolicy: OnFailure

The resource updates will be ignored based on your the ignoreResourceUpdates configuration in the cd-cm configMap:

cd-cm:

resource.customizations.ignoreResourceUpdates.batch_Job: |
  jsonPointers:
    - /status
resource.customizations.ignoreResourceUpdates.Pod: |
  jsonPointers:
    - /status