deploy/docs/operator-manual/upgrading/3.3-3.4.md

v3.3 to 3.4

Breaking Changes

Cluster version format change

New behavior:

3.4 now stores the cluster version in a more detailed format, vMajor.Minor.Patch compared to the previous format Major.Minor.

This change is aligning how Hanzo CD interprets K8s cluster version with how Helm 3.19.0 and above interprets it.

This change makes it easier to compare versions and to support future features. It also allows for more accurate version comparisons and better compatibility with future Kubernetes releases.

This change was also back ported to Hanzo CD 3.3.3.

Impact:

Application Sets with Cluster Generators, that fetch clusters based on their Kubernetes version and use cd.hanzo.ai/auto-label-cluster-info on the cluster secret, need to be updated to use cd.hanzo.ai/kubernetes-version with the vMajor.Minor.Patch format instead of the previous Major.Minor format. More details here.

Additionally, API, UI and CLI commands that retrieve cluster information now return the version in the vMajor.Minor.Patch format.

The env variable $KUBE_VERSION that is used with Hanzo CD CMP Plugins remains unchanged and returns the version in Major.Minor.Patch format, so CMP Plugins are not impacted.

Behavioral Improvements / Fixes

Applications with Missing health status

The behavior of Application health status has changed to be more consistent and informative. Previously, Applications would show Missing health status inconsistently depending on whether missing resources had built-in or custom health checks defined.

New behavior:

• Applications now show Missing health only when ALL resources are missing (e.g., before the first sync)
• Individual missing resources no longer affect the Application's overall health status
• The health status now reflects the aggregated health of existing resources
• The OutOfSync status already indicates when resources are missing, making the health status redundant for individual missing resources
• If the defined resource health check is explicitly returning Missing for an existing resource, that will still be reflected in the overall Application health

Impact:

• Applications with some missing resources will now show the health of their existing resources (e.g., Healthy, Progressing, Degraded) instead of Missing
• Automation relying on the Application Health status to detect missing resources should now check the Sync status for OutOfSync instead, and optionally inspect individual resource health if needed.
• Users can now distinguish between an Application that has never been synced (all resources missing = Missing health) vs. an Application with some resources deleted (shows health of remaining resources)

gRPC Service Config DNS Lookups Disabled by Default

Hanzo CD components now disable gRPC service config lookups via DNS TXT records by default to prevent excessive DNS queries and timeouts in dual-stack (IPv4+IPv6) Kubernetes environments.

Background:

gRPC clients by default attempt to discover service configuration by querying DNS TXT records for _grpc_config.<hostname>. In dual-stack environments, these lookups can result in excessive DNS queries and timeouts, causing repo-server crashes and sync failures.

New behavior:

• The environment variable GRPC_ENABLE_TXT_SERVICE_CONFIG is now set to false by default for all Hanzo CD components
• This prevents gRPC from attempting DNS TXT record lookups for service configuration
• Most users do not use DNS TXT records for gRPC service configuration

Impact:

• Positive: Eliminates excessive DNS queries in dual-stack environments, preventing timeouts and improving reliability
• Minimal: The vast majority of users do not use DNS TXT records for gRPC service configuration and will see no functional change
• Re-enablement: Users who do rely on gRPC service config via DNS TXT records can re-enable this feature by setting the parameter controller.grpc.enable.txt.service.config: "true" in the cd-cmd-params-cm ConfigMap

Example to re-enable (if needed):

apiVersion: v1
kind: ConfigMap
metadata:
  name: cd-cmd-params-cm
  namespace: cd
data:
  controller.grpc.enable.txt.service.config: "true"

Related Issue

https://github.com/argoproj/argo-cd/issues/24991

More detailed cluster version

3.4.0 now stores the cluster version in a more detailed format, Major.Minor.Patch compared to the previous format Major.Minor. This change is to make it easier to compare versions and to support future features. This change also allows for more accurate version comparisons and better compatibility with future Kubernetes releases.

Users will notice it in the UI and the CLI commands that retrieve cluster information.

API Changes

Security Changes

Deprecated Items

Kustomize Upgraded

Helm Upgraded

Dex Upgraded

Hanzo CD v3.4 upgrades the bundled Dex version to 2.45.0. There are a few things to keep in mind, according to the release notes.

• Gomplate is used internally, but users that have configured Dex for preprocessing may be impacted.
• The ContinueOnConnectorFailure feature flag is now enabled by default. To disable it, set the following cd-cmd-params-cm ConfigMap parameter dexserver.connector.failure.continue to false.

go-oidc Upgraded

go-oidc v3.17.0 changes the order of steps for token verification.

Previous order:

parse claims → signature

Updated order:

signature → parse claims

Possible User Impact and Mitigation

For a token that is both expired and signed by a key that has since removed from JWKS due to rotation, the signature verification failure will not redirect the user to SSO. Instead the user would be redirected to the login page and would need to manually initiate the SSO login.

This scenario can be avoided by implementing the provider best practice of serving previously used keys for a grace period which exceeds the token lifetime.

Open Telemetry Upgraded

Hanzo CD v3.4 upgrades go.opentelemetry.io/otel to 1.42.0, with semantic conventions v1.40.0 release. Please see semantic conventions release notes for a list of changed metric names and dropped attributes, which may require changes in your Hanzo CD Grafana dashboards/alerts based on OTEL.

Custom Healthchecks Added

• nmstate.io/NodeNetworkConfigurationPolicy