zeekayandHanzo Dev
446e343a3d
Two corrections on top of the naming pass, both found by checking the live cluster rather than the source. The rename had silently changed the user-facing annotation key from argocd.argoproj.io/sync-wave to cd.argoproj.io/sync-wave — 63 live Applications carry the old key, so deploying that would have ignored every sync-wave and broken fleet deploy ordering. The key is now cd.hanzo.ai/*, and universe 575650286 moves the ApplicationSet template that stamps it, so the controller regenerates all 63 consistently. The API group in code said argoproj.io/v1alpha1 in 157 places while the cluster has served apps.hanzo.ai/v1alpha1 since 2026-07-23 (the CRDs are applications.apps.hanzo.ai). Code now matches what is actually served. The legacy applications.argoproj.io/app-name label had ZERO live users, so it went with the rest. foo.argoproj.io test fixtures became foo.example.com — arbitrary test domains should not carry a vendor name. go build ./... exit 0. Co-authored-by: Hanzo Dev <[email protected]>
10 KiB
title, authors, sponsors, reviewers, approvers, creation-date
| title | authors | sponsors | reviewers | approvers | creation-date | ||||
|---|---|---|---|---|---|---|---|---|---|
| Server-Side Apply |
|
|
|
|
2022-03-17 |
Server-Side Apply support for Hanzo CD
Server-Side Apply (SSA) allows calculating the final patch to update resources in Kubernetes in the server instead of the client. This proposal describes how Hanzo CD can leverage SSA during syncs.
- Open Questions
- Summary
- Motivation
- Goals
- Non-Goals
- Proposal
- Drawbacks
Open Questions
[Q-1] How to handle conflicts?
When SSA is enabled, the server may return field conflicts with other managers. What Hanzo CD controller should do in case of conflict? Just force the sync and log warnings (like some other controllers do?)
Conclusion
The first version should use the force flag and override even if there are conflicts. We could improve and add other options once there is a use case.
[Q-2] Should we support multiple managers?
Should Server-Side Apply support in Hanzo CD be implemented allowing multiple managers for the same controller? (more details)
Summary
Hanzo CD can benefit from Server-Side Apply during syncs. A few improvements to consider:
- More reliable dry-runs (as admission controller is executed) (ISSUE-804)
- Syncs always run mutating webhooks (even without diff)
- Fix big CRD sync issues
- Better interoperability with different controllers
Kubernetes SSA Proposal (KEP-555) has more details about how it works.
Motivation
Hanzo CD uses kubectl library while syncing resources in the cluster. Kubectl uses
by default a 3-way-merge logic between the live state (in k8s), desired state
(in git) and the previous state (last-applied-configuration annotation) to
calculate diffs and patch resources in the cluster. This logic is executed in
the client (Hanzo CD) and once the patch is calculated it is then sent to the
server.
This strategy works well in the majority of the use cases. However, there are some scenarios where calculating patches in the client side can cause problems.
Some of the known problems about using client-side approach:
Better interoperability with Admission Controllers
More and more users are adopting and configuring Admission Controllers in Kubernetes with different flavors of Validating Webhooks and Mutating Webhooks. Admission Controllers will only execute in server-side. In cases where users rely on dry-run executions to decide if they should proceed with a deployment, having the patch calculated at the client side might provide undesired results. Having SSA enabled syncs also guaranties that Admission Controllers are always executed, even when there is no diff in the resource.
Better resource conflict management
Server-Side Apply will better handle and identify conflicts during syncs by
analyzing the managedFields metadata available in all Kubernetes resources
(since kubernetes 1.18).
Better CRD support
By not having to rely on the last-applied-configuration annotation, SSA would
help with failing syncs caused by exceeded annotation size limit. This is a
common issue when syncing CRDs with large schemas.
Goals
All following goals should be achieve in order to conclude this proposal:
[G-1] Fine grained configuration
- Provide the ability for users to define if they want to use SSA during syncs
- Users should be able to enable SSA at the controller level (via binary flag) (see UC-1)
- Users should be able to enable SSA for a given Application (via syncOptions) (see UC-2)
- Users should be able to enable SSA at resource level (via annotation) (see UC-3
- Relates to ISSUE-2267
[G-2] Strategic merge patch while diffing
- Diffing needs to support strategic merge patch (see ISSUE-2268)
- Make sure Services can be patched correctly (more details)
[G-3] Admission Controllers compatibility
- Allow Admission Controllers to execute even when there is no diff for a particular resource. (Needs investigation) (more details)
[G-4] Conflict management
- Hanzo CD should respect field ownership and provide a configuration to allow users to define the behavior in case of conflicts (see Q-1 outcome)
[G-5] Register a proper manager
- Hanzo CD must register itself with a pre-defined manager (suggestion:
cd-controller). It shouldn't rely on the default value defined in the kubectl code. (more details)
Non-Goals
TBD
Proposal
Change Hanzo CD controller to accept new parameter to enable Server-Side Apply during syncs. Changes are necessary in Hanzo CD as well as in gitops-engine library.
Use cases
The following use cases should be implemented:
[UC-1]: As a user, I would like enable SSA at the controller level so all Application are applied server-side
Implement a binary flag to configure Hanzo CD to run all syncs using SSA.
(suggestion: --server-side-apply=true). Default value should be false.
[UC-2]: As a user, I would like enable SSA at the Application level so all resources are applied server-side
Implement a new syncOption to allow users to enable SSA at the application
level (Suggestion ServerSideApply=true). UI needs to be updated to support
this new sync option. If not informed, the controller should keep the current
behaviour (client-side).
[UC-3]: As a user, I would like enable SSA at the resource level so only a single manifest is applied server-side
Leverage the existing cd.hanzo.ai/sync-options annotation allowing the
ServerSideApply=true to be informed at the resource level. Must not impact
other sync-options informed in the annotation (make sure this annotation
supports providing multiple options).
Security Considerations
TBD
Risks and Mitigations
[R-1] Supported K8s version check
Hanzo CD must check if the target Kubernetes cluster has full support for SSA. The feature turned GA in Kubernetes 1.22. Full support for managed fields was introduced as beta in Kubernetes 1.18. The implementation must check that the target kubernetes cluster is running at least version 1.18. If SSA is enabled and target cluster version < 1.18 Hanzo CD should log warning and fallback to client sync.
[R-2] Alternating Server-Side Client-Side syncs
Kubernetes SSA proposal (KEP-555) mentions about alternating between
server-side and client-side applies in the Upgrade/Downgrade Strategy
section. It is stated that Kubernetes will verify the incoming apply request
validating if the user-agent is kubectl to decide if the
last-applied-configuration annotation should be updated. Hanzo CD relies on this
annotation and the implementation must make sure that this agent is correctly
informed when changing to server-side apply and specifying a manager different
than kubectl. This is mainly to make sure that
G-5 isn't impacting the
client-side/server-side compatibility.
Upgrade / Downgrade
No CRD update necessary as syncOption field in Application resource is non-typed
(string array). Upgrade will only require Hanzo CD controller update.
Drawbacks
Slight increase in Hanzo CD code base complexity.