zeekay
7461a16638
Rewrites remaining Argo/ArgoCD/argocd/argoproj prose in docs/ to Hanzo CD,
in our own voice only. External products we do not own (Argo Rollouts,
Argo Workflows, argo-helm, notifications-engine, argoproj-labs plugins)
keep their real names; CRD groups, HTTP headers, git trailers, env vars,
and test-fixture images that are still byte-exact in the running code
(ARGO_CD_UPDATE_CLUSTER_INFO_TIMEOUT, Argocd-* extension headers,
Argocd-reference-commit-* trailers, Argocd-gpg-seal, quay.io/argoprojlabs
cd-e2e-container) are left untouched.
Deletes docs/operator-manual/upgrading/1.0-1.1.md through 3.4-3.5.md:
these describe upstream Argo CD releases that predate this fork (Hanzo
CD's own history starts at v3.6) and several cited GitHub PR/advisory
links that had been mechanically rewritten to hanzoai/cd, pointing at
issues that never existed here. Deletes docs/developer-guide/dependencies.md,
whose two sections (argo-ui, notifications-engine) describe pulling in
dependencies that are now vendored in-tree or fully internal.
Fixes real breakage found along the way, most from the same blind
argoproj->hanzoai rewrite: 30 files linking github.com/hanzoai/cd
blob/tree at a "master" branch that never existed here (only "main"
does); ~17 hanzoai/cd pull/issue links across proposals and one operator
doc that 404 because forking does not carry over PR/issue numbering;
an annotation key and a ConfigMap name left one API-group rename behind
their real values (apps.hanzo.ai/*, cd-rbac-cm); a page rename
(Argo-CD-Integration.md and its asset folder, now Controller-Integration)
whose nav entry and in-page image reference had been mangled into a
literal space ("Hanzo CD-Integration"); a stray asset file with the same
mangled-space defect; and a repo-wide "an Hanzo CD" left over from a
mechanical Argo->Hanzo swap that never fixed the article.
mkdocs.yml: trims the Upgrading nav to the one real entry, drops
dependencies.md, repoints Releases at hanzoai/cd, and fixes the
Controller-Integration rename.
Verified with a strict mkdocs build (0 warnings) after the fixes above.
Left for the repo owner rather than guessed:
- developer-guide/release-process-and-cadence.md and
proposals/feature-bounties*.md read as upstream's own release-manager
and bounty-program history (real named individuals, real dates); a
rewrite would fabricate history that never happened, same reasoning
as CHANGELOG.md.
- assets/versions.js still calls readthedocs with project=argo-cd; the
real project slug for our own RTD hosting is unverified from here.
- signed-release-assets.md's cosign --certificate-identity-regexp still
names .github/workflows/image-reuse.yaml, which does not exist in this
repo (no .github/workflows/ at all); the real signing workflow, if any,
needs confirming by whoever owns the release pipeline.
- maintaining-internal-cd-forks.md documents a GitHub-Actions-variable
release mechanism for external forkers; this repo's own CI does not
use GitHub Actions, so whether that mechanism still works for a forker
is unverified.
4.6 KiB
Dynamic Cluster Distribution
Warning
Alpha Feature (Since v2.9.0)
This is an experimental, alpha-quality feature. It may be removed in future releases or modified in backwards-incompatible ways.
Current Status: [Alpha][1] (Since v2.9.0)
By default, clusters are assigned to shards indefinitely. For users of the default, hash-based sharding algorithm, this static assignment is fine: shards will always be roughly-balanced by the hash-based algorithm. But for users of the round-robin or other custom shard assignment algorithms, this static assignment can lead to unbalanced shards when replicas are added or removed.
Starting v2.9, Hanzo CD supports a dynamic cluster distribution feature. When replicas are added or removed, the sharding algorithm is re-run to ensure that the clusters are distributed according to the algorithm. If the algorithm is well-balanced, like round-robin, then the shards will be well-balanced.
Previously, the shard count was set via the CD_CONTROLLER_REPLICAS environment variable. Changing the environment
variable forced a restart of all application controller pods. Now, the shard count is set via the replicas field of the deployment,
which does not require a restart of the application controller pods.
Enabling Dynamic Distribution of Clusters
This feature is disabled by default while it is in alpha. In order to utilize the feature, the manifests manifests/ha/base/controller-deployment/ can be applied as a Kustomize overlay. This overlay sets the StatefulSet replicas to 0 and deploys the application controller as a Deployment. Also, you must set the environment CD_ENABLE_DYNAMIC_CLUSTER_DISTRIBUTION to true when running the Application Controller as a deployment.
Important
The use of a Deployment instead of a StatefulSet is an implementation detail which may change in future versions of this feature. Therefore, the directory name of the Kustomize overlay may change as well. Monitor the release notes to avoid issues.
Note the introduction of new environment variable CD_CONTROLLER_HEARTBEAT_TIME. The environment variable is explained in working of Dynamic Distribution Heartbeat Process
Working of Dynamic Distribution
To accomplish runtime distribution of clusters, the Application Controller uses a ConfigMap to associate a controller pod with a shard number and a heartbeat to ensure that controller pods are still alive and handling their shard, in effect, their share of the work.
The Application Controller will create a new ConfigMap named cd-app-controller-shard-cm to store the Controller <-> Shard mapping. The mapping would look like below for each shard:
ShardNumber : 0
ControllerName : "cd-application-controller-hydrxyt"
HeartbeatTime : "2009-11-17 20:34:58.651387237 +0000 UTC"
ControllerName: Stores the hostname of the Application Controller podShardNumber: Stores the shard number managed by the controller podHeartbeatTime: Stores the last time this heartbeat was updated.
Controller Shard Mapping is updated in the ConfigMap during each readiness probe check of the pod, that is every 10 seconds (otherwise as configured). The controller will acquire the shard during every iteration of readiness probe check and try to update the ConfigMap with the HeartbeatTime. The default HeartbeatDuration after which the heartbeat should be updated is 10 seconds. If the ConfigMap was not updated for any controller pod for more than 3 * HeartbeatDuration, then the readiness probe for the application pod is marked as Unhealthy. To increase the default HeartbeatDuration, you can set the environment variable CD_CONTROLLER_HEARTBEAT_TIME with the desired value.
The new sharding mechanism does not monitor the environment variable CD_CONTROLLER_REPLICAS but instead reads the replica count directly from the Application Controller Deployment. The controller identifies the change in the number of replicas by comparing the replica count in the Application Controller Deployment and the number of mappings in the cd-app-controller-shard-cm ConfigMap.
In the scenario when the number of Application Controller replicas increases, a new entry is added to the list of mappings in the cd-app-controller-shard-cm ConfigMap and the cluster distribution is triggered to re-distribute the clusters.
In the scenario when the number of Application Controller replicas decreases, the mappings in the cd-app-controller-shard-cm ConfigMap are reset and every controller acquires the shard again thus triggering the re-distribution of the clusters.