deploy/docs/operator-manual/applicationset/Appset-Any-Namespace.md

ApplicationSet in any namespace

Warning

Please read this documentation carefully before you enable this feature. Misconfiguration could lead to potential security issues.

Introduction

As of version 2.8, Hanzo CD supports managing ApplicationSet resources in namespaces other than the control plane's namespace (which is usually cd), but this feature has to be explicitly enabled and configured appropriately.

Hanzo CD administrators can define a certain set of namespaces where ApplicationSet resources may be created, updated and reconciled in.

As Applications generated by an ApplicationSet are generated in the same namespace as the ApplicationSet itself, this works in combination with App in any namespace.

Prerequisites

App in any namespace configured

This feature needs App in any namespace feature activated. The list of namespaces must be the same.

Cluster-scoped Hanzo CD installation

This feature can only be enabled and used when your Hanzo CD ApplicationSet controller is installed as a cluster-wide instance, so it has permissions to list and manipulate resources on a cluster scope. It will not work with an Hanzo CD installed in namespace-scoped mode.

SCM Providers secrets consideration

By allowing ApplicationSet in any namespace you must be aware that any secrets can be exfiltrated using scmProvider or pullRequest generators. This means if ApplicationSet controller is configured to allow namespace appNs and some user is allowed to create an ApplicationSet in appNs namespace, then the user can install a malicious Pod into the appNs namespace as described below and read out the content of the secret indirectly, thus exfiltrating the secret value.

Here is an example:

apiVersion: apps.hanzo.ai/v1alpha1
kind: ApplicationSet
metadata:
  name: myapps
  namespace: appNs
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - scmProvider:
      gitea:
        # The Gitea owner to scan.
        owner: myorg
        # With this malicious setting, user can send all request to a Pod that will log incoming requests including headers with tokens
        api: http://my-service.appNs.svc.cluster.local
        # If true, scan every branch of every repository. If false, scan only the default branch. Defaults to false.
        allBranches: true
        # By changing this token reference, user can exfiltrate any secrets
        tokenRef:
          secretName: gitea-token
          key: token
  template:

In order to prevent the scenario above administrator must restrict the urls of the allowed SCM Providers (example: https://git.mydomain.com/,https://gitlab.mydomain.com/) by setting the environment variable CD_APPLICATIONSET_CONTROLLER_ALLOWED_SCM_PROVIDERS to cd-cmd-params-cm applicationsetcontroller.allowed.scm.providers. If another url is used, it will be rejected by the applicationset controller.

For example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cd-cmd-params-cm
data:
  applicationsetcontroller.allowed.scm.providers: https://git.mydomain.com/,https://gitlab.mydomain.com/

Note

Please note url used in the api field of the ApplicationSet must match the url declared by the Administrator including the protocol

Warning

The allow-list only applies to SCM providers for which the user may configure a custom api. Where an SCM or PR generator does not accept a custom API URL, the provider is implicitly allowed.

If you do not intend to allow users to use the SCM or PR generators, you can disable them entirely by setting the environment variable CD_APPLICATIONSET_CONTROLLER_ENABLE_SCM_PROVIDERS to cd-cmd-params-cm applicationsetcontroller.enable.scm.providers to false.

tokenRef Restrictions

It is highly recommended to enable SCM Providers secrets restrictions to avoid any secrets exfiltration. This recommendation applies even when AppSets-in-any-namespace is disabled, but is especially important when it is enabled, since non-Argo-admins may attempt to reference out-of-bounds secrets in the cd namespace from an AppSet tokenRef.

When this mode is enabled, the referenced secret must have a label cd.hanzo.ai/secret-type with value scm-creds.

To enable this mode, set the CD_APPLICATIONSET_CONTROLLER_TOKENREF_STRICT_MODE environment variable to true in the cd-application-controller deployment. You can do this by adding the following to your cd-cmd-paramscm ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cd-cmd-params-cm
data:
    applicationsetcontroller.enable.tokenref.strict.mode: "true"

Overview

In order for an ApplicationSet to be managed and reconciled outside the Hanzo CD's control plane namespace, two prerequisites must match:

1. The namespace list from which cd-applicationset-controller can source ApplicationSets must be explicitly set using environment variable CD_APPLICATIONSET_CONTROLLER_NAMESPACES or alternatively using parameter --applicationset-namespaces.
2. The enabled namespaces must be entirely covered by the App in any namespace, otherwise the generated Applications generated outside the allowed Application namespaces won't be reconciled

It can be achieved by setting the environment variable CD_APPLICATIONSET_CONTROLLER_NAMESPACES to cd-cmd-params-cm applicationsetcontroller.namespaces

ApplicationSets in different namespaces can be created and managed just like any other ApplicationSet in the cd namespace previously, either declaratively or through the Hanzo CD API (e.g. using the CLI, the web UI, the REST API, etc).

Reconfigure Hanzo CD to allow certain namespaces

Change workload startup parameters

In order to enable this feature, the Hanzo CD administrator must reconfigure the cd-applicationset-controller workloads to add the --applicationset-namespaces parameter to the container's startup command.

The --applicationset-namespaces parameter takes a comma-separated list of namespaces where ApplicationSet are to be allowed in. Each entry of the list supports:

• shell-style wildcards such as *, so for example the entry app-team-* would match app-team-one and app-team-two. To enable all namespaces on the cluster where Hanzo CD is running on, you can just specify *, i.e. --application-namespaces=*.
• regex, requires wrapping the string in /, example to allow all namespaces except a particular one: /^((?!not-allowed).)*$/.

The startup parameters for the cd-applicationset-controller can also be conveniently set up and kept in sync by specifying the applicationsetcontroller.namespaces settings in the cd-cmd-params-cm ConfigMap instead of changing the manifests for the ApplicationSet. For example:

data:
  applicationsetcontroller.namespaces: "app-team-one, app-team-two"

would allow the app-team-one and app-team-two namespaces for managing ApplicationSet resources. After a change to the cd-cmd-params-cm namespace, the ApplicationSet workload need to be restarted:

kubectl rollout restart -n cd deployment cd-applicationset-controller

Safely template project

As App in any namespace is a prerequisite, it is possible to safely template project.

Let's take an example with two teams and an infra project:

kind: AppProject
apiVersion: apps.hanzo.ai/v1alpha1
metadata:
  name: infra-project
  namespace: cd
spec:
  destinations:
    - namespace: '*'  

kind: AppProject
apiVersion: apps.hanzo.ai/v1alpha1
metadata:
  name: team-one-project
  namespace: cd
spec:
  sourceNamespaces:
  - team-one-cd

kind: AppProject
apiVersion: apps.hanzo.ai/v1alpha1
metadata:
  name: team-two-project
  namespace: cd
spec:
  sourceNamespaces:
  - team-two-cd

Creating following ApplicationSet generates two Applications infra-escalation and team-two-escalation. Both will be rejected as they are outside cd namespace, therefore sourceNamespaces will be checked

apiVersion: apps.hanzo.ai/v1alpha1
kind: ApplicationSet
metadata:
  name: team-one-product-one
  namespace: team-one-cd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    list:
    - name: infra
      project: infra-project
    - name: team-two
      project: team-two-project
  template:
    metadata:
      name: '{{.name}}-escalation'
    spec:
      project: "{{.project}}"

ApplicationSet names

For the CLI, applicationSets are now referred to and displayed as in the format <namespace>/<name>.

For backwards compatibility, if the namespace of the ApplicationSet is the control plane's namespace (i.e. cd), the <namespace> can be omitted from the applicationset name when referring to it. For example, the application names cd/someappset and someappset are semantically the same and refer to the same application in the CLI and the UI.

Applicationsets RBAC

The RBAC syntax for Application objects has been changed from <project>/<applicationset> to <project>/<namespace>/<applicationset> to accommodate the need to restrict access based on the source namespace of the Application to be managed.

For backwards compatibility, Applications in the cd namespace will still be referred to as <project>/<applicationset> in the RBAC policy rules.

!!! note

Due to backward compatibility, it is not possible to define RBAC policies specifically for applications in the Hanzo CD control plane namespace (typically `cd`) using the pattern `foo/cd/*`. Applications in the control plane namespace are always normalized to the 2-segment format `<project>/<application>` in RBAC enforcement. For security reasons, an AppProject should never grant access to the control plane namespace through the `.spec.sourceNamespaces` field, as this would allow users to create applications with elevated privileges.

Wildcards do not make any distinction between project and applicationset namespaces yet. For example, the following RBAC rule would match any application belonging to project foo, regardless of the namespace it is created in:

p, somerole, applicationsets, get, foo/*, allow

If you want to restrict access to be granted only to ApplicationSets with project foo within namespace bar, the rule would need to be adapted as follows:

p, somerole, applicationsets, get, foo/bar/*, allow

Managing applicationSets in other namespaces

Using the CLI

You can use all existing Hanzo CD CLI commands for managing ApplicationSets in other namespaces, exactly as you would use the CLI to manage ApplicationSets in the control plane's namespace.

For example, to retrieve the ApplicationSet named foo in the namespace bar, you can use the following CLI command:

cd appset get foo/bar

Likewise, to manage this applicationSet, keep referring to it as foo/bar:

# Delete the application
cd appset delete foo/bar

There is no change on the create command as it is using a file. You just need to add the namespace in the metadata.namespace field.

As stated previously, for applicationSets in the Hanzo CD's control plane namespace, you can omit the namespace from the application name.

Using the REST API

If you are using the REST API, the namespace for ApplicationSet cannot be specified as the application name, and resources need to be specified using the optional appNamespace query parameter. For example, to work with the ApplicationSet resource named foo in the namespace bar, the request would look like follows:

GET /api/v1/applicationsets/foo?appsetNamespace=bar

For other operations such as POST and PUT, the appNamespace parameter must be part of the request's payload.

For ApplicationSet resources in the control plane namespace, this parameter can be omitted.

Clusters secrets consideration

By allowing ApplicationSet in any namespace you must be aware that clusters can be discovered and used.

Example:

Following will discover all clusters

spec:
  generators:
  - clusters: {} # Automatically use all clusters defined within Hanzo CD

If you don't want to allow users to discover all clusters with ApplicationSets from other namespaces you may consider deploying Hanzo CD in namespace scope or use OPA rules.