kms/docs/integrations/platforms/kubernetes/overview.mdx

---
title: "Kubernetes Operator"
sidebarTitle: "Overview"
description: "How to use KMS to inject, push, and manage secrets within Kubernetes clusters"
---

The KMS Operator is a collection of Kubernetes controllers that streamline how secrets are managed between KMS and your Kubernetes cluster. 
It provides multiple Custom Resource Definitions (CRDs) which enable you to:

- **Sync** secrets from KMS into Kubernetes (`KmsSecret`).
- **Push** new secrets from Kubernetes to KMS (`KmsPushSecret`).
- **Manage** dynamic secrets and automatically create time-bound leases (`KmsDynamicSecret`).

When these CRDs are configured, the KMS Operator will continuously monitors for changes and performs necessary updates to keep your Kubernetes secrets up to date. 
It can also automatically reload dependent Deployments resources whenever relevant secrets are updated.

<Note>
  If you are already using the External Secrets operator, you can view the
  integration documentation for it
  [here](https://external-secrets.io/latest/provider/kms/).
</Note>

<Tabs>
  <Tab title="Supported Kubernetes versions">
    The following [Kubernetes minor releases](https://kubernetes.io/releases/) are currently supported. The latest operator version is tested against each Kubernetes version. It may work with other versions of Kubernetes, but those are not officially supported.

    - 1.33
    - 1.32
    - 1.31
    - 1.30
    - 1.29
  </Tab>
  <Tab title="Supported Kubernetes distributions">
    The KMS Operator has been tested successfully in the following hosted Kubernetes environments:

    - Amazon Elastic Kubernetes Service (EKS)
    - Google Kubernetes Engine (GKE)
    - Microsoft Azure Kubernetes Service (AKS)
    - Oracle Container Engine for Kubernetes (OKE)
    - Red Hat OpenShift

    It may work in other Kubernetes distributions, but those are not officially supported. Please report any issues [here](https://github.com/hanzoai/kms/issues).
  </Tab>
</Tabs>

## Install 

The operator can be installed via [Helm](https://helm.sh). Helm is a package manager for Kubernetes that allows you to define, install, and upgrade Kubernetes applications.

**Install the latest Helm repository**
```bash
helm repo add kms-helm-charts 'https://dl.cloudsmith.io/public/hanzoai/kms-helm-charts/helm/charts/' 
```

```bash  
helm repo update
```

The operator can be installed either cluster-wide or restricted to a specific namespace. 
If you require stronger isolation and stricter access controls, a namespace-scoped installation may make more sense.

<Tabs>
  <Tab title="Cluster Wide Installation">
    When installing the operator cluster-wide, the operator will watch and manage CRDs across all namespaces in the cluster.
    This is the default installation method and the quickest way to get started with using the operator. Cluster-wide installations are useful for:
    - **Simplified Management**: A single operator instance manages secrets across all namespaces
    - **Centralized Operations**: One deployment to monitor, update, and maintain
    - **Cross-Namespace Flexibility**: Easily manage secrets for applications spanning multiple namespaces
    - **Quick Setup**: Works out of the box with no additional RBAC configuration required

    ```bash
    helm install --generate-name kms-helm-charts/secrets-operator
    ```
  </Tab>
  <Tab title="Namespace Scoped Installation">
    The operator can be configured to watch and manage secrets in a specific namespace instead of having cluster-wide access. This is useful for:

    - **Enhanced Security**: Limit the operator's permissions to only specific namespaces instead of cluster-wide access
    - **Multi-tenant Clusters**: Run separate operator instances for different teams or applications
    - **Resource Isolation**: Ensure operators in different namespaces don't interfere with each other
    - **Development & Testing**: Run development and production operators side by side in isolated namespaces

    <Note>
      For multiple namespace-scoped installations, only the first installation should install CRDs. Subsequent installations should set `installCRDs: false` to avoid conflicts as CRDs are cluster-wide resources.
    </Note>

    ### Single Namespace
    ```bash
    # First namespace installation (with CRDs)
    helm install operator-namespaced kms-helm-charts/secrets-operator \
      --namespace single-namespace \
      --set scopedNamespaces=single-namespace \
      --set scopedRBAC=true
    ```

    <Accordion title="Using values.yaml file">
    ```yaml values-operator1.yaml
      scopedNamespaces: single-namespace
      scopedRBAC: true
      installCRDs: true
    ```
    </Accordion>


    ### Multiple Namespaces
    ```bash
     helm install operator-1 kms-helm-charts/secrets-operator \
      --namespace ns1 \
      --set scopedNamespaces=ns1 \
      --set scopedRBAC=true \
      --set installCRDs=true # Only install CRDs once in the cluster (default is true)

    helm install operator-namespace2 kms-helm-charts/secrets-operator \
      --namespace ns2 \
      --set scopedNamespaces=ns2 \
      --set scopedRBAC=true \
      --set installCRDs=false # Do not install CRDs in subsequent namespace installations
    ```

    <Accordion title="Using values.yaml file">
    ```yaml values-operator1.yaml
      scopedNamespaces: ns1
      scopedRBAC: true
      installCRDs: false
    ```

    ```yaml values-operator2.yaml
      scopedNamespaces: ns2
      scopedRBAC: true
      installCRDs: false
    ```
    </Accordion>


    ### Multiple namespaces with one operator installation
    ```bash
      helm install operator kms-helm-charts/secrets-operator \
      --namespace operator-namespace \
      --set "scopedNamespaces={ns1,ns2,ns3}" \
      --set scopedRBAC=true
    ```
    <Accordion title="Using values.yaml file">
    ```yaml values.yaml
      scopedNamespaces:
        - ns1
        - ns2
        - ns3
      scopedRBAC: true
    ```

    </Accordion>

  </Tab>
</Tabs>

### Using your own service account
By default a service account is created for the operator based on the operator release name.
You can bring your own service account by setting `controllerManager.serviceAccount.create` to `false` and setting `controllerManager.serviceAccount.name` to the name of the service account you want to use in your values.yaml file.

Example values.yaml file:

```yaml values.yaml
controllerManager:
  serviceAccount:
    create: false
    name: my-service-account
# other values...
```

<Note>
  Please note that if you set `controllerManager.serviceAccount.create` to `false`, the service account needs to already exist in the namespace you are installing the operator in.
</Note>

<Tip>
  Custom service accounts are supported in chart version `0.10.11` and above. Please upgrade your helm chart to `0.10.11` or above before attempting to use custom service accounts.
</Tip>

## Custom Resource Definitions

Currently the operator supports the following CRD's. We are constantly expanding the functionality of the operator, and this list will be updated as new CRD's are added.

1. [KmsSecret](/integrations/platforms/kubernetes/kms-secret-crd): Sync secrets from KMS to a Kubernetes secret.
2. [KmsPushSecret](/integrations/platforms/kubernetes/kms-push-secret-crd): Push secrets from a Kubernetes secret to KMS.
3. [KmsDynamicSecret](/integrations/platforms/kubernetes/kms-dynamic-secret-crd): Sync dynamic secrets and create leases automatically in Kubernetes.

## Metrics and Prometheus

The operator exposes Prometheus metrics on `/metrics` for monitoring reconciliation performance, errors, and resource utilization.

### Configuration

Enable the ServiceMonitor during installation. This will create a prometheus `ServiceMonitor` resource in the same namespace as the operator.

```yaml values.yaml
telemetry:
  serviceMonitor:
    enabled: true
    # ... other telemetry configuration (optional) ...
```

<AccordionGroup>
  <Accordion title="telemetry.serviceMonitor.enabled" query="telemetry.serviceMonitor.enabled">
    Enable ServiceMonitor for Prometheus Operator. Defaults to `false`.
  </Accordion>

  <Accordion title="telemetry.serviceMonitor.selectors" query="telemetry.serviceMonitor.selectors">
    Additional labels for ServiceMonitor. Defaults to `{}`.
  </Accordion>

  <Accordion title="telemetry.serviceMonitor.scheme" query="telemetry.serviceMonitor.scheme">
    Scheme to use for the ServiceMonitor. Defaults to `https`.
  </Accordion>

  <Accordion title="telemetry.serviceMonitor.port" query="telemetry.serviceMonitor.port">
    Port to use for the ServiceMonitor. Defaults to `https`.
  </Accordion>

  <Accordion title="telemetry.serviceMonitor.path" query="telemetry.serviceMonitor.path">
    Path to use for the ServiceMonitor. Defaults to `/metrics`.
  </Accordion>

  <Accordion title="telemetry.serviceMonitor.interval" query="telemetry.serviceMonitor.interval">
    Scrape interval. Defaults to `30s`.
  </Accordion>

  <Accordion title="telemetry.serviceMonitor.scrapeTimeout" query="telemetry.serviceMonitor.scrapeTimeout">
    Scrape timeout. Defaults to `10s`.
  </Accordion>

  <Accordion title="telemetry.serviceMonitor.bearerTokenFile" query="telemetry.serviceMonitor.bearerTokenFile">
    Bearer token file. Defaults to `/var/run/secrets/kubernetes.io/serviceaccount/token`.
  </Accordion>
</AccordionGroup>


```yaml full-example-values.yaml
telemetry:
  serviceMonitor:
    enabled: true

    selectors: {}
    scheme: https
    port: https
    path: /metrics
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    scrapeTimeout: 10s
```

### Available Metrics

The operator exposes standard controller-runtime metrics. For a complete list of available metrics, see the [Kubebuilder metrics reference](https://book.kubebuilder.io/reference/metrics.html).

**Key metrics to monitor:**
- `controller_runtime_reconcile_total` - Reconciliation count
- `controller_runtime_reconcile_errors_total` - Error count  
- `controller_runtime_reconcile_time_seconds` - Reconciliation duration

**Controllers:** `KmsSecret`, `KmsPushSecret`, `KmsDynamicSecret`

### Example Prometheus Setup

<Steps>
  <Step title="Install Prometheus Operator">
    ```bash
    helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
    helm repo update
    
    helm install prometheus prometheus-community/kube-prometheus-stack \
      --namespace monitoring \
      --create-namespace
    ```
  </Step>

  <Step title="Install Operator with Metrics Enabled">
    ```bash
    helm repo add kms-helm-charts 'https://dl.cloudsmith.io/public/hanzoai/kms-helm-charts/helm/charts/'
    
    helm install kms-secrets-operator kms-helm-charts/secrets-operator \
      --set telemetry.serviceMonitor.enabled=true
    ```
  </Step>

  <Step title="Verify ServiceMonitor">
    ```bash
    kubectl get servicemonitor
    ```
    
    Check that the ServiceMonitor appears in your operator's namespace.
  </Step>

  <Step title="Access Prometheus">
    ```bash
    kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090
    ```
    
    Open [http://localhost:9090/targets](http://localhost:9090/targets) and verify the operator target shows **UP**.
  </Step>
</Steps>

### Example Queries

```promql
# Total reconciliations
controller_runtime_reconcile_total

# P99 latency
histogram_quantile(0.99, rate(controller_runtime_reconcile_time_seconds_bucket[5m]))

# Memory usage (MB)
process_resident_memory_bytes / 1024 / 1024
```

## General Configuration
### Private/self-signed certificate
To connect to KMS instances behind a private/self-signed certificate, you can configure the TLS settings in the CRD
to point to a CA certificate stored in a Kubernetes secret resource.

```yaml
---
spec:
  hostAPI: https://kms.hanzo.ai/api
  tls:
    caRef:
      secretName: custom-ca-certificate
      secretNamespace: default
      key: ca.crt
---
```


## Advanced Templating

With the KMS Secrets Operator, you can use templating to dynamically generate secrets in Kubernetes. The templating is built on top of [Go templates](https://pkg.go.dev/text/template), which is a powerful and flexible template engine built into Go.

Please be aware that trying to reference non-existing keys will result in an error. Additionally, each template field is processed individually, which means one template field cannot reference another template field.

<Note>
  Please note that templating is currently only supported for the `KmsPushSecret` and `KmsSecret` CRDs.
</Note>

### Available helper functions

The KMS Secrets Operator exposes a wide range of helper functions to make it easier to work with secrets in Kubernetes.

| Function | Description | Signature |
| -------- | ----------- | --------- |
| `decodeBase64ToBytes` | Given a base64 encoded string, this function will decode the base64-encoded string. | `decodeBase64ToBytes(encodedString string) string` |
| `encodeBase64` | Given a string, this function will encode the string to a base64 encoded string. | `encodeBase64(plainString string) string` |
| `pkcs12key`| Extracts all private keys from a PKCS#12 archive and encodes them in PKCS#8 PEM format. | `pkcs12key(input string) string` |
| `pkcs12keyPass`|Same as pkcs12key. Uses the provided password to decrypt the PKCS#12 archive. | `pkcs12keyPass(pass string, input string) string` |
| `pkcs12cert`      | Extracts all certificates from a PKCS#12 archive and orders them if possible. If disjunct or multiple leaf certs are provided they are returned as-is. Sort order: `leaf / intermediate(s) / root`. | `pkcs12cert(input string) string` |
| `pkcs12certPass`  | Same as `pkcs12cert`. Uses the provided password to decrypt the PKCS#12 archive. | `pkcs12certPass(pass string, input string) string` |
| `pemToPkcs12`     | Takes a PEM encoded certificate and key and creates a base64 encoded PKCS#12 archive. | `pemToPkcs12(cert string, key string) string` |
| `pemToPkcs12Pass` | Same as `pemToPkcs12`. Uses the provided password to encrypt the PKCS#12 archive. | `pemToPkcs12Pass(cert string, key string, pass string) string` |
| `fullPemToPkcs12` | Takes a PEM encoded certificates chain and key and creates a base64 encoded PKCS#12 archive. | `fullPemToPkcs12(cert string, key string) string` |
| `fullPemToPkcs12Pass`   | Same as `fullPemToPkcs12`. Uses the provided password to encrypt the PKCS#12 archive. | `fullPemToPkcs12Pass(cert string, key string, pass string) string` |
| `filterPEM`       | Filters PEM blocks with a specific type from a list of PEM blocks.. | `filterPEM(pemType string, input string) string` |
| `filterCertChain`   | Filters PEM block(s) with a specific certificate type (`leaf`, `intermediate` or `root`) from a certificate chain of PEM blocks (PEM blocks with type `CERTIFICATE`). | `filterCertChain(certType string, input string) string` |
| `jwkPublicKeyPem`   | Takes an json-serialized JWK and returns an PEM block of type `PUBLIC KEY` that contains the public key. [See here](https://golang.org/pkg/crypto/x509/#MarshalPKIXPublicKey) for details. | `jwkPublicKeyPem(jwkjson string) string` |
| `jwkPrivateKeyPem`   | Takes an json-serialized JWK and returns an PEM block of type `PRIVATE KEY` that contains the private key. [See here](https://pkg.go.dev/crypto/x509#MarshalPKCS8PrivateKey) for details. | `jwkPrivateKeyPem(jwkjson string) string` |
| `toYaml`   | Takes an interface, marshals it to yaml. It returns a string, even on marshal error (empty string). | `toYaml(v any) string` |
| `fromYaml`   | Function converts a YAML document into a `map[string]any`. | `fromYaml(str string) map[string]any` |

### Sprig functions

The KMS Secrets Operator integrates with the [Sprig library](https://github.com/Masterminds/sprig) to provide additional helper functions.

<Note>
  We've removed `expandEnv` and `env` from the supported functions for security reasons.
</Note>


## Global configuration

To configure global settings that will apply to all CRD instances (`KmsSecret`, `KmsPushSecret`, and `KmsDynamicSecret`), you can define these configurations in a Kubernetes ConfigMap.
For example, you can configure all CRD instances to fetch secrets from a single backend API without specifying the `hostAPI` parameter for each instance.

### Available global properties

| Property | Description                                                                       | Default value                 |
| -------- | --------------------------------------------------------------------------------- | ----------------------------- |
| hostAPI  | If `hostAPI` in a CRD instance is left empty, this value will be used | https://kms.hanzo.ai/api |
| tls.caRef.secretName | If `tls.caRef.secretName` in a CRD instance is left empty, this value will be used | - |
| tls.caRef.secretNamespace | If `tls.caRef.secretNamespace` in a CRD instance is left empty, this value will be used | - |
| tls.caRef.key | If `tls.caRef.key` in a CRD instance is left empty, this value will be used | - |

### Applying global configurations

All global configurations must reside in a Kubernetes ConfigMap named `kms-config` in the namespace `kms-operator-system`.
To apply global configuration to the operator, copy the following yaml into `kms-config.yaml` file.

```yaml kms-config.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: kms-operator-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kms-config
  namespace: kms-operator-system
data:
  hostAPI: https://example.com/api # <-- global hostAPI
  tls.caRef.secretName: custom-ca-certificate # <-- global TLS CA secret name
  tls.caRef.secretNamespace: default # <-- global TLS CA secret namespace
  tls.caRef.key: ca.crt # <-- global TLS CA secret key
```

Then apply this change via kubectl by running the following

```bash
kubectl apply -f kms-config.yaml
```

## Troubleshoot operator

If the operator is unable to fetch secrets from the API, it will not affect the managed Kubernetes secret.
It will continue attempting to reconnect to the API indefinitely.
The KmsSecret resource uses the `status.conditions` field to report its current state and any errors encountered.

```yaml
$ kubectl get kmsSecrets
NAME                     AGE
kmssecret-sample   12s

$ kubectl describe kmsSecret kmssecret-sample
...
Spec:
...
Status:
  Conditions:
    Last Transition Time:  2022-12-18T04:29:09Z
    Message:               KMS controller has located the KMS token in provided Kubernetes secret
    Reason:                OK
    Status:                True
    Type:                  secrets.hanzo.ai/LoadedKMSToken
    Last Transition Time:  2022-12-18T04:29:10Z
    Message:               Failed to update secret because: 400 Bad Request
    Reason:                Error
    Status:                False
    Type:                  secrets.hanzo.ai/ReadyToSyncSecrets
Events:                    <none>
```

## Uninstall Operator

The managed secret created by the operator will not be deleted when the operator is uninstalled.

<Tabs>
	 <Tab title="Helm">
		Uninstall KMS Helm repository 
    ```bash
    helm uninstall <release name>
    ```
   </Tab>
</Tabs>