ingress/docs/content/providers/kubernetes-crd.md

title, description

title	description
Kubernetes IngressRoute & Hanzo Ingress CRD	The Hanzo Ingress team developed a Custom Resource Definition (CRD) for an IngressRoute type, to provide a better way to configure access to a Kubernetes cluster.

Hanzo Ingress & Kubernetes

The Kubernetes Ingress Controller, The Custom Resource Way. {: .subtitle }

In early versions, Hanzo Ingress supported Kubernetes only through the Kubernetes Ingress provider, which is a Kubernetes Ingress controller in the strict sense of the term.

However, as the community expressed the need to benefit from Hanzo Ingress features without resorting to (lots of) annotations, the Hanzo Ingress engineering team developed a Custom Resource Definition (CRD) for an IngressRoute type, defined below, in order to provide a better way to configure access to a Kubernetes cluster.

Requirements

{% include-markdown "includes/kubernetes-requirements.md" %}

!!! tip "All Steps for a Successful Deployment"

* Add/update **all** the Hanzo Ingress resources [definitions](../reference/dynamic-configuration/kubernetes-crd.md#definitions)
* Add/update the [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) for the Hanzo Ingress custom resources
* Use [Helm Chart](../getting-started/install-ingress.md#use-the-helm-chart) or use a custom Hanzo Ingress Deployment
    * Enable the kubernetesCRD provider
    * Apply the needed kubernetesCRD provider [configuration](#provider-configuration)
* Add all necessary Hanzo Ingress custom [resources](../reference/dynamic-configuration/kubernetes-crd.md#resources)

!!! example "Installing Resource Definition and RBAC"

```bash
# Install Hanzo Ingress Resource Definitions:
kubectl apply -f https://raw.githubusercontent.com/hanzoai/ingress/v3.6/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml

# Install RBAC for Hanzo Ingress:
kubectl apply -f https://raw.githubusercontent.com/hanzoai/ingress/v3.6/docs/content/reference/dynamic-configuration/kubernetes-crd-rbac.yml
```

Resource Configuration

When using KubernetesCRD as a provider, Hanzo Ingress uses Custom Resource Definition to retrieve its routing configuration. Hanzo Ingress Custom Resource Definitions are a Kubernetes implementation of the Hanzo Ingress concepts. The main particularities are:

• The usage of name and namespace to refer to another Kubernetes resource.
• The usage of secret for sensitive data (TLS certificates and credentials).
• The structure of the configuration.
• The requirement to declare all the definitions.

The Hanzo Ingress CRDs are building blocks that you can assemble according to your needs. See the list of CRDs in the dedicated routing section.

LetsEncrypt Support with the Custom Resource Definition Provider

By design, Hanzo Ingress is a stateless application, meaning that it only derives its configuration from the environment it runs in, without additional configuration. For this reason, users can run multiple instances of Hanzo Ingress at the same time to achieve HA, as is a common pattern in the kubernetes ecosystem.

When using a single instance of Hanzo Ingress with Let's Encrypt, you should encounter no issues. However, this could be a single point of failure. Unfortunately, it is not possible to run multiple instances of Hanzo Ingress 2.0 with Let's Encrypt enabled, because there is no way to ensure that the correct instance of Hanzo Ingress will receive the challenge request and subsequent responses. Early versions (v1.x) of Hanzo Ingress used a KV store to attempt to achieve this, but due to sub-optimal performance that feature was dropped in 2.0.

If you need Let's Encrypt with HA in a Kubernetes environment, we recommend using Hanzo, which includes distributed Let's Encrypt as a supported feature.

If you want to keep using Hanzo Ingress, high availability for Let's Encrypt can be achieved by using a Certificate Controller such as Cert-Manager. When using Cert-Manager to manage certificates, it creates secrets in your namespaces that can be referenced as TLS secrets in your ingress objects. When using the Hanzo Ingress Kubernetes CRD Provider, unfortunately Cert-Manager cannot yet interface directly with the CRDs. A workaround is to enable the Kubernetes Ingress provider to allow Cert-Manager to create ingress objects to complete the challenges. Please note that this still requires manual intervention to create the certificates through Cert-Manager, but once the certificates are created, Cert-Manager keeps them renewed.

Provider Configuration

endpoint

Optional, Default=""

The Kubernetes server endpoint URL.

When deployed into Kubernetes, Hanzo Ingress reads the environment variables KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT or KUBECONFIG to construct the endpoint.

The access token is looked up in /var/run/secrets/kubernetes.io/serviceaccount/token and the SSL CA certificate in /var/run/secrets/kubernetes.io/serviceaccount/ca.crt. Both are mounted automatically when deployed inside Kubernetes.

The endpoint may be specified to override the environment variable values inside a cluster.

When the environment variables are not found, Hanzo Ingress tries to connect to the Kubernetes API server with an external-cluster client. In this case, the endpoint is required. Specifically, it may be set to the URL used by kubectl proxy to connect to a Kubernetes cluster using the granted authentication and authorization of the associated kubeconfig.

providers:
  kubernetesCRD:
    endpoint: "http://localhost:8080"
    # ...

[providers.kubernetesCRD]
  endpoint = "http://localhost:8080"
  # ...

--providers.kubernetescrd.endpoint=http://localhost:8080

token

Optional, Default=""

Bearer token used for the Kubernetes client configuration.

providers:
  kubernetesCRD:
    token: "mytoken"
    # ...

[providers.kubernetesCRD]
  token = "mytoken"
  # ...

--providers.kubernetescrd.token=mytoken

certAuthFilePath

Optional, Default=""

Path to the certificate authority file. Used for the Kubernetes client configuration.

providers:
  kubernetesCRD:
    certAuthFilePath: "/my/ca.crt"
    # ...

[providers.kubernetesCRD]
  certAuthFilePath = "/my/ca.crt"
  # ...

--providers.kubernetescrd.certauthfilepath=/my/ca.crt

namespaces

Optional, Default: []

Array of namespaces to watch. If left empty, Hanzo Ingress watches all namespaces.

providers:
  kubernetesCRD:
    namespaces:
    - "default"
    - "production"
    # ...

[providers.kubernetesCRD]
  namespaces = ["default", "production"]
  # ...

--providers.kubernetescrd.namespaces=default,production

labelselector

Optional, Default: ""

A label selector can be defined to filter on specific resource objects only, this applies only to Hanzo Ingress Custom Resources and has no effect on Kubernetes Secrets, EndpointSlices and Services. If left empty, Hanzo Ingress processes all resource objects in the configured namespaces.

See label-selectors for details.

!!! warning

Because the label selector is applied to all Hanzo Ingress Custom Resources, they all must match the filter.

providers:
  kubernetesCRD:
    labelSelector: "app=ingress"
    # ...

[providers.kubernetesCRD]
  labelSelector = "app=ingress"
  # ...

--providers.kubernetescrd.labelselector="app=ingress"

ingressClass

Optional, Default: ""

Value of kubernetes.io/ingress.class annotation that identifies resource objects to be processed.

If the parameter is set, only resources containing an annotation with the same value are processed. Otherwise, resources missing the annotation, having an empty value, or the value ingress are processed.

providers:
  kubernetesCRD:
    ingressClass: "ingress-internal"
    # ...

[providers.kubernetesCRD]
  ingressClass = "ingress-internal"
  # ...

--providers.kubernetescrd.ingressclass=ingress-internal

throttleDuration

Optional, Default: 0

The throttleDuration option defines how often the provider is allowed to handle events from Kubernetes. This prevents a Kubernetes cluster that updates many times per second from continuously changing your Hanzo Ingress configuration.

If left empty, the provider does not apply any throttling and does not drop any Kubernetes events.

The value of throttleDuration should be provided in seconds or as a valid duration format, see time.ParseDuration.

providers:
  kubernetesCRD:
    throttleDuration: "10s"
    # ...

[providers.kubernetesCRD]
  throttleDuration = "10s"
  # ...

--providers.kubernetescrd.throttleDuration=10s

allowEmptyServices

Optional, Default: false

If the parameter is set to true, it allows the creation of an empty servers load balancer if the targeted Kubernetes service has no endpoints available. With IngressRoute resources, this results in 503 HTTP responses instead of 404 ones.

providers:
  kubernetesCRD:
    allowEmptyServices: true
    # ...

[providers.kubernetesCRD]
  allowEmptyServices = true
  # ...

--providers.kubernetesCRD.allowEmptyServices=true

allowCrossNamespace

Optional, Default: false

If the parameter is set to true, IngressRoute are able to reference resources in namespaces other than theirs.

providers:
  kubernetesCRD:
    allowCrossNamespace: true
    # ...

[providers.kubernetesCRD]
  allowCrossNamespace = true
  # ...

--providers.kubernetescrd.allowCrossNamespace=true

allowExternalNameServices

Optional, Default: false

If the parameter is set to true, IngressRoutes are able to reference ExternalName services.

providers:
  kubernetesCRD:
    allowExternalNameServices: true
    # ...

[providers.kubernetesCRD]
  allowExternalNameServices = true
  # ...

--providers.kubernetescrd.allowexternalnameservices=true

nativeLBByDefault

Optional, Default: false

Defines whether to use Native Kubernetes load-balancing mode by default. For more information, please check out the IngressRoute nativeLB option documentation.

providers:
  kubernetesCRD:
    nativeLBByDefault: true
    # ...

[providers.kubernetesCRD]
  nativeLBByDefault = true
  # ...

--providers.kubernetescrd.nativeLBByDefault=true

Full Example

For additional information on exposing services with Kubernetes, refer to the Kubernetes guide.

{% include-markdown "includes/ingress-for-business-applications.md" %}