mirror of https://github.com/hanzoai/coredns-manager-operator.git synced 2026-10-30 18:43:53 +00:00

Files

T

105cf45897 Bug fix complex ns1 hostname (#10 ) (#11 )

* Update issue templates

* remove bug template

* Create SECURITY.md

* docs - add medium article.

* bugFix - fix validation expression; regen manifests

* security: update version

* update to 1.0.3

* qa - new test scenario

* changelog - 1.0.3

---------

Co-authored-by: Nikolay Udovik <[email protected]>

2024-06-13 13:46:06 -05:00

24 KiB

Raw Permalink Blame History

Manual Testing Guide for CoreDNS Operator

>IMPORTANT: Follow this scenario precisely. All steps should function as intended, leading to results that match the provided example. If unexpected behavior occurs, or the outcome deviates from the example, STOP. Troubleshoot the issue using available resources or report it with details.

NOTE: This guide shows examples of operator log output. However, the format might vary depending on the log level configuration.

Prerequisites

k3s version: => v1.29.5+k3s1 or KinD => 0.20.0
kube-dns service is exposed
coredns-controller-manager is running
It is recommended to switch the current context to the kube-system

Basic Functionality Test

To install demo resources from config/samples use the following command.

$ kubectl apply -k config/samples/

This will install the following components:

DNSConnector: Integrates with k3s's coredns.
DNSZone
- market-example-zone: Normal zone.
- no-connector-test-zone: Zone to test behavior when the wrong connector is requested.
DNSRecords
- A total of 12 records will be deployed for market.example.com zone.
- 1 DNSRecord without a DNSZone.

Run apply and wait for about a minute for reconciliation to complete. Keep an eye on controller logs to ensure they remain error-free.

DNSConnector Review

Describe DNSConnector resource.

$ kubectl get dnsconnectors coredns -o json | jq '{spec: .spec, status: .status}'

{
  "spec": {
    "corednsCM": {
      "corefileKey": "Corefile",
      "name": "coredns"
    },
    "corednsDeployment": {
      "name": "coredns",
      "type": "Deployment",
      "zonefilesMountDir": "/opt/coredns"
    },
    "corednsZoneEnaledPlugins": [
      "errors",
      "log"
    ],
    "waitForUpdateTimeout": 120
  },
  "status": {
    "conditions": [
      {
        "lastTransitionTime": "2024-06-04T13:01:57Z",
        "message": "CoreDNS Ready",
        "observedGeneration": 1,
        "reason": "Active",
        "status": "True",
        "type": "Ready"
      }
    ],
    "provisionedZones": [
      {
        "domain": "market.example.com",
        "name": "market-example-zone",
        "serialNumber": "0604130139"
      }
    ]
  }
}

The setup should be ready and include provisionedZones.

Inspect coredns deployment
```
$ kubectl describe deployments coredns | grep Mount -A4
```
Ensure that it includes market.example.com mount.
DNSConnector will backup the original coredns configMap. Make sure it exists
```
$ kubectl describe cm coredns-original-configmap
```
Inspect the coredns configMap
```
$ kubectl describe cm coredns
```
At the end of the file, it must contain the following data.

DNSZone Review

Get DNSZone resources.

$ kubectl get dnszones
NAME                     DOMAIN NAME          RECORD COUNT   LAST CHANGE            CURRENT SERIAL   STATE
no-connector-test-zone   test.com.            0              2024-06-04T12:57:41Z   0604125740       Pending
market-example-zone      market.example.com   12             2024-06-04T12:57:40Z   0604130139       Active

no-connector-test-zone - should be in the Pending state
market-example-zone - should be in the Active state

Compare that the "no-connector-test-zone" has the same serial as the DNSConnector.

# Get connector status
$ kubectl get dnsconnector coredns -o jsonpath='{.status.provisionedZones}' | jq .

# Get dnszone serial
$ kubectl get dnszone market-example-zone -o jsonpath='{.status.currentZoneSerial}' | jq

The DNSZone resource produces the zone configMap. If you attempt to delete it, the finalizer should prevent you.
```
$ kubectl delete cm coredns-zone-market-example-zone
^C
```

Review DNSRecords

Get DNSRecord resources. The "record-no-zone-test" should be in the Pending state, while the rest should be Ready.

$ kubectl get dnsrecords
NAME                              RECORD NAME                RECORD TYPE   RECORD VALUE                                                                    ZONE REFERENCE             LAST CHANGE            STATE
record-no-zone-test               haproxy.cpe                A             10.149.149.10                                                                   this-zone-does-not-exist   2024-06-04T12:57:40Z   Pending
dnskey-market-example             @                          DNSKEY        257 3 8 AwEAAaKdjGVkvFjdLv...5A5t5y7yQ8r                                        market-example-zone        2024-06-04T12:57:40Z   Ready
ftp-cname-market-example          ftp                        CNAME         www.market.example.com.                                                         market-example-zone        2024-06-04T12:57:40Z   Ready
hinfo-market-example              @                          HINFO         Intel x86_64                                                                    market-example-zone        2024-06-04T12:57:40Z   Ready
mail-a-market-example             mail                       A             10.100.100.20                                                                   market-example-zone        2024-06-04T12:57:40Z   Ready
mail-mx-market-example            @                          MX            10 mail.market.example.com.                                                     market-example-zone        2024-06-04T12:57:40Z   Ready
ns1-aaaa-market-example           ns1                        AAAA          2001:db8::2                                                                     market-example-zone        2024-06-04T12:57:40Z   Ready
ns2-a-market-example              ns2                        A             192.168.1.2                                                                     market-example-zone        2024-06-04T12:57:40Z   Ready
rrsig-market-example              @                          RRSIG         A 8 3 86400 20240601000000 20240501000000 12345 market.example.com. b1N2...==   market-example-zone        2024-06-04T12:57:40Z   Ready
spf-txt-market-example            @                          TXT           v=spf1 include:example.com ~all                                                 market-example-zone        2024-06-04T12:57:40Z   Ready
srv-market-example                _sip._tcp                  SRV           10 5 5060 sipserver.example.com.                                                market-example-zone        2024-06-04T12:57:40Z   Ready
www-a-market-example              www                        A             10.100.100.10                                                                   market-example-zone        2024-06-04T12:57:40Z   Ready
app1-a-fqdn-test-market-example   app1.market.example.com.   A             10.100.100.10                                                                   market-example-zone        2024-06-04T13:01:39Z   Ready

Inspect "record-no-zone-test"

$ kubectl get dnsrecords record-no-zone-test -o jsonpath="{.status}" | jq .conditions[].message
"Record has been constructed. Awaiting for the dnszone controller to pick up the record"

Inspect the A record "www-a-market-example". Ensure that all Status fields are populated correctly.
```
$ kubectl describe dnsrecords www-a-market-example
```

Try to resolve some record.

$ dig +short @192.168.122.10 app1.market.example.com
10.100.100.10

Remove record test

Delete the record and monitor the logs. The process should take a few seconds and complete without errors in controller logs.
```
$ kubectl delete dnsrecords spf-txt-market-example
```

Get dnszone resource and verify that the number of records is 11 and the DNSZone is still Active.

kubectl get DNSZones 
NAME                     DOMAIN NAME          RECORD COUNT   LAST CHANGE            CURRENT SERIAL   STATE
no-connector-test-zone   test.com.            0              2024-06-04T17:17:20Z   0604171720       Pending
market-example-zone      market.example.com   11             2024-06-04T17:17:20Z   0604172502       Active

Ensure that the DNSConnector and DNSZone have the same serial, as the DNSZone serial has been changed.

# Get connector status
$ kubectl get dnsconnector coredns -o jsonpath='{.status.provisionedZones}' | jq .

# Get dnszone serial
$ kubectl get dnszone market-example-zone -o jsonpath='{.status.currentZoneSerial}' | jq

Remove a functional zone

Remove the DNSZone "market-example-zone". Monitor the logs. The process should take a few seconds without any errors.

$ kubectl delete dnszones market-example-zone

All DNSRecords should be in Pending state.
```
$ kubectl get dnsrecords
```

The zone cm has been dissapeared.

$ kubectl get cm coredns-zone-market-example-zone
Error from server (NotFound): configmaps "coredns-zone-market-example-zone" not found

Inspect the CoreDNS deployment and config map.

# check volume mounts and mounts. ensure that the zone mount has been removed
$ kubectl describe deployments.apps coredns
# ensure that market.example.com has been removed
$ kubectl describe cm coredns

Remove an empty zone

Should work

 $ kubectl delete dnszones no-connector-test-zone

Remove DNSConnector

Delete the resource, monitor the logs, and wait for reconciliation. Ensure there are no errors.
```
$ kubectl delete dnsconnector coredns
```
Ensure that coredns is up and running.
```
kubectl get deployments.apps coredns
```
Original coredns configMap has been restored
```
$ kubectl describe cm coredns
```

Ensure that no Zone Mounts in here

kubectl describe deployments.apps coredns

Remove active DNSConnector

Restore the demo resources and wait to Reconcilation to complete.
```
$ kubectl apply -k config/samples/
```
Remove connector.
```
$ kubectl delete dnsconnectors coredns
```

Dns records should become Pending

$ kubectl get dnszone market-example-zone -o jsonpath='{.status}' | jq .

{
  "checkpoint": true,
  "conditions": [
    {
      "lastTransitionTime": "2024-06-04T23:48:04Z",
      "message": "DNSConnector has been removed",
      "observedGeneration": 1,
      "reason": "Pending",
      "status": "False",
      "type": "Ready"
    }
  ],
  "currentZoneSerial": "0604232845",
  "recordCount": 12,
  "validationPassed": true,
  "zoneConfigmap": "coredns-zone-market-example-zone"
}

Original coredns configMap has been restored
```
$ kubectl describe cm coredns
```
In a few minutes coredns will reload the restored coredns cm. However the operator will not remove Volume, VolumeMount and Mount of zone files. You can remove it manually.
```
# remove zone volumes and volumeMounts
$ kubectl edit deployments.apps coredns
# ensure that coredns is healthy
$ kubectl describe deployments.apps coredns
```

Test Exceptions

Restore the demo resources

Restore the demo resources.

$ kubectl apply -k config/samples/

Test bad DNSRecord creation

We will create a record that will fail due to a bad domain name.

Get the current serial number of the DNSZone.

$ kubectl get dnszone market-example-zone -o jsonpath='{.status.currentZoneSerial}' | jq .

Create the DNSRecord resource

$ cat << EOF | kubectl apply -f -
---
apiVersion: monkale.monkale.io/v1alpha1
kind: DNSRecord
metadata:
  name: bad-domain-record-test
  namespace: kube-system
spec:
  record:
    name: "test.."
    value: "192.0.2.1"
    type: "A"
  dnsZoneRef:
    name: "market-example-zone"
EOF

Check status.

$ kubectl get dnsrecords bad-domain-record-test -o jsonpath='{.status}' | jq .

   {
  "conditions": [
    {
      "lastTransitionTime": "2024-06-04T14:04:18Z",
      "message": "Record validation failure: error parsing records: dns: bad owner name: \"test..\" at line: 1:6",
      "observedGeneration": 1,
      "reason": "Degraded",
      "status": "False",
      "type": "Ready"
    }
  ],
  "generatedRecord": "test.. IN A 192.0.2.1"
}

Controller log tries to reconcile.

2024-06-02T18:39:26-05:00       ERROR   DNSRecord instance. Proccess Record Failure     {"DNSRecord.Name": "bad-domain-record-test", "DNSZone.Name": "market-example-zone", "error": "handle ARecord Error: record validation failure: error parsing records: dns: bad owner name: \"test..\" at line: 1:6"}
2024-06-02T18:39:26-05:00       ERROR   Reconciler error        {"controller": "dnsrecord", "controllerGroup": "monkale.monkale.io", "controllerKind": "DNSRecord", "DNSRecord": {"name":"bad-domain-record-test","namespace":"kube-system"}, "namespace": "kube-system", "name": "bad-domain-record-test", "reconcileID": "f537266d-46bf-491c-8a86-7f28f90a6834", "error": "handle ARecord Error: record validation failure: error parsing records: dns: bad owner name: \"test..\" at line: 1:6"}

Ensure that the DNSZone has not been affected by this change. The serial should remain the same.
```
$ kubectl get dnszone market-example-zone -o jsonpath='{.status.currentZoneSerial}' | jq
```

Test bad DNSRecord fix

Let's fix the record. The record should become "Active".

# Fix
$ kubectl patch DNSRecord bad-domain-record-test --type='merge' -p='{"spec":{"record": {"name": "test"}}}'

# Check status 
$ kubectl get dnsrecords bad-domain-record-test -o jsonpath='{.status}' | jq .

# Serial has Changed
$ kubectl get dnszone market-example-zone -o jsonpath='{.status.currentZoneSerial}' | jq

Now you should be able to resolve this test.market.example.com

Test bad DNSRecord modification

Make sure you can resolve the test record

$ dig +short @192.168.122.10 test.market.example.com
192.0.2.1

Misconfigure the dnsrecord

$ kubectl patch DNSRecord bad-domain-record-test --type='merge' -p='{"spec":{"record": {"name": "test.."}}}'

Check DNSRecord status and logs again.
dig test.market.example.com should return NXDOMAIN

Test bad DNSRecord deletion

Delete the record.

$ kubectl delete dnsrecords.monkale.monkale.io bad-domain-record-test
dnsrecord.monkale.monkale.io "bad-domain-record-test" deleted

Test bad DNSZone creation

To break the DNSZone, let's request NS Record type AAAA for the IPv4 NS.

Create bad DNSZone

$ cat << EOF | kubectl apply -f -
---
apiVersion: monkale.monkale.io/v1alpha1
kind: DNSZone
metadata:
  name: i-am-broken-zone
  namespace: kube-system
spec:
  connectorName: coredns
  domain: "broken.com."
  primaryNS:
    recordType: "AAAA"
    ipAddress: "10.120.100.11"
  respPersonEmail: "[email protected]"
EOF

Make sure that coredns and dnsconnector are ok

$ kubectl get dnsconnector
NAME      LAST CHANGE            STATE    MESSAGE
coredns   2024-06-04T16:20:15Z   Active   CoreDNS Ready
$ kubectl get deployments.apps coredns 
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
coredns   1/1     1            1           4d

Ensure that DNSConnector does not contain this zone

$ kubectl get dnsconnector coredns -o jsonpath='{.status.provisionedZones}' | jq .

Remove this zone and repeat step 2.

$ kubectl delete dnszone i-am-broken-zone

Test bad DNSZone modification

To break the DNSZone, let's patch primary NS IP to IPv5 :)

Get the current serial number of the DNSZone.

$ kubectl get dnszone market-example-zone -o jsonpath='{.status.currentZoneSerial}' | jq

Patch

$ kubectl patch DNSzone market-example-zone --type='merge' -p='{"spec":{"primaryNS":{"ipAddress": "10.120.120.10.254"}}}'

Check status.

$ kubectl get DNSzone market-example-zone -o jsonpath='{.status}' | jq .

{
  "conditions": [
    {
      "lastTransitionTime": "2024-06-04T14:06:24Z",
      "message": "Zone validation failure. Preserving the previous version. Error: error parsing records: dns: bad A A: \"10.120.120.10.254\" at line: 11:26",
      "observedGeneration": 2,
      "reason": "UpdateError",
      "status": "False",
      "type": "Ready"
    }
  ],
  "currentZoneSerial": "0604140605",
  "recordCount": 12,
  "validationPassed": true,
  "zoneConfigmap": "coredns-zone-market-example-zone"
}

The DNSZone notifies that an error occurred, the change is not applied, and it will keep the previous version.
Serial should remain the same

Check controller's logs

2024-06-02T18:58:58-05:00       ERROR   DNSZone instance. Reconciling ZoneCM.   {"ConfigMap.metadata.name": "coredns-zone-market-example-zone", "DNSZone.Name": "market-example-zone", "error": "error parsing records: dns: bad A A: \"10.120.120.10.254\" at line: 11:26"}
2024-06-02T18:58:58-05:00       ERROR   DNSZone instance. Generate ZoneCM. Failed to create or update Zone CM   {"DNSZone.Name": "market-example-zone", "error": "error parsing records: dns: bad A A: \"10.120.120.10.254\" at line: 11:26"}
2024-06-02T18:58:58-05:00       ERROR   Reconciler error        {"controller": "dnszone", "controllerGroup": "monkale.monkale.io", "controllerKind": "DNSZone", "DNSZone": {"name":"market-example-zone","namespace":"kube-system"}, "namespace": "kube-system", "name": "market-example-zone", "reconcileID": "6c33c720-f8f8-452c-8ab1-1e5f2f27a70f", "error": "error parsing records: dns: bad A A: \"10.120.120.10.254\" at line: 11:26"}

Check that DNSConnector is still in the Ready state and has the same Serial number.
```
$ kubectl get dnsconnector coredns -o jsonpath='{.status}' | jq .
```
Inspect the coredns configMap
```
$ kubectl describe cm coredns
```
At the end of the file, it must contain the following data.

Name resolution is still working

$ dig +short @192.168.122.10 app1.market.example.com
10.100.100.10

Fix it back to IPv4

# Fix
$ kubectl patch DNSzone market-example-zone --type='merge' -p='{"spec":{"primaryNS":{"ipAddress": "10.120.120.254"}}}'

# Serial has Changed
$ kubectl get dnszone market-example-zone -o jsonpath='{.status.currentZoneSerial}' | jq .

# Connector is functional and has the same serial.
$ kubectl get dnsconnector coredns -o jsonpath='{.status}' | jq .

Test DNSZone domain change

We will test whether it is possible to change the zone domain name. Additionally, we will test if it is possible to have a long domain name with special characters.

Patch

$ kubectl patch DNSzone market-example-zone --type='merge' -p='{"spec":{"domain":"extremely-huge-example.market-place.com."}}'

Watch coredns connector status. Should be Active with the newly provisioned domain name.

$ kubectl get dnsconnector coredns -o jsonpath='{.status}' | jq .

{
  "conditions": [
    {
      "lastTransitionTime": "2024-06-05T14:37:49Z",
      "message": "CoreDNS Ready",
      "observedGeneration": 1,
      "reason": "Active",
      "status": "True",
      "type": "Ready"
    }
  ],
  "provisionedZones": [
    {
      "domain": "extremely-huge-example.market-place.com.",
      "name": "market-example-zone",
      "serialNumber": "0605093713"
    }
  ]
}

The DNSZone should be Active too

$ kubectl get dnszones
NAME                     DOMAIN NAME                                RECORD COUNT   LAST CHANGE            CURRENT SERIAL   STATE
market-example-zone      extremely-huge-example.market-place.com.   12             2024-06-05T14:37:22Z   0605093713       Active
no-connector-test-zone   test.com.                                  0              2024-06-05T14:35:02Z   0605093502       Pending

Ensure that the zone mount has been updated

$ kubectl describe deployments coredns | grep Mount -A4
 Mounts:
   /etc/coredns from config-volume (ro)
   /etc/coredns/custom from custom-config-volume (ro)
   /opt/coredns/extremely-huge-example.market-place.com.zone from dnszone-extremely-huge-example-market-place-com (ro,path="extremely-huge-example.market-place.com.zone")

Try to resolve

$ dig +short @192.168.122.10 www.extremely-huge-example.market-place.com.
10.100.100.10

Restore the demo resources.
```
$ kubectl apply -k config/samples/
```

Test DNSZone long ns1 hostname

We will test whether it is possible to use long ns hostname.

Patch

$ kubectl patch DNSzone market-example-zone --type='merge' -p='{"spec":{"primaryNS":{"hostname": "master-0.k3s.test"}}}'

Should be active

$ kubectl get dnszones market-example-zone 
NAME                  DOMAIN NAME          RECORD COUNT   LAST CHANGE            CURRENT SERIAL   STATE
market-example-zone   market.example.com   12             2024-06-13T18:27:37Z   0613182727       Active

Check SOA

$ kubectl describe cm coredns-zone-market-example-zone | grep SOA
@ IN SOA master-0.k3s.test.market.example.com. [email protected]. (

Test bad DNSConnector

To break the DNSConnector, we can enable a not supported CoreDNS plugin. If the DNSConnector degrades, the name resolution may still be functional because CoreDNS is deployed as a Deployment.

IMPORTANT: Unfortunetly if DNSConnector is degraded, the name resolution could be distruptive.

Apply the misconfigured DNSConnector. Wait until timeout and keep an eye on the logs.

cat << EOF | k apply -f -
apiVersion: monkale.monkale.io/v1alpha1
kind: DNSConnector
metadata:
  name: coredns
  namespace: kube-system
spec:
  waitForUpdateTimeout: 13
  corednsCM: 
    name: coredns
    corefileKey: Corefile
  corednsDeployment:
    name: coredns
    type: Deployment
  corednsZoneEnaledPlugins:
  - errors
  - banana
EOF

Check connector's status
```
$ kubectl get dnsconnector coredns -o jsonpath='{.status}' | jq .
```
- The message could be either "coredns is being updated" or "healthcheck failure: coredns is not healthy. Check coredns deployment log".

Inspect logs.

2024-06-02T19:24:39-05:00       ERROR   Reconciler error        {"controller": "dnsconnector", "controllerGroup": "monkale.monkale.io", "controllerKind": "DNSConnector", "DNSConnector": {"name":"coredns","namespace":"kube-system"}, "namespace": "kube-system", "name": "coredns", "reconcileID": "a6b0bf73-7e28-4599-addf-ff1e6a03c1e5", "error": "coredns is not healthy. Check coredns deployment log"}

Check coredns logs

$ kubectl logs  -l 'k8s-app=kube-dns'

[ERROR] Restart failed: /etc/coredns/Corefile:51 - Error during parsing: Unknown directive 'banana'
[ERROR] plugin/reload: Corefile changed but reload failed: starting with listener file descriptors: /etc/coredns/Corefile:51 - Error during parsing: Unknown directive 'banana'
 /etc/coredns/Corefile:52 - Error during parsing: Unknown directive 'banana'
 /etc/coredns/Corefile:52 - Error during parsing: Unknown directive 'banana'

Fix it and ensure that everything functions properly again.

$ cat << EOF | k apply -f -
apiVersion: monkale.monkale.io/v1alpha1
kind: DNSConnector
metadata:
  name: coredns
  namespace: kube-system
spec:
  waitForUpdateTimeout: 13
  corednsCM: 
    name: coredns
    corefileKey: Corefile
  corednsDeployment:
    name: coredns
    type: Deployment
  corednsZoneEnaledPlugins:
  - errors
  - log
EOF

24 KiB Raw Permalink Blame History

Manual Testing Guide for CoreDNS Operator

Prerequisites

Basic Functionality Test

DNSConnector Review

DNSZone Review

Review DNSRecords

Remove record test

Remove a functional zone

Remove an empty zone

Remove DNSConnector

Remove active DNSConnector

Test Exceptions

Restore the demo resources

Test bad DNSRecord creation

Test bad DNSRecord fix

Test bad DNSRecord modification

Test bad DNSRecord deletion

Test bad DNSZone creation

Test bad DNSZone modification

Test DNSZone domain change

Test DNSZone long ns1 hostname

Test bad DNSConnector

24 KiB

Raw Permalink Blame History