mirror of
https://github.com/hanzo-ml/spark-operator.git
synced 2026-10-30 20:39:07 +00:00
642f7381b1
* feat: optional PodDisruptionBudget for Spark drivers (#2949) Refs: https://github.com/kubeflow/spark-operator/issues/2949 Add an opt-in PodDisruptionBudget for the driver pod so a voluntary eviction (e.g. `kubectl drain`) cannot kill the driver mid-job. The apiserver's eviction admission handler consults a matching PDB; without one the eviction is admitted and the operator only learns of it after the fact via its watch. A PDB is the only lever the operator has here that does not put it on the eviction hot path. Gated twice, off by default so upgrades are no-op: --enable-driver-pdb controller flag (default false) spec.driverPodDisruptionBudget per-app *bool (nil/false = no PDB) A bare *bool suffices: the driver is a singleton, so MinAvailable is fixed at 1 and a selector struct has nothing to grow into. Lifecycle (driver_pdb.go): - ensureDriverPDB converges PDB existence to spec on every Submitted/Running reconcile, so it recovers from operator restart with the gate flipped on, live spec edits, and transient API failures (retry.RetryOnConflict + controller-runtime backoff). - createDriverPDB/deleteDriverPDB read the cache first for idempotency; AlreadyExists/NotFound are treated as success. - The PDB carries a controller OwnerReference plus LabelLaunchedBySparkOperator + LabelSparkAppName, so kube GC reaps it on app deletion and the cache informer can filter to owned PDBs. - cleanUpOnTermination deletes the PDB on Completed/Failed; the terminal-app reconcile re-runs it to reap PDBs orphaned when the controller itself was drained mid-transition. - deleteSparkResources drops the PDB before the driver pod so an in-flight external drain does not block on a stale PDB. RBAC: the PDB entry in cache.Options.ByObject starts an informer, which needs list+watch even though the code only Get/Create/Deletes. Granted get;list;watch;create;delete on poddisruptionbudgets via the kubebuilder marker (regenerated config/rbac/role.yaml), the Helm chart helper, and the chart rbac unit test. Missing verbs would loop the informer on Forbidden and saturate the shared client rate limiter. Helm: controller.driverPodDisruptionBudget.enable (default false) emits --enable-driver-pdb=true; ci-values.yaml turns it on for e2e. Tests: envtest specs cover the gate matrix and the convergence cases ensureDriverPDB owns (restart-with-gate-on, spec false->true and true->false on a running app, transient Create failure retried via an interceptor.Funcs client). E2e submits real apps with the field set and omitted and asserts PDB creation, GC on deletion, and absence when not opted in. Makefile: kind-load-spark-image preloads the Spark runtime image and e2e-test pins IMAGE_TAG=local with the load targets as prerequisites, so `make deploy` and `make e2e-test` are self-contained on a local Kind cluster instead of hitting ImagePullBackOff. Signed-off-by: Vincent Janelle <[email protected]> * test: use events.FakeRecorder in driver PDB test NewReconciler takes an events.EventRecorder, not the legacy record.EventRecorder. Align driver_pdb_test.go with controller_test.go. Signed-off-by: Vincent Janelle <[email protected]> * allow for kustomize to patch pdb value; default to true in ci Signed-off-by: Vincent Janelle <[email protected]> --------- Signed-off-by: Vincent Janelle <[email protected]> Co-authored-by: Hanzo Dev <[email protected]>
998 lines
32 KiB
YAML
998 lines
32 KiB
YAML
#
|
|
# Copyright 2024 The Kubeflow authors.
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# https://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
#
|
|
|
|
suite: Test controller deployment
|
|
|
|
templates:
|
|
- controller/deployment.yaml
|
|
|
|
release:
|
|
name: spark-operator
|
|
namespace: spark-operator
|
|
|
|
tests:
|
|
- it: Should use the specified image repository if `image.registry`, `image.repository` and `image.tag` are set
|
|
set:
|
|
image:
|
|
registry: test-registry
|
|
repository: test-repository
|
|
tag: test-tag
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.spec.containers[0].image
|
|
value: test-registry/test-repository:test-tag
|
|
|
|
- it: Should use the specified image pull policy if `image.pullPolicy` is set
|
|
set:
|
|
image:
|
|
pullPolicy: Always
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.spec.containers[*].imagePullPolicy
|
|
value: Always
|
|
|
|
- it: Should set replicas if `controller.replicas` is set
|
|
set:
|
|
controller:
|
|
replicas: 10
|
|
asserts:
|
|
- equal:
|
|
path: spec.replicas
|
|
value: 10
|
|
|
|
- it: Should set replicas if `controller.replicas` is set
|
|
set:
|
|
controller:
|
|
replicas: 0
|
|
asserts:
|
|
- equal:
|
|
path: spec.replicas
|
|
value: 0
|
|
|
|
- it: Should set revisionHistoryLimit to the default value
|
|
asserts:
|
|
- equal:
|
|
path: spec.revisionHistoryLimit
|
|
value: 10
|
|
|
|
- it: Should set revisionHistoryLimit if `controller.revisionHistoryLimit` is set
|
|
set:
|
|
controller:
|
|
revisionHistoryLimit: 2
|
|
asserts:
|
|
- equal:
|
|
path: spec.revisionHistoryLimit
|
|
value: 2
|
|
|
|
- it: Should add pod labels if `controller.labels` is set
|
|
set:
|
|
controller:
|
|
labels:
|
|
key1: value1
|
|
key2: value2
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.metadata.labels.key1
|
|
value: value1
|
|
- equal:
|
|
path: spec.template.metadata.labels.key2
|
|
value: value2
|
|
|
|
- it: Should add prometheus annotations if `metrics.enable` is true
|
|
set:
|
|
prometheus:
|
|
metrics:
|
|
enable: true
|
|
port: 10254
|
|
endpoint: /metrics
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.metadata.annotations["prometheus.io/scrape"]
|
|
value: "true"
|
|
- equal:
|
|
path: spec.template.metadata.annotations["prometheus.io/port"]
|
|
value: "10254"
|
|
- equal:
|
|
path: spec.template.metadata.annotations["prometheus.io/path"]
|
|
value: /metrics
|
|
|
|
- it: Should add pod annotations if `controller.annotations` is set
|
|
set:
|
|
controller:
|
|
annotations:
|
|
key1: value1
|
|
key2: value2
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.metadata.annotations.key1
|
|
value: value1
|
|
- equal:
|
|
path: spec.template.metadata.annotations.key2
|
|
value: value2
|
|
|
|
- it: Should add deployment annotations if `controller.deploymentAnnotations` is set
|
|
set:
|
|
controller:
|
|
deploymentAnnotations:
|
|
key1: value1
|
|
key2: value2
|
|
asserts:
|
|
- equal:
|
|
path: metadata.annotations.key1
|
|
value: value1
|
|
- equal:
|
|
path: metadata.annotations.key2
|
|
value: value2
|
|
|
|
- it: Should contain `--zap-log-level` arg if `controller.logLevel` is set
|
|
set:
|
|
controller:
|
|
logLevel: debug
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --zap-log-level=debug
|
|
|
|
- it: Should contain `--namespaces` arg if `spark.jobNamespaces` is set
|
|
set:
|
|
spark:
|
|
jobNamespaces:
|
|
- ns1
|
|
- ns2
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --namespaces=ns1,ns2
|
|
|
|
- it: Should set namespaces to all namespaces (`""`) if `spark.jobNamespaces` contains empty string
|
|
set:
|
|
spark:
|
|
jobNamespaces:
|
|
- ""
|
|
- default
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --namespaces=""
|
|
|
|
- it: Should contain `--controller-threads` arg if `controller.workers` is set
|
|
set:
|
|
controller:
|
|
workers: 30
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --controller-threads=30
|
|
|
|
- it: Should contain `--enable-ui-service` arg if `controller.uiService.enable` is set to `true`
|
|
set:
|
|
controller:
|
|
uiService:
|
|
enable: true
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --enable-ui-service=true
|
|
|
|
- it: Should contain `--ingress-url-format` arg if `controller.uiIngress.enable` is set to `true` and `controller.uiIngress.urlFormat` is set
|
|
set:
|
|
controller:
|
|
uiService:
|
|
enable: true
|
|
uiIngress:
|
|
enable: true
|
|
urlFormat: "{{$appName}}.example.com/{{$appNamespace}}/{{$appName}}"
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --ingress-url-format={{$appName}}.example.com/{{$appNamespace}}/{{$appName}}
|
|
|
|
- it: Should contain `--ingress-class-name` arg if `controller.uiIngress.enable` is set to `true` and `controller.uiIngress.ingressClassName` is set
|
|
set:
|
|
controller:
|
|
uiService:
|
|
enable: true
|
|
uiIngress:
|
|
enable: true
|
|
ingressClassName: nginx
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --ingress-class-name=nginx
|
|
|
|
- it: Should contain `--ingress-tls` arg if `controller.uiIngress.enable` is set to `true` and `controller.uiIngress.tls` is set
|
|
set:
|
|
controller:
|
|
uiService:
|
|
enable: true
|
|
uiIngress:
|
|
enable: true
|
|
tls:
|
|
- hosts:
|
|
- "*.test.com"
|
|
secretName: test-secret
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: '--ingress-tls=[{"hosts":["*.test.com"],"secretName":"test-secret"}]'
|
|
|
|
- it: Should contain `--ingress-annotations` arg if `controller.uiIngress.enable` is set to `true` and `controller.uiIngress.annotations` is set
|
|
set:
|
|
controller:
|
|
uiService:
|
|
enable: true
|
|
uiIngress:
|
|
enable: true
|
|
annotations:
|
|
cert-manager.io/cluster-issuer: "letsencrypt"
|
|
kubernetes.io/ingress.class: nginx
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: '--ingress-annotations={"cert-manager.io/cluster-issuer":"letsencrypt","kubernetes.io/ingress.class":"nginx"}'
|
|
|
|
- it: Should contain `--enable-batch-scheduler` arg if `controller.batchScheduler.enable` is `true`
|
|
set:
|
|
controller:
|
|
batchScheduler:
|
|
enable: true
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --enable-batch-scheduler=true
|
|
|
|
- it: Should not contain `--enable-driver-pdb` arg by default
|
|
asserts:
|
|
- notContains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --enable-driver-pdb=true
|
|
|
|
- it: Should contain `--enable-driver-pdb` arg if `controller.driverPodDisruptionBudget.enable` is `true`
|
|
set:
|
|
controller:
|
|
driverPodDisruptionBudget:
|
|
enable: true
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --enable-driver-pdb=true
|
|
|
|
- it: Should contain `--default-batch-scheduler` arg if `controller.batchScheduler.default` is set
|
|
set:
|
|
controller:
|
|
batchScheduler:
|
|
enable: true
|
|
default: yunikorn
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --default-batch-scheduler=yunikorn
|
|
|
|
- it: Should contain `--enable-metrics` arg if `prometheus.metrics.enable` is set to `true`
|
|
set:
|
|
prometheus:
|
|
metrics:
|
|
enable: true
|
|
port: 12345
|
|
portName: test-port
|
|
endpoint: /test-endpoint
|
|
prefix: test-prefix
|
|
jobSubmitLatencyBuckets: "1,2,4,8,16"
|
|
jobStartLatencyBuckets: "180,360,420,690"
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --enable-metrics=true
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --metrics-bind-address=:12345
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --metrics-endpoint=/test-endpoint
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --metrics-prefix=test-prefix
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --metrics-labels=app_type
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --metrics-job-submit-latency-buckets=1,2,4,8,16
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --metrics-job-start-latency-buckets=180,360,420,690
|
|
|
|
- it: Should set metrics-labels arg if `prometheus.metrics.labels` is set to not empty
|
|
set:
|
|
prometheus:
|
|
metrics:
|
|
enable: true
|
|
port: 12345
|
|
portName: test-port
|
|
endpoint: /test-endpoint
|
|
prefix: test-prefix
|
|
jobSubmitLatencyBuckets: "1,2,4,8,16"
|
|
jobStartLatencyBuckets: "180,360,420,690"
|
|
labels: test1,test2
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --enable-metrics=true
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --metrics-bind-address=:12345
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --metrics-endpoint=/test-endpoint
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --metrics-prefix=test-prefix
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --metrics-labels=test1,test2
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --metrics-job-start-latency-buckets=180,360,420,690
|
|
|
|
- it: Should enable leader election by default
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --leader-election=true
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --leader-election-lock-name=spark-operator-controller-lock
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --leader-election-lock-namespace=spark-operator
|
|
|
|
- it: Should disable leader election if `controller.leaderElection.enable` is set to `false`
|
|
set:
|
|
controller:
|
|
leaderElection:
|
|
enable: false
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --leader-election=false
|
|
|
|
- it: Should add metric ports if `prometheus.metrics.enable` is true
|
|
set:
|
|
prometheus:
|
|
metrics:
|
|
enable: true
|
|
port: 10254
|
|
portName: metrics
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[0].ports
|
|
content:
|
|
name: metrics
|
|
containerPort: 10254
|
|
count: 1
|
|
|
|
- it: Should add environment variables if `controller.env` is set
|
|
set:
|
|
controller:
|
|
env:
|
|
- name: ENV_NAME_1
|
|
value: ENV_VALUE_1
|
|
- name: ENV_NAME_2
|
|
valueFrom:
|
|
configMapKeyRef:
|
|
name: test-configmap
|
|
key: test-key
|
|
optional: false
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[0].env
|
|
content:
|
|
name: ENV_NAME_1
|
|
value: ENV_VALUE_1
|
|
- contains:
|
|
path: spec.template.spec.containers[0].env
|
|
content:
|
|
name: ENV_NAME_2
|
|
valueFrom:
|
|
configMapKeyRef:
|
|
name: test-configmap
|
|
key: test-key
|
|
optional: false
|
|
|
|
- it: Should add environment variable sources if `controller.envFrom` is set
|
|
set:
|
|
controller:
|
|
envFrom:
|
|
- configMapRef:
|
|
name: test-configmap
|
|
optional: false
|
|
- secretRef:
|
|
name: test-secret
|
|
optional: false
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[0].envFrom
|
|
content:
|
|
configMapRef:
|
|
name: test-configmap
|
|
optional: false
|
|
- contains:
|
|
path: spec.template.spec.containers[0].envFrom
|
|
content:
|
|
secretRef:
|
|
name: test-secret
|
|
optional: false
|
|
|
|
- it: Should add volume mounts if `controller.volumeMounts` is set
|
|
set:
|
|
controller:
|
|
volumeMounts:
|
|
- name: volume1
|
|
mountPath: /volume1
|
|
- name: volume2
|
|
mountPath: /volume2
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[0].volumeMounts
|
|
content:
|
|
name: volume1
|
|
mountPath: /volume1
|
|
- contains:
|
|
path: spec.template.spec.containers[0].volumeMounts
|
|
content:
|
|
name: volume2
|
|
mountPath: /volume2
|
|
|
|
- it: Should add resources if `controller.resources` is set
|
|
set:
|
|
controller:
|
|
resources:
|
|
requests:
|
|
memory: "64Mi"
|
|
cpu: "250m"
|
|
limits:
|
|
memory: "128Mi"
|
|
cpu: "500m"
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.spec.containers[0].resources
|
|
value:
|
|
requests:
|
|
memory: "64Mi"
|
|
cpu: "250m"
|
|
limits:
|
|
memory: "128Mi"
|
|
cpu: "500m"
|
|
|
|
- it: Should add container securityContext if `controller.securityContext` is set
|
|
set:
|
|
controller:
|
|
securityContext:
|
|
readOnlyRootFilesystem: true
|
|
runAsUser: 1000
|
|
runAsGroup: 2000
|
|
fsGroup: 3000
|
|
allowPrivilegeEscalation: false
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
runAsNonRoot: true
|
|
privileged: false
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.spec.containers[0].securityContext.readOnlyRootFilesystem
|
|
value: true
|
|
- equal:
|
|
path: spec.template.spec.containers[0].securityContext.runAsUser
|
|
value: 1000
|
|
- equal:
|
|
path: spec.template.spec.containers[0].securityContext.runAsGroup
|
|
value: 2000
|
|
- equal:
|
|
path: spec.template.spec.containers[0].securityContext.fsGroup
|
|
value: 3000
|
|
- equal:
|
|
path: spec.template.spec.containers[0].securityContext.allowPrivilegeEscalation
|
|
value: false
|
|
- equal:
|
|
path: spec.template.spec.containers[0].securityContext.capabilities
|
|
value:
|
|
drop:
|
|
- ALL
|
|
- equal:
|
|
path: spec.template.spec.containers[0].securityContext.runAsNonRoot
|
|
value: true
|
|
- equal:
|
|
path: spec.template.spec.containers[0].securityContext.privileged
|
|
value: false
|
|
|
|
- it: Should add sidecars if `controller.sidecars` is set
|
|
set:
|
|
controller:
|
|
sidecars:
|
|
- name: sidecar1
|
|
image: sidecar-image1
|
|
- name: sidecar2
|
|
image: sidecar-image2
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers
|
|
content:
|
|
name: sidecar1
|
|
image: sidecar-image1
|
|
- contains:
|
|
path: spec.template.spec.containers
|
|
content:
|
|
name: sidecar2
|
|
image: sidecar-image2
|
|
|
|
- it: Should add secrets if `image.pullSecrets` is set
|
|
set:
|
|
image:
|
|
pullSecrets:
|
|
- name: test-secret1
|
|
- name: test-secret2
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.spec.imagePullSecrets[0].name
|
|
value: test-secret1
|
|
- equal:
|
|
path: spec.template.spec.imagePullSecrets[1].name
|
|
value: test-secret2
|
|
|
|
- it: Should add volumes if `controller.volumes` is set
|
|
set:
|
|
controller:
|
|
volumes:
|
|
- name: volume1
|
|
emptyDir: {}
|
|
- name: volume2
|
|
emptyDir: {}
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.volumes
|
|
content:
|
|
name: volume1
|
|
emptyDir: {}
|
|
count: 1
|
|
- contains:
|
|
path: spec.template.spec.volumes
|
|
content:
|
|
name: volume2
|
|
emptyDir: {}
|
|
count: 1
|
|
|
|
- it: Should add nodeSelector if `controller.nodeSelector` is set
|
|
set:
|
|
controller:
|
|
nodeSelector:
|
|
key1: value1
|
|
key2: value2
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.spec.nodeSelector.key1
|
|
value: value1
|
|
- equal:
|
|
path: spec.template.spec.nodeSelector.key2
|
|
value: value2
|
|
|
|
- it: Should add affinity if `controller.affinity` is set
|
|
set:
|
|
controller:
|
|
affinity:
|
|
nodeAffinity:
|
|
requiredDuringSchedulingIgnoredDuringExecution:
|
|
nodeSelectorTerms:
|
|
- matchExpressions:
|
|
- key: topology.kubernetes.io/zone
|
|
operator: In
|
|
values:
|
|
- antarctica-east1
|
|
- antarctica-west1
|
|
preferredDuringSchedulingIgnoredDuringExecution:
|
|
- weight: 1
|
|
preference:
|
|
matchExpressions:
|
|
- key: another-node-label-key
|
|
operator: In
|
|
values:
|
|
- another-node-label-value
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.spec.affinity
|
|
value:
|
|
nodeAffinity:
|
|
requiredDuringSchedulingIgnoredDuringExecution:
|
|
nodeSelectorTerms:
|
|
- matchExpressions:
|
|
- key: topology.kubernetes.io/zone
|
|
operator: In
|
|
values:
|
|
- antarctica-east1
|
|
- antarctica-west1
|
|
preferredDuringSchedulingIgnoredDuringExecution:
|
|
- weight: 1
|
|
preference:
|
|
matchExpressions:
|
|
- key: another-node-label-key
|
|
operator: In
|
|
values:
|
|
- another-node-label-value
|
|
|
|
- it: Should add tolerations if `controller.tolerations` is set
|
|
set:
|
|
controller:
|
|
tolerations:
|
|
- key: key1
|
|
operator: Equal
|
|
value: value1
|
|
effect: NoSchedule
|
|
- key: key2
|
|
operator: Exists
|
|
effect: NoSchedule
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.spec.tolerations
|
|
value:
|
|
- key: key1
|
|
operator: Equal
|
|
value: value1
|
|
effect: NoSchedule
|
|
- key: key2
|
|
operator: Exists
|
|
effect: NoSchedule
|
|
|
|
- it: Should add priorityClassName if `controller.priorityClassName` is set
|
|
set:
|
|
controller:
|
|
priorityClassName: test-priority-class
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.spec.priorityClassName
|
|
value: test-priority-class
|
|
|
|
- it: Should add pod securityContext if `controller.podSecurityContext` is set
|
|
set:
|
|
controller:
|
|
podSecurityContext:
|
|
runAsUser: 1000
|
|
runAsGroup: 2000
|
|
fsGroup: 3000
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.spec.securityContext
|
|
value:
|
|
runAsUser: 1000
|
|
runAsGroup: 2000
|
|
fsGroup: 3000
|
|
|
|
- it: Should not contain topologySpreadConstraints if `controller.topologySpreadConstraints` is not set
|
|
set:
|
|
controller:
|
|
topologySpreadConstraints: []
|
|
asserts:
|
|
- notExists:
|
|
path: spec.template.spec.topologySpreadConstraints
|
|
|
|
- it: Should add topologySpreadConstraints if `controller.topologySpreadConstraints` is set and `controller.replicas` is greater than 1
|
|
set:
|
|
controller:
|
|
replicas: 2
|
|
topologySpreadConstraints:
|
|
- maxSkew: 1
|
|
topologyKey: topology.kubernetes.io/zone
|
|
whenUnsatisfiable: ScheduleAnyway
|
|
- maxSkew: 1
|
|
topologyKey: kubernetes.io/hostname
|
|
whenUnsatisfiable: DoNotSchedule
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.spec.topologySpreadConstraints
|
|
value:
|
|
- labelSelector:
|
|
matchLabels:
|
|
app.kubernetes.io/component: controller
|
|
app.kubernetes.io/instance: spark-operator
|
|
app.kubernetes.io/name: spark-operator
|
|
maxSkew: 1
|
|
topologyKey: topology.kubernetes.io/zone
|
|
whenUnsatisfiable: ScheduleAnyway
|
|
- labelSelector:
|
|
matchLabels:
|
|
app.kubernetes.io/component: controller
|
|
app.kubernetes.io/instance: spark-operator
|
|
app.kubernetes.io/name: spark-operator
|
|
maxSkew: 1
|
|
topologyKey: kubernetes.io/hostname
|
|
whenUnsatisfiable: DoNotSchedule
|
|
|
|
- it: Should fail if `controller.topologySpreadConstraints` is set and `controller.replicas` is not greater than 1
|
|
set:
|
|
controller:
|
|
replicas: 1
|
|
topologySpreadConstraints:
|
|
- maxSkew: 1
|
|
topologyKey: topology.kubernetes.io/zone
|
|
whenUnsatisfiable: ScheduleAnyway
|
|
- maxSkew: 1
|
|
topologyKey: kubernetes.io/hostname
|
|
whenUnsatisfiable: DoNotSchedule
|
|
asserts:
|
|
- failedTemplate:
|
|
errorMessage: "controller.replicas must be greater than 1 to enable topology spread constraints for controller pods"
|
|
|
|
- it: Should include hostUsers when kubernetes version >= 1.30 and hostUsers is true
|
|
capabilities:
|
|
majorVersion: 1
|
|
minorVersion: 30
|
|
set:
|
|
controller:
|
|
hostUsers: true
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.spec.hostUsers
|
|
value: true
|
|
|
|
- it: Should include hostUsers when kubernetes version >= 1.30 and hostUsers is false
|
|
capabilities:
|
|
majorVersion: 1
|
|
minorVersion: 34
|
|
set:
|
|
controller:
|
|
hostUsers: false
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.spec.hostUsers
|
|
value: false
|
|
|
|
- it: Should not contain hostUsers when kubernetes version < 1.29 and hostUsers is set
|
|
capabilities:
|
|
majorVersion: 1
|
|
minorVersion: 29
|
|
set:
|
|
controller:
|
|
hostUsers: true
|
|
asserts:
|
|
- notExists:
|
|
path: spec.template.spec.hostUsers
|
|
|
|
- it: Should not include hostUsers when kubernetes version >= 1.30 but value not set
|
|
capabilities:
|
|
majorVersion: 1
|
|
minorVersion: 34
|
|
asserts:
|
|
- notExists:
|
|
path: spec.template.spec.hostUsers
|
|
|
|
- it: Should should NOT include hostUsers when kubernetes version >= 1.30 but value is string
|
|
capabilities:
|
|
majorVersion: 1
|
|
minorVersion: 34
|
|
set:
|
|
controller:
|
|
hostUsers: "true"
|
|
asserts:
|
|
- notExists:
|
|
path: spec.template.spec.hostUsers
|
|
|
|
- it: Should not include hostUsers when kubernetes version >= 1.30 but value is null
|
|
capabilities:
|
|
majorVersion: 1
|
|
minorVersion: 34
|
|
set:
|
|
controller:
|
|
hostUsers: null
|
|
asserts:
|
|
- notExists:
|
|
path: spec.template.spec.hostUsers
|
|
|
|
- it: Should contain `--pprof-bind-address` arg if `controller.pprof.enable` is set to `true`
|
|
set:
|
|
controller:
|
|
pprof:
|
|
enable: true
|
|
port: 12345
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --pprof-bind-address=:12345
|
|
|
|
- it: Should add pprof ports if `controller.pprof.enable` is set to `true`
|
|
set:
|
|
controller:
|
|
pprof:
|
|
enable: true
|
|
port: 12345
|
|
portName: pprof-test
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].ports
|
|
content:
|
|
name: pprof-test
|
|
containerPort: 12345
|
|
count: 1
|
|
|
|
- it: Should contain `--workqueue-ratelimiter-max-delay` arg if `controller.workqueueRateLimiter.maxDelay.enable` is set to `true`
|
|
set:
|
|
controller:
|
|
workqueueRateLimiter:
|
|
bucketQPS: 1
|
|
bucketSize: 2
|
|
maxDelay:
|
|
enable: true
|
|
duration: 3h
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --workqueue-ratelimiter-bucket-qps=1
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --workqueue-ratelimiter-bucket-size=2
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --workqueue-ratelimiter-max-delay=3h
|
|
|
|
- it: Should contain `--workqueue-ratelimiter-max-delay` arg if `controller.workqueueRateLimiter.maxDelay.enable` is set to `true`
|
|
set:
|
|
controller:
|
|
maxDelay:
|
|
enable: false
|
|
duration: 1h
|
|
asserts:
|
|
- notContains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --workqueue-ratelimiter-max-delay=1h
|
|
|
|
- it: Should contain `driver-pod-creation-grace-period` arg if `controller.driverPodCreationGracePeriod` is set
|
|
set:
|
|
controller:
|
|
driverPodCreationGracePeriod: 30s
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --driver-pod-creation-grace-period=30s
|
|
|
|
- it: Should contain `--max-tracked-executor-per-app` arg if `controller.maxTrackedExecutorPerApp` is set
|
|
set:
|
|
controller:
|
|
maxTrackedExecutorPerApp: 123
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --max-tracked-executor-per-app=123
|
|
|
|
|
|
- it: Should add leader election parameters if `controller.leaderElection.leaseDuration`, `controller.leaderElection.renewDeadline` and `controller.leaderElection.retryPeriod` are set.
|
|
set:
|
|
controller:
|
|
leaderElection:
|
|
enable: true
|
|
leaseDuration: 120s
|
|
renewDeadline: 30s
|
|
retryPeriod: 15s
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --leader-election-lease-duration=120s
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --leader-election-renew-deadline=30s
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --leader-election-retry-period=15s
|
|
|
|
- it: Should not contain `--feature-gates` arg if `controller.featureGates` is not set
|
|
asserts:
|
|
- notContains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
pattern: --feature-gates=.*
|
|
|
|
- it: Should contain `--feature-gates` arg with single feature gate if `controller.featureGates` is set
|
|
set:
|
|
controller:
|
|
featureGates:
|
|
- name: PartialRestart
|
|
enabled: true
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --feature-gates=PartialRestart=true
|
|
|
|
- it: Should contain `--feature-gates` arg with multiple feature gates if `controller.featureGates` is set
|
|
set:
|
|
controller:
|
|
featureGates:
|
|
- name: PartialRestart
|
|
enabled: true
|
|
- name: AnotherFeature
|
|
enabled: false
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --feature-gates=PartialRestart=true,AnotherFeature=false
|
|
|
|
- it: Should render submitter args when RestSubmitter feature gate is enabled
|
|
set:
|
|
controller:
|
|
featureGates:
|
|
- name: RestSubmitter
|
|
enabled: true
|
|
submitter:
|
|
serviceUrl: http://my-submitter:8080/api/v1/spark-submit
|
|
requestTimeout: 3m
|
|
startupTimeout: 10m
|
|
maxRetries: 5
|
|
initialBackoff: 5s
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --submitter-service-url=http://my-submitter:8080/api/v1/spark-submit
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --submitter-request-timeout=3m
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --submitter-startup-timeout=10m
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --submitter-max-retries=5
|
|
- contains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --submitter-initial-backoff=5s
|
|
|
|
- it: Should not render submitter args when RestSubmitter feature gate is disabled
|
|
set:
|
|
controller:
|
|
featureGates:
|
|
- name: RestSubmitter
|
|
enabled: false
|
|
asserts:
|
|
- notContains:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].args
|
|
content: --submitter-service-url=http://spark-operator-submitter-svc:8080/api/v1/spark-submit
|
|
|
|
- it: Should render startup probe when RestSubmitter feature gate is enabled
|
|
set:
|
|
controller:
|
|
featureGates:
|
|
- name: RestSubmitter
|
|
enabled: true
|
|
submitter:
|
|
startupProbe:
|
|
periodSeconds: 10
|
|
failureThreshold: 30
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].startupProbe.httpGet.path
|
|
value: /healthz
|
|
- equal:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].startupProbe.periodSeconds
|
|
value: 10
|
|
- equal:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].startupProbe.failureThreshold
|
|
value: 30
|
|
|
|
- it: Should not render startup probe when RestSubmitter feature gate is disabled
|
|
set:
|
|
controller:
|
|
featureGates:
|
|
- name: RestSubmitter
|
|
enabled: false
|
|
asserts:
|
|
- isNull:
|
|
path: spec.template.spec.containers[?(@.name=="spark-operator-controller")].startupProbe
|
|
|