mirror of
https://github.com/hanzo-ml/spark-operator.git
synced 2026-10-30 20:39:07 +00:00
642f7381b1
* feat: optional PodDisruptionBudget for Spark drivers (#2949) Refs: https://github.com/kubeflow/spark-operator/issues/2949 Add an opt-in PodDisruptionBudget for the driver pod so a voluntary eviction (e.g. `kubectl drain`) cannot kill the driver mid-job. The apiserver's eviction admission handler consults a matching PDB; without one the eviction is admitted and the operator only learns of it after the fact via its watch. A PDB is the only lever the operator has here that does not put it on the eviction hot path. Gated twice, off by default so upgrades are no-op: --enable-driver-pdb controller flag (default false) spec.driverPodDisruptionBudget per-app *bool (nil/false = no PDB) A bare *bool suffices: the driver is a singleton, so MinAvailable is fixed at 1 and a selector struct has nothing to grow into. Lifecycle (driver_pdb.go): - ensureDriverPDB converges PDB existence to spec on every Submitted/Running reconcile, so it recovers from operator restart with the gate flipped on, live spec edits, and transient API failures (retry.RetryOnConflict + controller-runtime backoff). - createDriverPDB/deleteDriverPDB read the cache first for idempotency; AlreadyExists/NotFound are treated as success. - The PDB carries a controller OwnerReference plus LabelLaunchedBySparkOperator + LabelSparkAppName, so kube GC reaps it on app deletion and the cache informer can filter to owned PDBs. - cleanUpOnTermination deletes the PDB on Completed/Failed; the terminal-app reconcile re-runs it to reap PDBs orphaned when the controller itself was drained mid-transition. - deleteSparkResources drops the PDB before the driver pod so an in-flight external drain does not block on a stale PDB. RBAC: the PDB entry in cache.Options.ByObject starts an informer, which needs list+watch even though the code only Get/Create/Deletes. Granted get;list;watch;create;delete on poddisruptionbudgets via the kubebuilder marker (regenerated config/rbac/role.yaml), the Helm chart helper, and the chart rbac unit test. Missing verbs would loop the informer on Forbidden and saturate the shared client rate limiter. Helm: controller.driverPodDisruptionBudget.enable (default false) emits --enable-driver-pdb=true; ci-values.yaml turns it on for e2e. Tests: envtest specs cover the gate matrix and the convergence cases ensureDriverPDB owns (restart-with-gate-on, spec false->true and true->false on a running app, transient Create failure retried via an interceptor.Funcs client). E2e submits real apps with the field set and omitted and asserts PDB creation, GC on deletion, and absence when not opted in. Makefile: kind-load-spark-image preloads the Spark runtime image and e2e-test pins IMAGE_TAG=local with the load targets as prerequisites, so `make deploy` and `make e2e-test` are self-contained on a local Kind cluster instead of hitting ImagePullBackOff. Signed-off-by: Vincent Janelle <[email protected]> * test: use events.FakeRecorder in driver PDB test NewReconciler takes an events.EventRecorder, not the legacy record.EventRecorder. Align driver_pdb_test.go with controller_test.go. Signed-off-by: Vincent Janelle <[email protected]> * allow for kustomize to patch pdb value; default to true in ci Signed-off-by: Vincent Janelle <[email protected]> --------- Signed-off-by: Vincent Janelle <[email protected]> Co-authored-by: Hanzo Dev <[email protected]>
230 lines
4.8 KiB
Smarty
230 lines
4.8 KiB
Smarty
{{/*
|
|
Copyright 2024 The Kubeflow authors.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License");
|
|
you may not use this file except in compliance with the License.
|
|
You may obtain a copy of the License at
|
|
|
|
https://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
See the License for the specific language governing permissions and
|
|
limitations under the License.
|
|
*/}}
|
|
|
|
{{/*
|
|
Create the name of controller component
|
|
*/}}
|
|
{{- define "spark-operator.controller.name" -}}
|
|
{{- include "spark-operator.fullname" . }}-controller
|
|
{{- end -}}
|
|
|
|
{{/*
|
|
Common labels for the controller
|
|
*/}}
|
|
{{- define "spark-operator.controller.labels" -}}
|
|
{{ include "spark-operator.labels" . }}
|
|
app.kubernetes.io/component: controller
|
|
{{- end -}}
|
|
|
|
{{/*
|
|
Selector labels for the controller
|
|
*/}}
|
|
{{- define "spark-operator.controller.selectorLabels" -}}
|
|
{{ include "spark-operator.selectorLabels" . }}
|
|
app.kubernetes.io/component: controller
|
|
{{- end -}}
|
|
|
|
{{/*
|
|
Create the name of the service account to be used by the controller
|
|
*/}}
|
|
{{- define "spark-operator.controller.serviceAccountName" -}}
|
|
{{- if .Values.controller.serviceAccount.create -}}
|
|
{{ .Values.controller.serviceAccount.name | default (include "spark-operator.controller.name" .) }}
|
|
{{- else -}}
|
|
{{ .Values.controller.serviceAccount.name | default "default" }}
|
|
{{- end -}}
|
|
{{- end -}}
|
|
|
|
{{/*
|
|
Create the name of the cluster role to be used by the controller
|
|
*/}}
|
|
{{- define "spark-operator.controller.clusterRoleName" -}}
|
|
{{ include "spark-operator.controller.name" . }}
|
|
{{- end }}
|
|
|
|
{{/*
|
|
Create the name of the cluster role binding to be used by the controller
|
|
*/}}
|
|
{{- define "spark-operator.controller.clusterRoleBindingName" -}}
|
|
{{ include "spark-operator.controller.clusterRoleName" . }}
|
|
{{- end }}
|
|
|
|
{{/*
|
|
Create the name of the role to be used by the controller
|
|
*/}}
|
|
{{- define "spark-operator.controller.roleName" -}}
|
|
{{ include "spark-operator.controller.name" . }}
|
|
{{- end }}
|
|
|
|
{{/*
|
|
Create the name of the role binding to be used by the controller
|
|
*/}}
|
|
{{- define "spark-operator.controller.roleBindingName" -}}
|
|
{{ include "spark-operator.controller.roleName" . }}
|
|
{{- end }}
|
|
|
|
{{/*
|
|
Create the name of the deployment to be used by controller
|
|
*/}}
|
|
{{- define "spark-operator.controller.deploymentName" -}}
|
|
{{ include "spark-operator.controller.name" . }}
|
|
{{- end -}}
|
|
|
|
{{/*
|
|
Create the name of the lease resource to be used by leader election
|
|
*/}}
|
|
{{- define "spark-operator.controller.leaderElectionName" -}}
|
|
{{ include "spark-operator.controller.name" . }}-lock
|
|
{{- end -}}
|
|
|
|
{{/*
|
|
Create the name of the pod disruption budget to be used by controller
|
|
*/}}
|
|
{{- define "spark-operator.controller.podDisruptionBudgetName" -}}
|
|
{{ include "spark-operator.controller.name" . }}-pdb
|
|
{{- end -}}
|
|
|
|
{{/*
|
|
Create the name of the service used by controller
|
|
*/}}
|
|
{{- define "spark-operator.controller.serviceName" -}}
|
|
{{ include "spark-operator.controller.name" . }}-svc
|
|
{{- end -}}
|
|
|
|
{{/*
|
|
Create the role policy rules for the controller in every Spark job namespace
|
|
*/}}
|
|
{{- define "spark-operator.controller.policyRules" -}}
|
|
- apiGroups:
|
|
- ""
|
|
resources:
|
|
- pods
|
|
verbs:
|
|
- get
|
|
- list
|
|
- watch
|
|
- create
|
|
- update
|
|
- delete
|
|
- apiGroups:
|
|
- ""
|
|
resources:
|
|
- configmaps
|
|
verbs:
|
|
- get
|
|
- list
|
|
- watch
|
|
- create
|
|
- update
|
|
- patch
|
|
- apiGroups:
|
|
- ""
|
|
resources:
|
|
- services
|
|
verbs:
|
|
- get
|
|
- list
|
|
- watch
|
|
- create
|
|
- update
|
|
- patch
|
|
- delete
|
|
- apiGroups:
|
|
- ""
|
|
resources:
|
|
- events
|
|
verbs:
|
|
- create
|
|
- update
|
|
- patch
|
|
- apiGroups:
|
|
- extensions
|
|
- networking.k8s.io
|
|
resources:
|
|
- ingresses
|
|
verbs:
|
|
- get
|
|
- create
|
|
- update
|
|
- delete
|
|
- apiGroups:
|
|
- sparkoperator.k8s.io
|
|
resources:
|
|
- sparkapplications
|
|
verbs:
|
|
- get
|
|
- list
|
|
- watch
|
|
- create
|
|
- delete
|
|
- apiGroups:
|
|
- sparkoperator.k8s.io
|
|
resources:
|
|
- scheduledsparkapplications
|
|
- sparkconnects
|
|
verbs:
|
|
- get
|
|
- list
|
|
- watch
|
|
- apiGroups:
|
|
- sparkoperator.k8s.io
|
|
resources:
|
|
- sparkapplications/status
|
|
- sparkconnects/status
|
|
- scheduledsparkapplications/status
|
|
verbs:
|
|
- update
|
|
- apiGroups:
|
|
- sparkoperator.k8s.io
|
|
resources:
|
|
- sparkapplications/finalizers
|
|
verbs:
|
|
- update
|
|
- apiGroups:
|
|
- sparkoperator.k8s.io
|
|
resources:
|
|
- scheduledsparkapplications/finalizers
|
|
verbs:
|
|
- update
|
|
- apiGroups:
|
|
- sparkoperator.k8s.io
|
|
resources:
|
|
- sparkconnects/finalizers
|
|
verbs:
|
|
- update
|
|
{{- if .Values.controller.batchScheduler.enable }}
|
|
{{/* required for the `volcano` batch scheduler */}}
|
|
- apiGroups:
|
|
- scheduling.incubator.k8s.io
|
|
- scheduling.sigs.dev
|
|
- scheduling.volcano.sh
|
|
resources:
|
|
- podgroups
|
|
verbs:
|
|
- "*"
|
|
{{- end }}
|
|
- apiGroups:
|
|
- policy
|
|
resources:
|
|
- poddisruptionbudgets
|
|
verbs:
|
|
- get
|
|
- list
|
|
- watch
|
|
- create
|
|
- delete
|
|
{{- end -}}
|