hanzo-ml/spark-operator
2
0
Fork 0
mirror of https://github.com/hanzo-ml/spark-operator.git synced 2026-10-30 20:39:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
master
spark-operator/charts/spark-operator-chart/tests/controller/rbac_test.yaml
T
Vincent JanelleGitHubHanzo Dev
 642f7381b1 feat: add per-application driver PodDisruptionBudget (#2949) (#2954) 
* feat: optional PodDisruptionBudget for Spark drivers (#2949)

Refs: https://github.com/kubeflow/spark-operator/issues/2949

Add an opt-in PodDisruptionBudget for the driver pod so a voluntary
eviction (e.g. `kubectl drain`) cannot kill the driver mid-job. The
apiserver's eviction admission handler consults a matching PDB; without
one the eviction is admitted and the operator only learns of it after
the fact via its watch. A PDB is the only lever the operator has here
that does not put it on the eviction hot path.

Gated twice, off by default so upgrades are no-op:
  --enable-driver-pdb              controller flag (default false)
  spec.driverPodDisruptionBudget   per-app *bool (nil/false = no PDB)

A bare *bool suffices: the driver is a singleton, so MinAvailable is
fixed at 1 and a selector struct has nothing to grow into.

Lifecycle (driver_pdb.go):
  - ensureDriverPDB converges PDB existence to spec on every
    Submitted/Running reconcile, so it recovers from operator restart
    with the gate flipped on, live spec edits, and transient API
    failures (retry.RetryOnConflict + controller-runtime backoff).
  - createDriverPDB/deleteDriverPDB read the cache first for
    idempotency; AlreadyExists/NotFound are treated as success.
  - The PDB carries a controller OwnerReference plus
    LabelLaunchedBySparkOperator + LabelSparkAppName, so kube GC reaps
    it on app deletion and the cache informer can filter to owned PDBs.
  - cleanUpOnTermination deletes the PDB on Completed/Failed; the
    terminal-app reconcile re-runs it to reap PDBs orphaned when the
    controller itself was drained mid-transition.
  - deleteSparkResources drops the PDB before the driver pod so an
    in-flight external drain does not block on a stale PDB.

RBAC: the PDB entry in cache.Options.ByObject starts an informer, which
needs list+watch even though the code only Get/Create/Deletes. Granted
get;list;watch;create;delete on poddisruptionbudgets via the kubebuilder
marker (regenerated config/rbac/role.yaml), the Helm chart helper, and
the chart rbac unit test. Missing verbs would loop the informer on
Forbidden and saturate the shared client rate limiter.

Helm: controller.driverPodDisruptionBudget.enable (default false) emits
--enable-driver-pdb=true; ci-values.yaml turns it on for e2e.

Tests: envtest specs cover the gate matrix and the convergence cases
ensureDriverPDB owns (restart-with-gate-on, spec false->true and
true->false on a running app, transient Create failure retried via an
interceptor.Funcs client). E2e submits real apps with the field set and
omitted and asserts PDB creation, GC on deletion, and absence when not
opted in.

Makefile: kind-load-spark-image preloads the Spark runtime image and
e2e-test pins IMAGE_TAG=local with the load targets as prerequisites, so
`make deploy` and `make e2e-test` are self-contained on a local Kind
cluster instead of hitting ImagePullBackOff.

Signed-off-by: Vincent Janelle <[email protected]>

* test: use events.FakeRecorder in driver PDB test

NewReconciler takes an events.EventRecorder, not the legacy
record.EventRecorder. Align driver_pdb_test.go with controller_test.go.

Signed-off-by: Vincent Janelle <[email protected]>

* allow for kustomize to patch pdb value; default to true in ci
Signed-off-by: Vincent Janelle <[email protected]>

---------

Signed-off-by: Vincent Janelle <[email protected]>

Co-authored-by: Hanzo Dev <[email protected]>
2026-06-30 20:28:48 +00:00

628 lines
15 KiB
YAML
Raw Permalink Blame History

#
# Copyright 2024 The Kubeflow authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
suite: Test controller rbac
templates:
- controller/rbac.yaml
release:
name: spark-operator
namespace: spark-operator
tests:
- it: Should not create controller RBAC resources if `controller.rbac.create` is false
set:
controller:
rbac:
create: false
asserts:
- hasDocuments:
count: 0
- it: Should create controller ClusterRole by default
documentIndex: 0
asserts:
- containsDocument:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
name: spark-operator-controller
- it: Should create controller ClusterRoleBinding by default
documentIndex: 1
asserts:
- containsDocument:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
name: spark-operator-controller
- contains:
path: subjects
content:
kind: ServiceAccount
name: spark-operator-controller
namespace: spark-operator
count: 1
- equal:
path: roleRef
value:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: spark-operator-controller
- it: Should add extra annotations to controller ClusterRole if `controller.rbac.annotations` is set
set:
controller:
rbac:
annotations:
key1: value1
key2: value2
asserts:
- equal:
path: metadata.annotations.key1
value: value1
- equal:
path: metadata.annotations.key2
value: value2
- it: Should create role and rolebinding for controller in release namespace
documentIndex: 2
asserts:
- containsDocument:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
name: spark-operator-controller
namespace: spark-operator
- it: Should create role and rolebinding for controller in release namespace
documentIndex: 3
asserts:
- containsDocument:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
name: spark-operator-controller
namespace: spark-operator
- contains:
path: subjects
content:
kind: ServiceAccount
name: spark-operator-controller
namespace: spark-operator
count: 1
- equal:
path: roleRef
value:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: spark-operator-controller
- it: Should create roles and rolebindings for controller in every spark job namespace if `spark.jobNamespaces` is set and does not contain empty string
set:
spark:
jobNamespaces:
- default
- spark
documentIndex: 4
asserts:
- containsDocument:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
name: spark-operator-controller
namespace: default
- it: Should create roles and rolebindings for controller in every spark job namespace if `spark.jobNamespaces` is set and does not contain empty string
set:
spark:
jobNamespaces:
- default
- spark
documentIndex: 5
asserts:
- containsDocument:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
name: spark-operator-controller
namespace: default
- it: Should create roles and rolebindings for controller in every spark job namespace if `spark.jobNamespaces` is set and does not contain empty string
set:
spark:
jobNamespaces:
- default
- spark
documentIndex: 6
asserts:
- containsDocument:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
name: spark-operator-controller
namespace: spark
- it: Should create roles and rolebindings for controller in every spark job namespace if `spark.jobNamespaces` is set and does not contain empty string
set:
spark:
jobNamespaces:
- default
- spark
documentIndex: 7
asserts:
- containsDocument:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
name: spark-operator-controller
namespace: spark
# ---- RBAC rule content assertions (least privilege) ----
# These tests set spark.jobNamespaces to [""] to trigger cluster-wide
# policyRules inclusion in the ClusterRole (documentIndex 0).
- it: Should include controller policyRules in ClusterRole when jobNamespaceSelector is set
set:
spark:
jobNamespaces:
- default
jobNamespaceSelector: "spark=enabled"
documentIndex: 0
asserts:
- contains:
path: rules
content:
apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- list
- watch
- contains:
path: rules
content:
apiGroups:
- sparkoperator.k8s.io
resources:
- sparkapplications
verbs:
- get
- list
- watch
- create
- delete
- it: Should grant pods only get/list/watch/create/update/delete in ClusterRole
set:
spark:
jobNamespaces:
- ""
documentIndex: 0
asserts:
- contains:
path: rules
content:
apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- watch
- create
- update
- delete
- notContains:
path: rules
content:
resources:
- pods
verbs:
- patch
- notContains:
path: rules
content:
resources:
- pods
verbs:
- deletecollection
- it: Should grant configmaps only get/list/watch/create/update/patch in ClusterRole
set:
spark:
jobNamespaces:
- ""
documentIndex: 0
asserts:
- contains:
path: rules
content:
apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- create
- update
- patch
- notContains:
path: rules
content:
resources:
- configmaps
verbs:
- delete
- it: Should not grant any permissions on persistentvolumeclaims in ClusterRole
set:
spark:
jobNamespaces:
- ""
documentIndex: 0
asserts:
- notContains:
path: rules
content:
resources:
- persistentvolumeclaims
- it: Should not grant any permissions on nodes in ClusterRole
set:
spark:
jobNamespaces:
- ""
documentIndex: 0
asserts:
- notContains:
path: rules
content:
resources:
- nodes
- it: Should not grant any permissions on resourcequotas in ClusterRole
set:
spark:
jobNamespaces:
- ""
documentIndex: 0
asserts:
- notContains:
path: rules
content:
resources:
- resourcequotas
- it: Should grant services get/list/watch/create/update/patch/delete in ClusterRole
set:
spark:
jobNamespaces:
- ""
documentIndex: 0
asserts:
- contains:
path: rules
content:
apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- it: Should grant ingresses without list or watch in ClusterRole
set:
spark:
jobNamespaces:
- ""
documentIndex: 0
asserts:
- contains:
path: rules
content:
apiGroups:
- extensions
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- create
- update
- delete
- notContains:
path: rules
content:
resources:
- ingresses
verbs:
- list
- notContains:
path: rules
content:
resources:
- ingresses
verbs:
- watch
- it: Should grant sparkapplications only get/list/watch/create/delete in ClusterRole
set:
spark:
jobNamespaces:
- ""
documentIndex: 0
asserts:
- contains:
path: rules
content:
apiGroups:
- sparkoperator.k8s.io
resources:
- sparkapplications
verbs:
- get
- list
- watch
- create
- delete
- notContains:
path: rules
content:
resources:
- sparkapplications
verbs:
- update
- notContains:
path: rules
content:
resources:
- sparkapplications
verbs:
- patch
- it: Should grant scheduledsparkapplications and sparkconnects only get/list/watch in ClusterRole
set:
spark:
jobNamespaces:
- ""
documentIndex: 0
asserts:
- contains:
path: rules
content:
apiGroups:
- sparkoperator.k8s.io
resources:
- scheduledsparkapplications
- sparkconnects
verbs:
- get
- list
- watch
- notContains:
path: rules
content:
resources:
- scheduledsparkapplications
verbs:
- create
- notContains:
path: rules
content:
resources:
- scheduledsparkapplications
verbs:
- update
- notContains:
path: rules
content:
resources:
- scheduledsparkapplications
verbs:
- patch
- notContains:
path: rules
content:
resources:
- scheduledsparkapplications
verbs:
- delete
- notContains:
path: rules
content:
resources:
- sparkconnects
verbs:
- create
- notContains:
path: rules
content:
resources:
- sparkconnects
verbs:
- update
- notContains:
path: rules
content:
resources:
- sparkconnects
verbs:
- patch
- notContains:
path: rules
content:
resources:
- sparkconnects
verbs:
- delete
- it: Should grant only update on status subresources in ClusterRole
set:
spark:
jobNamespaces:
- ""
documentIndex: 0
asserts:
- contains:
path: rules
content:
apiGroups:
- sparkoperator.k8s.io
resources:
- sparkapplications/status
- sparkconnects/status
- scheduledsparkapplications/status
verbs:
- update
- notContains:
path: rules
content:
resources:
- sparkapplications/status
verbs:
- get
- notContains:
path: rules
content:
resources:
- sparkapplications/status
verbs:
- patch
- notContains:
path: rules
content:
resources:
- scheduledsparkapplications/status
verbs:
- get
- notContains:
path: rules
content:
resources:
- scheduledsparkapplications/status
verbs:
- patch
- notContains:
path: rules
content:
resources:
- sparkconnects/status
verbs:
- get
- notContains:
path: rules
content:
resources:
- sparkconnects/status
verbs:
- patch
- it: Should grant poddisruptionbudgets only get/list/watch/create/delete in ClusterRole
set:
spark:
jobNamespaces:
- ""
documentIndex: 0
asserts:
- contains:
path: rules
content:
apiGroups:
- policy
resources:
- poddisruptionbudgets
verbs:
- get
- list
- watch
- create
- delete
- notContains:
path: rules
content:
resources:
- poddisruptionbudgets
verbs:
- update
- notContains:
path: rules
content:
resources:
- poddisruptionbudgets
verbs:
- patch
- it: Should grant only update on finalizer subresources in ClusterRole
set:
spark:
jobNamespaces:
- ""
documentIndex: 0
asserts:
- contains:
path: rules
content:
apiGroups:
- sparkoperator.k8s.io
resources:
- sparkapplications/finalizers
verbs:
- update
- contains:
path: rules
content:
apiGroups:
- sparkoperator.k8s.io
resources:
- scheduledsparkapplications/finalizers
verbs:
- update
- contains:
path: rules
content:
apiGroups:
- sparkoperator.k8s.io
resources:
- sparkconnects/finalizers
verbs:
- update
- notContains:
path: rules
content:
resources:
- sparkapplications/finalizers
verbs:
- patch
- notContains:
path: rules
content:
resources:
- scheduledsparkapplications/finalizers
verbs:
- patch
- notContains:
path: rules
content:
resources:
- sparkconnects/finalizers
verbs:
- patch
Reference in New Issue View Git Blame Copy Permalink