Files
1df0651f91 [Feature] Add NetworkIsolation support to RayClusters (#4638)
* [Feature] Add NetworkIsolation support to RayClusters

Co-authored-by: Pat O'Connor <[email protected]>
Co-authored-by: Bryan Keane <[email protected]>

* add feature gate reg to integration tests + removed redundant if err

Signed-off-by: Pat O'Connor <[email protected]>

* separate function for head and base ingress rules

Signed-off-by: Pat O'Connor <[email protected]>

* logger updates as per review

Signed-off-by: Pat O'Connor <[email protected]>

* adjust DNS egress rule to allow all port 53 egress

Signed-off-by: Pat O'Connor <[email protected]>

* resolve helm chart CI failure

Signed-off-by: Pat O'Connor <[email protected]>

* restrict KubeRay rule via NamespaceSelector

Signed-off-by: Pat O'Connor <[email protected]>

* check if update is necessary on NP via DeepEqual

Signed-off-by: Pat O'Connor <[email protected]>

* prevent same-name NP mod + add namespace fallback to default

Signed-off-by: Pat O'Connor <[email protected]>

* add missing rayStartParams ports to custom ports example

Signed-off-by: Pat O'Connor <[email protected]>

* emit warning when existing NP conflicts with new one via name + fix test

Signed-off-by: Pat O'Connor <[email protected]>

* remove redundant samples + add new samples

Signed-off-by: Pat O'Connor <[email protected]>

* remove permissive pod selector rule + test updates

Signed-off-by: Pat O'Connor <[email protected]>

* prop labels to jobsubmitter for networkpolicy rule

Signed-off-by: Pat O'Connor <[email protected]>

* Add opt-in ALLOW_ALL_RAYJOB_SUBMITTERS env var for broad submitter ingress on standalone RayClusters

Signed-off-by: Pat O'Connor <[email protected]>

* updated API and config to fix CI failures

Signed-off-by: Pat O'Connor <[email protected]>

* Moved label prop to getSummitterTemplate + review changes

Signed-off-by: Pat O'Connor <[email protected]>

* Review feedback: failed deletion event + reconcile on dupe + nits

Signed-off-by: Pat O'Connor <[email protected]>

* use rayStartParams for ports + review nits

Signed-off-by: Pat O'Connor <[email protected]>

* test fix + convert constants to CamelCase

Signed-off-by: Pat O'Connor <[email protected]>

* helm updates + check if RayCluster is externally managed

Signed-off-by: Pat O'Connor <[email protected]>

* API updates per review + autoscaling egress rule

Signed-off-by: Pat O'Connor <[email protected]>

* Use standard lib for CIDR + use logConstructor consistently

Signed-off-by: Pat O'Connor <[email protected]>

* Use EndpointSlice IPs for API server egress rule

Signed-off-by: Pat O'Connor <[email protected]>

* Deduplicates ports + peers across EndpointSlices in egress rules

Signed-off-by: Pat O'Connor <[email protected]>

* Remove operator namespace fallback and prop error if namespace not found

Signed-off-by: Pat O'Connor <[email protected]>

* Remove patch verb for networkpolicies

Signed-off-by: Pat O'Connor <[email protected]>

* Remove client port rule for operator

Signed-off-by: Pat O'Connor <[email protected]>

* Add reconcileConcurrency and GenerationChangedPredicate to NetworkPolicy controller setup

Signed-off-by: Pat O'Connor <[email protected]>

* solve 3 points, close to merge

Signed-off-by: Future-Outlier <[email protected]>

* helper

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* CI failure fixes after DNS logic removal

Signed-off-by: Pat O'Connor <[email protected]>

* Revert "helper"

This reverts commit 26c4053ae5775517f60a01cc4fb9457265d832c4.

* Skip networkpolicy integration test when missing namespace

Signed-off-by: Pat O'Connor <[email protected]>

* Normalize defaulted NetworkPolicy ports before comparing

Signed-off-by: Pat O'Connor <[email protected]>

* Drop network policy integration tests pending followup e2e

Signed-off-by: Pat O'Connor <[email protected]>

* Remove operator namespace ingress rule

Signed-off-by: Pat O'Connor <[email protected]>

* networkpolicy: separate head/worker custom rules in API Replace the flat ingressRules/egressRules on NetworkIsolationConfig with per-role head/worker sub-structs.Since the API has not shipped yet, this avoids locking in a shared-only design that would require a backward-compatible migration later. Head and workers have fundamentally different security profiles — head needs external access (dashboard, submitter, Prometheus) while workers typically need outbound access (S3, model registries). A shared-only API forces users to over-permit workers just to allow access to the head. New API shape:   networkIsolation:     mode: DenyAll     head:       ingressRules: [...]       egressRules: [...]     worker:       ingressRules: [...]       egressRules: [...]

Co-authored-by: Cursor <[email protected]>

* NetworkIsolation API change cleanup

Signed-off-by: Pat O'Connor <[email protected]>

* Unit tests to ensure custom policies don't leak between head and worker

Signed-off-by: Pat O'Connor <[email protected]>

* Sample comment indendation and removal of operator rule from API docs

Signed-off-by: Pat O'Connor <[email protected]>

* Indent update to pass linter

Signed-off-by: Pat O'Connor <[email protected]>

* Additional review nits - comments etc

Signed-off-by: Pat O'Connor <[email protected]>

* Remove redundant comments in controller code

Signed-off-by: Pat O'Connor <[email protected]>

---------

Signed-off-by: Pat O'Connor <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Co-authored-by: Laura Fitzgerald <[email protected]>
Co-authored-by: Bryan Keane <[email protected]>
Co-authored-by: Future-Outlier <[email protected]>
Co-authored-by: Cursor <[email protected]>
2026-06-23 10:03:32 -04:00

100 lines
3.3 KiB
Go

package features
import (
"testing"
"github.com/go-logr/logr"
"k8s.io/apimachinery/pkg/util/runtime"
utilfeature "k8s.io/apiserver/pkg/util/feature"
"k8s.io/component-base/featuregate"
featuregatetesting "k8s.io/component-base/featuregate/testing"
)
const (
// owner: @rueian @kevin85421 @andrewsykim
// rep: https://github.com/ray-project/enhancements/pull/54
// alpha: v1.2
// beta: v1.3
//
// Enables new conditions in RayCluster status
RayClusterStatusConditions featuregate.Feature = "RayClusterStatusConditions"
// owner: @andrewsykim @seanlaii
// rep: N/A
// alpha: v1.3
// beta: v1.6
// Enables new deletion policy API in RayJob
RayJobDeletionPolicy featuregate.Feature = "RayJobDeletionPolicy"
// owner: @aaronliang @ryanaoleary
// rep: N/A
// alpha: v1.5
// beta: v1.6
//
// Enables multi-host worker indexing
RayMultiHostIndexing featuregate.Feature = "RayMultiHostIndexing"
// owner: @ryanaoleary
// rep: https://github.com/ray-project/enhancements/pull/58
// alpha: v1.5
//
// Enabled NewClusterWithIncrementalUpgrade type for RayService zero-downtime upgrades.
RayServiceIncrementalUpgrade featuregate.Feature = "RayServiceIncrementalUpgrade"
// owner: @machichima
// rep: N/A
// alpha: v1.6
//
// Enables RayCronJob controller for scheduled RayJob execution.
RayCronJob featuregate.Feature = "RayCronJob"
// owner: @justinyeh1995
// rep: N/A
// alpha: v1.7
//
// Enables per-container restart policy for SidecarMode submitter to handle transient failures.
// Requires Kubernetes v1.35+ since it supports ContainerRestartPolicy by default starting in v1.35 and Ray v2.54.0+.
SidecarSubmitterRestart featuregate.Feature = "SidecarSubmitterRestart"
// owner: @chipspeak @kryanbeane
// rep: N/A
// alpha: v1.7
//
// Enables NetworkPolicy-based network isolation for RayClusters (spec.networkIsolation).
RayClusterNetworkIsolation featuregate.Feature = "RayClusterNetworkIsolation"
)
func init() {
runtime.Must(utilfeature.DefaultMutableFeatureGate.Add(defaultFeatureGates))
}
var defaultFeatureGates = map[featuregate.Feature]featuregate.FeatureSpec{
RayClusterStatusConditions: {Default: true, PreRelease: featuregate.Beta},
RayJobDeletionPolicy: {Default: true, PreRelease: featuregate.Beta},
RayMultiHostIndexing: {Default: true, PreRelease: featuregate.Beta},
RayServiceIncrementalUpgrade: {Default: false, PreRelease: featuregate.Alpha},
RayCronJob: {Default: false, PreRelease: featuregate.Alpha},
SidecarSubmitterRestart: {Default: false, PreRelease: featuregate.Alpha},
RayClusterNetworkIsolation: {Default: false, PreRelease: featuregate.Alpha},
}
// SetFeatureGateDuringTest is a helper method to override feature gates in tests.
func SetFeatureGateDuringTest(tb testing.TB, f featuregate.Feature, value bool) {
featuregatetesting.SetFeatureGateDuringTest(tb, utilfeature.DefaultFeatureGate, f, value)
}
// Enabled is helper for `utilfeature.DefaultFeatureGate.Enabled()`
func Enabled(f featuregate.Feature) bool {
return utilfeature.DefaultFeatureGate.Enabled(f)
}
func LogFeatureGates(log logr.Logger) {
features := make(map[featuregate.Feature]bool, len(defaultFeatureGates))
for f := range utilfeature.DefaultMutableFeatureGate.GetAll() {
if _, ok := defaultFeatureGates[f]; ok {
features[f] = Enabled(f)
}
}
log.Info("Loaded feature gates", "featureGates", features)
}