Files
1df0651f91 [Feature] Add NetworkIsolation support to RayClusters (#4638)
* [Feature] Add NetworkIsolation support to RayClusters

Co-authored-by: Pat O'Connor <[email protected]>
Co-authored-by: Bryan Keane <[email protected]>

* add feature gate reg to integration tests + removed redundant if err

Signed-off-by: Pat O'Connor <[email protected]>

* separate function for head and base ingress rules

Signed-off-by: Pat O'Connor <[email protected]>

* logger updates as per review

Signed-off-by: Pat O'Connor <[email protected]>

* adjust DNS egress rule to allow all port 53 egress

Signed-off-by: Pat O'Connor <[email protected]>

* resolve helm chart CI failure

Signed-off-by: Pat O'Connor <[email protected]>

* restrict KubeRay rule via NamespaceSelector

Signed-off-by: Pat O'Connor <[email protected]>

* check if update is necessary on NP via DeepEqual

Signed-off-by: Pat O'Connor <[email protected]>

* prevent same-name NP mod + add namespace fallback to default

Signed-off-by: Pat O'Connor <[email protected]>

* add missing rayStartParams ports to custom ports example

Signed-off-by: Pat O'Connor <[email protected]>

* emit warning when existing NP conflicts with new one via name + fix test

Signed-off-by: Pat O'Connor <[email protected]>

* remove redundant samples + add new samples

Signed-off-by: Pat O'Connor <[email protected]>

* remove permissive pod selector rule + test updates

Signed-off-by: Pat O'Connor <[email protected]>

* prop labels to jobsubmitter for networkpolicy rule

Signed-off-by: Pat O'Connor <[email protected]>

* Add opt-in ALLOW_ALL_RAYJOB_SUBMITTERS env var for broad submitter ingress on standalone RayClusters

Signed-off-by: Pat O'Connor <[email protected]>

* updated API and config to fix CI failures

Signed-off-by: Pat O'Connor <[email protected]>

* Moved label prop to getSummitterTemplate + review changes

Signed-off-by: Pat O'Connor <[email protected]>

* Review feedback: failed deletion event + reconcile on dupe + nits

Signed-off-by: Pat O'Connor <[email protected]>

* use rayStartParams for ports + review nits

Signed-off-by: Pat O'Connor <[email protected]>

* test fix + convert constants to CamelCase

Signed-off-by: Pat O'Connor <[email protected]>

* helm updates + check if RayCluster is externally managed

Signed-off-by: Pat O'Connor <[email protected]>

* API updates per review + autoscaling egress rule

Signed-off-by: Pat O'Connor <[email protected]>

* Use standard lib for CIDR + use logConstructor consistently

Signed-off-by: Pat O'Connor <[email protected]>

* Use EndpointSlice IPs for API server egress rule

Signed-off-by: Pat O'Connor <[email protected]>

* Deduplicates ports + peers across EndpointSlices in egress rules

Signed-off-by: Pat O'Connor <[email protected]>

* Remove operator namespace fallback and prop error if namespace not found

Signed-off-by: Pat O'Connor <[email protected]>

* Remove patch verb for networkpolicies

Signed-off-by: Pat O'Connor <[email protected]>

* Remove client port rule for operator

Signed-off-by: Pat O'Connor <[email protected]>

* Add reconcileConcurrency and GenerationChangedPredicate to NetworkPolicy controller setup

Signed-off-by: Pat O'Connor <[email protected]>

* solve 3 points, close to merge

Signed-off-by: Future-Outlier <[email protected]>

* helper

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* CI failure fixes after DNS logic removal

Signed-off-by: Pat O'Connor <[email protected]>

* Revert "helper"

This reverts commit 26c4053ae5775517f60a01cc4fb9457265d832c4.

* Skip networkpolicy integration test when missing namespace

Signed-off-by: Pat O'Connor <[email protected]>

* Normalize defaulted NetworkPolicy ports before comparing

Signed-off-by: Pat O'Connor <[email protected]>

* Drop network policy integration tests pending followup e2e

Signed-off-by: Pat O'Connor <[email protected]>

* Remove operator namespace ingress rule

Signed-off-by: Pat O'Connor <[email protected]>

* networkpolicy: separate head/worker custom rules in API Replace the flat ingressRules/egressRules on NetworkIsolationConfig with per-role head/worker sub-structs.Since the API has not shipped yet, this avoids locking in a shared-only design that would require a backward-compatible migration later. Head and workers have fundamentally different security profiles — head needs external access (dashboard, submitter, Prometheus) while workers typically need outbound access (S3, model registries). A shared-only API forces users to over-permit workers just to allow access to the head. New API shape:   networkIsolation:     mode: DenyAll     head:       ingressRules: [...]       egressRules: [...]     worker:       ingressRules: [...]       egressRules: [...]

Co-authored-by: Cursor <[email protected]>

* NetworkIsolation API change cleanup

Signed-off-by: Pat O'Connor <[email protected]>

* Unit tests to ensure custom policies don't leak between head and worker

Signed-off-by: Pat O'Connor <[email protected]>

* Sample comment indendation and removal of operator rule from API docs

Signed-off-by: Pat O'Connor <[email protected]>

* Indent update to pass linter

Signed-off-by: Pat O'Connor <[email protected]>

* Additional review nits - comments etc

Signed-off-by: Pat O'Connor <[email protected]>

* Remove redundant comments in controller code

Signed-off-by: Pat O'Connor <[email protected]>

---------

Signed-off-by: Pat O'Connor <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Co-authored-by: Laura Fitzgerald <[email protected]>
Co-authored-by: Bryan Keane <[email protected]>
Co-authored-by: Future-Outlier <[email protected]>
Co-authored-by: Cursor <[email protected]>
2026-06-23 10:03:32 -04:00

385 lines
16 KiB
Go

package main
import (
"flag"
"fmt"
"os"
"strings"
"github.com/go-logr/zapr"
routev1 "github.com/openshift/api/route/v1"
"go.uber.org/zap"
"go.uber.org/zap/zapcore"
"gopkg.in/natefinch/lumberjack.v2"
batchv1 "k8s.io/api/batch/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/serializer"
utilruntime "k8s.io/apimachinery/pkg/util/runtime"
utilfeature "k8s.io/apiserver/pkg/util/feature"
clientgoscheme "k8s.io/client-go/kubernetes/scheme"
"k8s.io/klog/v2"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/cache"
"sigs.k8s.io/controller-runtime/pkg/healthz"
k8szap "sigs.k8s.io/controller-runtime/pkg/log/zap"
ctrlmetrics "sigs.k8s.io/controller-runtime/pkg/metrics"
metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
gwv1 "sigs.k8s.io/gateway-api/apis/v1"
configapi "github.com/ray-project/kuberay/ray-operator/apis/config/v1alpha1"
rayv1 "github.com/ray-project/kuberay/ray-operator/apis/ray/v1"
"github.com/ray-project/kuberay/ray-operator/controllers/ray"
"github.com/ray-project/kuberay/ray-operator/controllers/ray/batchscheduler"
"github.com/ray-project/kuberay/ray-operator/controllers/ray/metrics"
"github.com/ray-project/kuberay/ray-operator/controllers/ray/utils"
"github.com/ray-project/kuberay/ray-operator/internal/managercache"
"github.com/ray-project/kuberay/ray-operator/pkg/features"
webhooks "github.com/ray-project/kuberay/ray-operator/pkg/webhooks/v1"
)
var (
scheme = runtime.NewScheme()
setupLog = ctrl.Log.WithName("setup")
userAgent = fmt.Sprintf("kuberay-operator/%s", utils.KUBERAY_VERSION)
)
func init() {
utilruntime.Must(clientgoscheme.AddToScheme(scheme))
utilruntime.Must(rayv1.AddToScheme(scheme))
utilruntime.Must(routev1.Install(scheme))
utilruntime.Must(batchv1.AddToScheme(scheme))
utilruntime.Must(configapi.AddToScheme(scheme))
// +kubebuilder:scaffold:scheme
}
func main() {
var metricsAddr string
var enableLeaderElection bool
var leaderElectionNamespace string
var probeAddr string
var reconcileConcurrency int
var watchNamespace string
var forcedClusterUpgrade bool
var logFile string
var logFileEncoder string
var logStdoutEncoder string
var useKubernetesProxy bool
var configFile string
var featureGates string
var enableBatchScheduler bool
var batchScheduler string
var enableMetrics bool
var qps float64
var burst int
// TODO: remove flag-based config once Configuration API graduates to v1.
flag.StringVar(&metricsAddr, "metrics-addr", configapi.DefaultMetricsAddr, "The address the metric endpoint binds to.")
flag.StringVar(&probeAddr, "health-probe-bind-address", configapi.DefaultProbeAddr, "The address the probe endpoint binds to.")
flag.BoolVar(&enableLeaderElection, "enable-leader-election", configapi.DefaultEnableLeaderElection,
"Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.")
flag.StringVar(&leaderElectionNamespace, "leader-election-namespace", "",
"Namespace where the leader election resource lives. Defaults to the pod namespace if not set.")
flag.IntVar(&reconcileConcurrency, "reconcile-concurrency", configapi.DefaultReconcileConcurrency, "max concurrency for reconciling")
flag.StringVar(
&watchNamespace,
"watch-namespace",
"",
"Specify a list of namespaces to watch for custom resources, separated by commas. If left empty, all namespaces will be watched.")
flag.BoolVar(&forcedClusterUpgrade, "forced-cluster-upgrade", false,
"(Deprecated) Forced cluster upgrade flag")
flag.StringVar(&logFile, "log-file-path", "",
"Synchronize logs to local file")
flag.StringVar(&logFileEncoder, "log-file-encoder", "json",
"Encoder to use for log file. Valid values are 'json' and 'console'. Defaults to 'json'")
flag.StringVar(&logStdoutEncoder, "log-stdout-encoder", "json",
"Encoder to use for logging stdout. Valid values are 'json' and 'console'. Defaults to 'json'")
flag.BoolVar(&enableBatchScheduler, "enable-batch-scheduler", false,
"(Deprecated) Enable batch scheduler. Currently is volcano, which supports gang scheduler policy. Please use --batch-scheduler instead.")
flag.StringVar(&batchScheduler, "batch-scheduler", "",
"Batch scheduler name, supported values are volcano, yunikorn, kai-scheduler.")
flag.StringVar(&configFile, "config", "", "Path to structured config file. Flags are ignored if config file is set.")
flag.BoolVar(&useKubernetesProxy, "use-kubernetes-proxy", false,
"Use Kubernetes proxy subresource when connecting to the Ray Head node.")
flag.StringVar(&featureGates, "feature-gates", "", "A set of key=value pairs that describe feature gates. E.g. FeatureOne=true,FeatureTwo=false,...")
flag.BoolVar(&enableMetrics, "enable-metrics", false, "Enable the emission of control plane metrics.")
flag.Float64Var(&qps, "qps", float64(configapi.DefaultQPS), "The QPS value for the client communicating with the Kubernetes API server.")
flag.IntVar(&burst, "burst", configapi.DefaultBurst, "The maximum burst for throttling requests from this client to the Kubernetes API server.")
opts := k8szap.Options{
TimeEncoder: zapcore.ISO8601TimeEncoder,
}
opts.BindFlags(flag.CommandLine)
flag.Parse()
var config configapi.Configuration
if configFile != "" {
var err error
configData, err := os.ReadFile(configFile)
exitOnError(err, "failed to read config file")
config, err = decodeConfig(configData, scheme)
exitOnError(err, "failed to decode config file")
} else {
config.MetricsAddr = metricsAddr
config.ProbeAddr = probeAddr
config.EnableLeaderElection = &enableLeaderElection
config.LeaderElectionNamespace = leaderElectionNamespace
config.ReconcileConcurrency = reconcileConcurrency
config.WatchNamespace = watchNamespace
config.LogFile = logFile
config.LogFileEncoder = logFileEncoder
config.LogStdoutEncoder = logStdoutEncoder
config.EnableBatchScheduler = enableBatchScheduler
config.BatchScheduler = batchScheduler
config.UseKubernetesProxy = useKubernetesProxy
config.DeleteRayJobAfterJobFinishes = os.Getenv(utils.DELETE_RAYJOB_CR_AFTER_JOB_FINISHES) == "true"
config.EnableMetrics = enableMetrics
config.QPS = &qps
config.Burst = &burst
}
stdoutEncoder, err := newLogEncoder(logStdoutEncoder)
exitOnError(err, "failed to create log encoder for stdout")
opts.Encoder = stdoutEncoder
if config.LogFile != "" {
fileWriter := &lumberjack.Logger{
Filename: config.LogFile,
MaxSize: 500, // megabytes
MaxBackups: 10, // files
MaxAge: 30, // days
}
fileEncoder, err := newLogEncoder(logFileEncoder)
exitOnError(err, "failed to create log encoder for file")
k8sLogger := k8szap.NewRaw(k8szap.UseFlagOptions(&opts))
zapOpts := append(opts.ZapOpts, zap.AddCallerSkip(1))
combineLogger := zap.New(zapcore.NewTee(
k8sLogger.Core(),
zapcore.NewCore(fileEncoder, zapcore.AddSync(fileWriter), zap.InfoLevel),
)).WithOptions(zapOpts...)
combineLoggerR := zapr.NewLogger(combineLogger)
ctrl.SetLogger(combineLoggerR)
// By default, the log from kubernetes/client-go is not json format.
// This will apply the logger to kubernetes/client-go and change it to json format.
klog.SetLogger(combineLoggerR)
} else {
k8sLogger := k8szap.New(k8szap.UseFlagOptions(&opts))
ctrl.SetLogger(k8sLogger)
// By default, the log from kubernetes/client-go is not json format.
// This will apply the logger to kubernetes/client-go and change it to json format.
klog.SetLogger(k8sLogger)
}
if forcedClusterUpgrade {
setupLog.Info("Deprecated feature flag forced-cluster-upgrade is enabled, which has no effect.")
}
// validate the batch scheduler configs,
// exit with error if the configs is invalid.
if err := configapi.ValidateBatchSchedulerConfig(setupLog, config); err != nil {
exitOnError(err, "batch scheduler configs validation failed")
}
if err := utilfeature.DefaultMutableFeatureGate.Set(featureGates); err != nil {
exitOnError(err, "Unable to set flag gates for known features")
}
features.LogFeatureGates(setupLog)
if features.Enabled(features.RayServiceIncrementalUpgrade) {
utilruntime.Must(gwv1.Install(scheme))
}
// Although ContainerRestartPolicy is introduced in Kubernetes v1.34 as an alpha feature, which requires
// manually enabling the ContainerRestartRules feature gate, it becomes beta in v1.35.
// We require v1.35+ to avoid asking users to opt in to a separate K8s feature gate.
// NOTE: this checks the API server version only. Please make sure worker nodes are also v1.35+,
// as kubelets are allowed to be up to 3 minor versions older per the Kubernetes version skew policy.
// If ContainerRestartRules is not enabled on a kubelet, the per-container restart policy on the
// submitter container will be ignored. In this scenario, if the submitter container exit on failure,
// a 30-second timeout will be applied by the operator and could incorrectly mark the RayJob as Failed
// even if the Ray job is still running.
if features.Enabled(features.SidecarSubmitterRestart) {
serverVersion, err := utils.GetKubernetesVersion()
if err != nil {
exitOnError(err, "SidecarSubmitterRestart feature gate enabled but unable to detect K8s version. Feature requires K8s 1.35+.")
}
isAtLeast, err := utils.IsK8sVersionAtLeast(serverVersion, 1, 35, 0)
if err != nil {
exitOnError(err, "Failed to compare K8s version.")
}
if !isAtLeast {
exitOnError(fmt.Errorf("current version %s is below 1.35", serverVersion.GitVersion),
"SidecarSubmitterRestart feature gate requires K8s 1.35+")
}
}
// Manager options
options := ctrl.Options{
Cache: cache.Options{
DefaultNamespaces: map[string]cache.Config{},
},
Scheme: scheme,
Metrics: metricsserver.Options{
BindAddress: config.MetricsAddr,
},
HealthProbeBindAddress: config.ProbeAddr,
LeaderElection: *config.EnableLeaderElection,
LeaderElectionID: "ray-operator-leader",
LeaderElectionNamespace: config.LeaderElectionNamespace,
}
// Manager Cache
// Set the informers label selectors to narrow the scope of the resources being watched and cached.
// This improves the scalability of the system, both for KubeRay itself by reducing the size of the
// informers cache, and for the API server / etcd, by reducing the number of watch events.
// For example, KubeRay is only interested in:
// - the batch Jobs it creates when reconciling RayJobs (app.kubernetes.io/created-by=kuberay-operator), and
// - Ray-managed Pods (ray.io/node-type in head|worker|redis-cleanup).
// These labels are provided to the manager cache as selectors for Job and Pod resources.
selectorsByObject, err := managercache.K8sControllerRuntimeCacheSelectors()
exitOnError(err, "unable to build manager cache ByObject")
options.Cache.ByObject = selectorsByObject
if watchNamespaces := strings.Split(config.WatchNamespace, ","); len(watchNamespaces) == 1 { // It is not possible for len(watchNamespaces) == 0 to be true. The length of `strings.Split("", ",")` is still 1.
if watchNamespaces[0] == "" {
setupLog.Info("Flag watchNamespace is not set. Watch custom resources in all namespaces.")
} else {
setupLog.Info("Only watch custom resources in the namespace.", "namespace", watchNamespaces[0])
options.Cache.DefaultNamespaces[watchNamespaces[0]] = cache.Config{}
}
} else {
setupLog.Info("Only watch custom resources in multiple namespaces.", "namespaces", watchNamespaces)
for _, namespace := range watchNamespaces {
options.Cache.DefaultNamespaces[namespace] = cache.Config{}
}
}
setupLog.Info("Setup manager")
restConfig := ctrl.GetConfigOrDie()
restConfig.UserAgent = userAgent
restConfig.QPS = float32(*config.QPS)
restConfig.Burst = *config.Burst
mgr, err := ctrl.NewManager(restConfig, options)
exitOnError(err, "unable to start manager")
ctx := ctrl.SetupSignalHandler()
var rayClusterMetricsManager *metrics.RayClusterMetricsManager
var rayJobMetricsManager *metrics.RayJobMetricsManager
var rayServiceMetricsManager *metrics.RayServiceMetricsManager
var batchSchedulerManager *batchscheduler.SchedulerManager
if config.EnableMetrics {
mgrClient := mgr.GetClient()
rayClusterMetricsManager = metrics.NewRayClusterMetricsManager(ctx, mgrClient)
rayJobMetricsManager = metrics.NewRayJobMetricsManager(ctx, mgrClient)
rayServiceMetricsManager = metrics.NewRayServiceMetricsManager(ctx, mgrClient)
ctrlmetrics.Registry.MustRegister(
rayClusterMetricsManager,
rayJobMetricsManager,
rayServiceMetricsManager,
)
}
batchSchedulerManager, err = batchscheduler.NewSchedulerManager(ctx, config, restConfig, mgr.GetClient())
exitOnError(err, "unable to create batch scheduler manager")
batchSchedulerManager.AddToScheme(mgr.GetScheme())
// TODO: Remove isOpenShift and related reconciler options once Gateway API support is mature
// and Route-based dashboard access is no longer needed.
// See: https://github.com/ray-project/kuberay/pull/4365#issuecomment-4143407845
isOpenShift, err := utils.IsOpenShiftCluster(restConfig)
exitOnError(err, "unable to detect cluster type (OpenShift vs Kubernetes)")
rayClusterOptions := ray.RayClusterReconcilerOptions{
HeadSidecarContainers: config.HeadSidecarContainers,
WorkerSidecarContainers: config.WorkerSidecarContainers,
IsOpenShift: isOpenShift,
UseIngressOnOpenShift: utils.ShouldUseIngressOnOpenShift(),
RayClusterMetricsManager: rayClusterMetricsManager,
BatchSchedulerManager: batchSchedulerManager,
DefaultContainerEnvs: config.DefaultContainerEnvs,
}
exitOnError(ray.NewReconciler(mgr, rayClusterOptions).SetupWithManager(mgr, config.ReconcileConcurrency),
"unable to create controller", "controller", "RayCluster")
exitOnError(ray.NewRayServiceReconciler(ctx, mgr, config).SetupWithManager(mgr, config.ReconcileConcurrency),
"unable to create controller", "controller", "RayService")
rayJobOptions := ray.RayJobReconcilerOptions{
RayJobMetricsManager: rayJobMetricsManager,
BatchSchedulerManager: batchSchedulerManager,
}
exitOnError(ray.NewRayJobReconciler(ctx, mgr, rayJobOptions, config).SetupWithManager(mgr, config.ReconcileConcurrency),
"unable to create controller", "controller", "RayJob")
if os.Getenv("ENABLE_WEBHOOKS") == "true" {
exitOnError(webhooks.SetupRayClusterWebhookWithManager(mgr),
"unable to create webhook", "webhook", "RayCluster")
}
if features.Enabled(features.RayCronJob) {
setupLog.Info("RayCronJob feature gate is enabled, starting RayCronJob controller")
exitOnError(ray.NewRayCronJobReconciler(mgr).SetupWithManager(mgr, config.ReconcileConcurrency),
"unable to create controller", "controller", "RayCronJob")
} else {
setupLog.Info("RayCronJob feature gate is disabled, skipping RayCronJob controller setup")
}
if features.Enabled(features.RayClusterNetworkIsolation) {
setupLog.Info("RayClusterNetworkIsolation feature gate is enabled, starting NetworkPolicy controller")
networkPolicyController, err := ray.NewNetworkPolicyController(mgr)
exitOnError(err, "unable to create controller", "controller", "NetworkPolicy")
exitOnError(networkPolicyController.SetupWithManager(mgr, config.ReconcileConcurrency),
"unable to setup controller", "controller", "NetworkPolicy")
} else {
setupLog.Info("RayClusterNetworkIsolation feature gate is disabled, skipping NetworkPolicy controller setup")
}
// +kubebuilder:scaffold:builder
exitOnError(mgr.AddHealthzCheck("healthz", healthz.Ping), "unable to set up health check")
exitOnError(mgr.AddReadyzCheck("readyz", healthz.Ping), "unable to set up ready check")
setupLog.Info("starting manager")
exitOnError(mgr.Start(ctx), "problem running manager")
}
func exitOnError(err error, msg string, keysAndValues ...any) {
if err != nil {
setupLog.Error(err, msg, keysAndValues...)
os.Exit(1)
}
}
// decodeConfig decodes raw config data and returns the Configuration type.
func decodeConfig(configData []byte, scheme *runtime.Scheme) (configapi.Configuration, error) {
cfg := configapi.Configuration{}
codecs := serializer.NewCodecFactory(scheme)
if err := runtime.DecodeInto(codecs.UniversalDecoder(), configData, &cfg); err != nil {
return cfg, err
}
scheme.Default(&cfg)
return cfg, nil
}
// newLogEncoder returns a zapcore.Encoder based on the encoder type ('json' or 'console')
func newLogEncoder(encoderType string) (zapcore.Encoder, error) {
pe := zap.NewProductionEncoderConfig()
pe.EncodeTime = zapcore.ISO8601TimeEncoder
if encoderType == "json" || encoderType == "" {
return zapcore.NewJSONEncoder(pe), nil
}
if encoderType == "console" {
return zapcore.NewConsoleEncoder(pe), nil
}
return nil, fmt.Errorf("invalid encoder %q (must be 'json' or 'console')", encoderType)
}