This PR - Cleans up and updates sample configs and test files - Increments Ray versions to the latest Ray release (2.1.0) - Memory resource requests are increased in some sample configs. - The Ray head pod has a high risk of running out of memory if it is allocated less than 2Gb memory, so that's fixed. Signed-off-by: Dmitri Gekhtman <[email protected]>
6.6 KiB
Development
This section walks through how to build and test the operator in a running Kubernetes cluster.
Requirements
| software | version | link |
|---|---|---|
| kubectl | v1.21.0+ | download |
| go | v1.17 | download |
| docker | 19.03+ | download |
The instructions assume you have access to a running Kubernetes cluster via kubectl. If you want to test locally, consider using Minikube.
Setup on Kind
For a local kind environment setup, you can follow the Jupyter Notebook example: KubeRay-on-kind.
Use go v1.17
Currently, Kuberay does not support go v1.16 (#568) or go v1.18 (#518). Hence, we strongly recommend you to use go v1.17. The following commands can help you switch to go v1.17.6.
go install golang.org/dl/go1.17.6@latest
go1.17.6 download
export GOROOT=$(go1.17.6 env GOROOT)
export PATH="$GOROOT/bin:$PATH"
Development
IDE Setup (VS Code)
- Step 1: Install the VS Code Go extension.
- Step 2: Import the KubeRay workspace configuration by using the file
kuberay.code-workspacein the root of the KubeRay git repo:- "File" -> "Open Workspace from File" -> "kuberay.code-workspace"
Setting up workspace configuration is required because KubeRay contains multiple Go modules. See the VS Code Go documentation for details.
Build the source code
make build
Building the container image
Once building is finished, push it to DockerHub so it can be pulled down and run in the Kubernetes cluster.
IMG=kuberay/operator:nightly make docker-build
Note: replace
kuberay/operator:nightlywith your own registry, image name and tag.
Running the tests
make test
example results:
✗ make test
...
go fmt ./...
go vet ./...
...
setting up env vars
? github.com/ray-project/kuberay/ray-operator [no test files]
ok github.com/ray-project/kuberay/ray-operator/api/v1alpha1 0.023s coverage: 0.9% of statements
ok github.com/ray-project/kuberay/ray-operator/controllers 9.587s coverage: 66.8% of statements
ok github.com/ray-project/kuberay/ray-operator/controllers/common 0.016s coverage: 75.6% of statements
ok github.com/ray-project/kuberay/ray-operator/controllers/utils 0.015s coverage: 31.4% of statements
Manually test new image in running cluster
Build and apply the CRD:
make install
Deploy the manifests and controller
IMG=kuberay/operator:nightly make deploy
Note: remember to replace with your own image
CI/CD
Helm chart linter
We have chart lint tests with Helm v3.4.1 and Helm v3.9.4 on GitHub Actions. We also provide a script to execute the lint tests on your laptop. If you cannot reproduce the errors on GitHub Actions, the possible reason is the different version of Helm. Issue #537 is an example that some errors only happen in old helm versions.
Run tests with docker
./helm-chart/script/chart-test.sh
Run tests on your local environment
- Step1: Install
ct(chart-testing) and related dependencies. See https://github.com/helm/chart-testing for more details. - Step2:
./helm-chart/script/chart-test.sh local
Consistency check
We have several consistency checks on GitHub Actions. There are several files which need synchronization.
ray-operator/apis/ray/v1alpha1/*_types.goshould be synchronized with the CRD YAML files (ray-operator/config/crd/bases/)ray-operator/apis/ray/v1alpha1/*_types.goshould be synchronized with generated API (ray-operator/pkg/client)- CRD YAML files in
ray-operator/config/crd/bases/andhelm-chart/kuberay-operator/crds/should be the same. - Kubebuilder markers in
ray-operator/controllers/ray/*_controller.goshould be synchronized with RBAC YAML files inray-operator/config/rbac. - RBAC YAML files in
helm-chart/kuberay-operator/templatesandray-operator/config/rbacshould be synchronized. Currently, we need to synchronize this manually. See #631 as an example.
# Synchronize consistency 1 and 4:
make manifests
# Synchronize consistency 2:
./hack/update-codegen.sh
# Synchronize consistency 3:
make helm
# Synchronize 1, 2, 3, and 4 in one command
# [Note]: Currently, we need to synchronize consistency 5 manually.
make sync
# Reproduce CI error for job "helm-chart-verify-rbac" (consistency 5)
python3 ../scripts/rbac-check.py
Run end-to-end tests locally
We have some end-to-end tests on GitHub Actions. These tests operate small Ray clusters running within a kind (Kubernetes-in-docker) environment. To run the tests yourself, follow these steps:
-
Step1: Install related dependencies, including kind and kubectl.
-
Step2: You must be in
/path/to/your/kuberay/.# [Usage]: RAY_IMAGE=$RAY_IMAGE OPERATOR_IMAGE=$OPERATOR_IMAGE python3 tests/compatibility-test.py # These 3 environment variables are optional. # [Example]: RAY_IMAGE=rayproject/ray:2.1.0 OPERATOR_IMAGE=kuberay/operator:nightly python3 tests/compatibility-test.py
Running configuration tests locally.
The sample RayCluster and RayService CRs under ray-operator/config/samples are tested in tests/framework/test_sample_raycluster_yamls.py
and tests/framework/test_sample_rayservice_yamls.py.
# Test RayCluster doc examples.
RAY_IMAGE=rayproject/ray:2.1.0 OPERATOR_IMAGE=kuberay/operator:nightly python3 tests/framework/test_sample_raycluster_yamls.py
# Test RayService doc examples.
RAY_IMAGE=rayproject/ray:2.1.0 OPERATOR_IMAGE=kuberay/operator:nightly python3 tests/framework/test_sample_rayservice_yamls.py
Currently, only a few of these sample configurations are tested in the CI. See KubeRay issue #695.