4879adacdb
* new local website docs Signed-off-by: vara-bonthu <[email protected]> * homepage UX: ribbon logo, architecture diagram, CNCF footer, code-block fixes Signed-off-by: vara-bonthu <[email protected]> * docs: favicon, social cards, image optimization, consistent paragraph sizing Signed-off-by: vara-bonthu <[email protected]> * Potential fix for pull request finding Signed-off-by: Vara Bonthu <[email protected]> * docs: address PR review feedback (brand link, storage guard, cron, typos) Signed-off-by: vara-bonthu <[email protected]> * docs: address PR review (spec.mode, jar typo, CSS @import order, external-link host check, brand-link fallback) + strict docs build gate Signed-off-by: vara-bonthu <[email protected]> * docs: embed Kubeflow Spark Operator intro video on homepage Signed-off-by: vara-bonthu <[email protected]> * docs: clarify homepage video is the CNCF KubeCon NA 2024 best-practices talk Signed-off-by: vara-bonthu <[email protected]> * docs: fix header logo opening homepage in a new tab (exclude brand link from external-link handling) Signed-off-by: vara-bonthu <[email protected]> * docs: fix header logo navigation and point Kubeflow.org links to kubeflow.org Signed-off-by: vara-bonthu <[email protected]> * ci: add documentation build and link-check workflow Signed-off-by: vara-bonthu <[email protected]> * ci(docs): make linkcheck non-blocking; fix broken links and ignore JS-rendered anchors Signed-off-by: vara-bonthu <[email protected]> * docs: fix benchmarking links (Data on EKS benchmark kit and contributor handle) Signed-off-by: vara-bonthu <[email protected]> * docs: refine homepage copy and add muted color accents to landing cards Signed-off-by: vara-bonthu <[email protected]> * docs: move website under docs/website and manage Python deps with uv - Move the Sphinx site from docs/ to docs/website/ - Replace requirements.txt with pyproject.toml (uv 'docs' group) and uv.lock - Build with uv in Read the Docs, the Makefile targets, and CI - Leave pre-existing docs/{api-docs,kustomize-installation,release}.md in place Signed-off-by: vara-bonthu <[email protected]> * docs: improve sidebar navigation readability and hierarchy Signed-off-by: vara-bonthu <[email protected]> --------- Signed-off-by: vara-bonthu <[email protected]> Signed-off-by: Vara Bonthu <[email protected]> Co-authored-by: Hanzo Dev <[email protected]>
4.2 KiB
Working with SparkApplications
Creating a New SparkApplication
A SparkApplication can be created from a YAML file storing the SparkApplication specification using the kubectl apply -f <YAML file path> command. Once a SparkApplication is successfully created, the operator will receive it and submit the application as configured in the specification to run on the Kubernetes cluster. Please note that the operator submits SparkApplications in cluster mode only.
Deleting a SparkApplication
A SparkApplication can be deleted using either the kubectl delete <name> command. Deleting a SparkApplication deletes the Spark application associated with it. If the application is running when the deletion happens, the application is killed and all Kubernetes resources associated with the application are deleted or garbage collected.
Updating a SparkApplication
A SparkApplication can be updated using the kubectl apply -f <updated YAML file> command. When a SparkApplication is successfully updated, the operator will receive both the updated and old SparkApplication objects. If the specification of the SparkApplication has changed, the operator submits the application to run, using the updated specification. If the application is currently running, the operator kills the running application before submitting a new run with the updated specification. There is planned work to enhance the way SparkApplication updates are handled. For example, if the change was to increase the number of executor instances, instead of killing the currently running application and starting a new run, it is a much better user experience to incrementally launch the additional executor pods.
Checking a SparkApplication
A SparkApplication can be checked using the kubectl describe sparkapplications <name> command. The output of the command shows the specification and status of the SparkApplication as well as events associated with it. The events communicate the overall process and errors of the SparkApplication.
Configuring Automatic Application Restart and Failure Handling
The operator supports automatic application restart with a configurable RestartPolicy using the optional field
.spec.restartPolicy. The following is an example of a sample RestartPolicy:
restartPolicy:
type: OnFailure
onFailureRetries: 3
onFailureRetryInterval: 10
onSubmissionFailureRetries: 5
onSubmissionFailureRetryInterval: 20
The valid types of restartPolicy include Never, OnFailure, and Always. Upon termination of an application,
the operator determines if the application is subject to restart based on its termination state and the
RestartPolicy in the specification. If the application is subject to restart, the operator restarts it by
submitting a new run of it. For OnFailure, the Operator further supports setting limits on number of retries
via the onFailureRetries and onSubmissionFailureRetries fields. Additionally, if the submission retries has not been reached,
the operator retries submitting the application using a linear backoff with the interval specified by
onFailureRetryInterval and onSubmissionFailureRetryInterval which are required for both OnFailure and Always RestartPolicy.
The old resources like driver pod, ui service/ingress etc. are deleted if it still exists before submitting the new run, and a new driver pod is created by the submission
client so effectively the driver gets restarted.
Setting TTL for a SparkApplication
The v1beta2 version of the SparkApplication API starts having TTL support for SparkApplications through a new optional field named .spec.timeToLiveSeconds, which if set, defines the Time-To-Live (TTL) duration in seconds for a SparkApplication after its termination. The SparkApplication object will be garbage collected if the current time is more than the .spec.timeToLiveSeconds since its termination. The example below illustrates how to use the field:
spec:
timeToLiveSeconds: 3600
Note that this feature requires that informer cache resync to be enabled, which is true by default with a resync internal of 30 seconds. You can change the resync interval by setting the flag -resync-interval=<interval>.