4879adacdb
* new local website docs Signed-off-by: vara-bonthu <[email protected]> * homepage UX: ribbon logo, architecture diagram, CNCF footer, code-block fixes Signed-off-by: vara-bonthu <[email protected]> * docs: favicon, social cards, image optimization, consistent paragraph sizing Signed-off-by: vara-bonthu <[email protected]> * Potential fix for pull request finding Signed-off-by: Vara Bonthu <[email protected]> * docs: address PR review feedback (brand link, storage guard, cron, typos) Signed-off-by: vara-bonthu <[email protected]> * docs: address PR review (spec.mode, jar typo, CSS @import order, external-link host check, brand-link fallback) + strict docs build gate Signed-off-by: vara-bonthu <[email protected]> * docs: embed Kubeflow Spark Operator intro video on homepage Signed-off-by: vara-bonthu <[email protected]> * docs: clarify homepage video is the CNCF KubeCon NA 2024 best-practices talk Signed-off-by: vara-bonthu <[email protected]> * docs: fix header logo opening homepage in a new tab (exclude brand link from external-link handling) Signed-off-by: vara-bonthu <[email protected]> * docs: fix header logo navigation and point Kubeflow.org links to kubeflow.org Signed-off-by: vara-bonthu <[email protected]> * ci: add documentation build and link-check workflow Signed-off-by: vara-bonthu <[email protected]> * ci(docs): make linkcheck non-blocking; fix broken links and ignore JS-rendered anchors Signed-off-by: vara-bonthu <[email protected]> * docs: fix benchmarking links (Data on EKS benchmark kit and contributor handle) Signed-off-by: vara-bonthu <[email protected]> * docs: refine homepage copy and add muted color accents to landing cards Signed-off-by: vara-bonthu <[email protected]> * docs: move website under docs/website and manage Python deps with uv - Move the Sphinx site from docs/ to docs/website/ - Replace requirements.txt with pyproject.toml (uv 'docs' group) and uv.lock - Build with uv in Read the Docs, the Makefile targets, and CI - Leave pre-existing docs/{api-docs,kustomize-installation,release}.md in place Signed-off-by: vara-bonthu <[email protected]> * docs: improve sidebar navigation readability and hierarchy Signed-off-by: vara-bonthu <[email protected]> --------- Signed-off-by: vara-bonthu <[email protected]> Signed-off-by: Vara Bonthu <[email protected]> Co-authored-by: Hanzo Dev <[email protected]>
2.3 KiB
Using SparkApplications
The operator runs Spark applications specified in Kubernetes objects of the SparkApplication custom resource type. The most common way of using a SparkApplication is store the SparkApplication specification in a YAML file and use the kubectl command to work with the SparkApplication. The operator automatically submits the application as configured in a SparkApplication to run on the Kubernetes cluster and uses the SparkApplication to collect and surface the status of the driver and executors to the user.
As with all other Kubernetes API objects, a SparkApplication needs the apiVersion, kind, and metadata fields. For general information about working with manifests, see object management using kubectl.
A SparkApplication also needs a .spec section. This section contains fields for specifying various aspects of an application including its type (Scala, Java, Python, or R), deployment mode (cluster or client), main application resource URI (e.g., the URI of the application jar), main class, arguments, etc. Node selectors are also supported via the optional field .spec.nodeSelector.
It also has fields for specifying the unified container image (to use for both the driver and executors) and the image pull policy, namely, .spec.image and .spec.imagePullPolicy respectively. If a custom init-container (in both the driver and executor pods) image needs to be used, the optional field .spec.initContainerImage can be used to specify it. If set, .spec.initContainerImage overrides .spec.image for the init-container image. Otherwise, the image specified by .spec.image will be used for the init-container. It is invalid if both .spec.image and .spec.initContainerImage are not set.
Below is an example showing part of a SparkApplication specification:
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
name: spark-pi
namespace: default
spec:
type: Scala
mode: cluster
image: spark:3.5.1
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.1.jar