mirror of
https://github.com/hanzo-ml/kuberay.git
synced 2026-10-30 19:48:47 +00:00
+6








a5adbbe18b
* [RayService] More envtests that follow the most common scenario in the RayService code path (#2880) Signed-off-by: Rueian <[email protected]> * [RayService] Remove outdated env tests (#2886) Signed-off-by: kaihsun <[email protected]> * [RayService] Refactor envtests (#2888) * [docs][ray-operator] fix typo in Golang version (#2893) Project uses Golang 1.22 not 1.20. Signed-off-by: David Xia <[email protected]> * [Fix][kubectl-plugin] Fix no context nil error SIGSEGV in tests (#2892) Signed-off-by: Chi-Sheng Liu <[email protected]> * [CI] Enable testifylint rule (#2896) Signed-off-by: Chi-Sheng Liu <[email protected]> * [CI] Enable `testifylint` `error-nil` rule (#2907) * [kubectl-plugin][feat] support specifying number of head GPUs (#2895) when creating a RayCluster with `kubectl ray create cluster NAME --head-gpu N`. Similar to the `--worker-gpu` switch. Signed-off-by: David Xia <[email protected]> * Use webhook.CustomValidator instead of deprecated webhook.Validator. (#2803) * Use webhook.CustomValidator instead of deprecated webhook.Validator. * Move RayClusterWebhook to pkg/webhook/v1. * [kubectl-plugin] update context error messages (#2891) to tell user they can use `--context` to set the K8s context. Add tests Signed-off-by: David Xia <[email protected]> Co-authored-by: Chi-Sheng Liu <[email protected]> * [Fix][kubectl-plugin] make tests use a temporary kube config (#2894) file that's not at the default path. Right now tests fail if there's a K8s current context set with a command like `kubectl config use-context my-context`. This change allows tests to pass regardless of current context. Signed-off-by: David Xia <[email protected]> * [CI] Enable testifylint formatter rule (#2915) Signed-off-by: 400Ping <[email protected]> * [CI] Enable `testifylint` `empty` rule (#2908) * [CI] Enable testifylint bool-compare rule (#2911) Signed-off-by: 400Ping <[email protected]> * [CI] Auto download golang tools in pre-commit (#2917) Signed-off-by: Chi-Sheng Liu <[email protected]> * [RayService] a safeguard for preventing overriding the pending cluster during a upgrade (#2887) Signed-off-by: Rueian <[email protected]> * [RayService] Refactor unit tests for ShouldPrepareNewCluster (#2928) Signed-off-by: kaihsun <[email protected]> * [chore][kubectl-plugin] use consistent capitalization (#2922) in comments and console messages. * capitalize "Ray" when used as a proper noun * capitalize Ray K8s CRDs like "RayJob" and "RayCluster" * capitalize acronyms like "YAML" * fix some minor typos No functional changes. Signed-off-by: David Xia <[email protected]> * [CI] Enable `testifylint` `require-error` rule (#2909) * [kubectl-plugin] support general `kubectl` switches like `--context` (#2883) * [CI] Fix lint error (require-error) (#2931) Signed-off-by: Chi-Sheng Liu <[email protected]> * [CI] Enable `testifylint` `float-compare` rule (#2910) * [docs] move pre-commit instructions to main dev docs (#2921) * [CI] Enable `testifylint` `expected-actual` rule (#2914) * [CI] Generate CRD json schema separately in pre-commit (#2930) * [release][1/N] Update YAMLs from Ray 2.9 to Ray 2.41 (#2934) * Delete `[raycluster|rayjob|rayservice]_types_test.go` unnecessary tests (#2935) * [release][2/N] Update RayCluster Helm chart from Ray 2.9 to Ray 2.41 (#2936) Signed-off-by: kaihsun <[email protected]> * [release][3/N] Update RayService e2e tests YAML files from Ray 2.9 to Ray 2.41 (#2937) * [release][4/N] Update Ray images / versions in kubectl plugin (#2938) Signed-off-by: kaihsun <[email protected]> * [release][5/N] Update some RayJob YAMLs from Ray 2.9 to Ray 2.41 (#2941) Signed-off-by: kaihsun <[email protected]> * [release][6/N] Remove unnecessary YAMLs (#2946) Signed-off-by: kaihsun <[email protected]> * [docs][kubectl-plugin] add dev docs (#2912) Signed-off-by: David Xia <[email protected]> Co-authored-by: Chi-Sheng Liu <[email protected]> * [CI] Add shellcheck and fix error of it (#2933) * [chore][kubectl-plugin] use better test assertions (#2955) Fix two places. * use `assert.EqualError()` instead of `assert.Error()` to check the error string is what we expect * use testify assertion instead of string checking for better error output Signed-off-by: David Xia <[email protected]> * [chore] add Markdown linting pre-commit hook (#2953) Ignore most rules we violate for now. Fix these two violations. ``` pre-commit run markdownlint --all-files --show-diff-on-failure ... clients/python-client/README.md:106 MD001/heading-increment Heading levels should only increment by one level at a time [Expected: h3; Actual: h4] CHANGELOG.md:420:1 MD030/list-marker-space Spaces after list markers [Expected: 1; Actual: 2] ``` Signed-off-by: David Xia <[email protected]> * [chore][kubectl-plugin] use consistent capitalization (#2950) in comments and console messages. capitalize "Ray" when used as a proper noun. Similar to #2922. Signed-off-by: David Xia <[email protected]> * [Fix][RayJob] Invalid quote for RayJob submitter (#2949) Closes: ray-project/kuberay#2943 Signed-off-by: Chi-Sheng Liu <[email protected]> * [release][7/N] Update RayService YAMLs (#2956) * [CI] Enable testifylint len rule (#2945) Signed-off-by: LeoLiao123 <[email protected]> * [docs][kubectl-plugin] improve help messages (#2952) Signed-off-by: David Xia <[email protected]> * [kubectl-plugin] Fix panic when GPU resource is not set (#2954) Signed-off-by: win5923 <[email protected]> * [release][8/N] Upgrade Stable Diffusion RayService to Ray 2.41 (#2960) * [docs][kubectl-plugin] fix incorrect example commands (#2951) Signed-off-by: David Xia <[email protected]> Co-authored-by: Chi-Sheng Liu <[email protected]> * [fix][kubectl-plugin] set worker group CPU limit (#2958) when creating a new worker group with `kubectl ray create workergroup`. Write a unit test. I noticed we are setting resource limits equal to resource requests everywhere else but in this command. I have a K8s [LimitRange] that prevented the creation of these worker Pods that had CPU limit defaulting to less than their CPU requests. Describing the RayCluster showed this warning event. `Failed to create worker Pod hyperkube/, Pod "dxia-test-other-group-worker-pm2sh" is invalid: spec.containers[0].resources.requests: Invalid value: "2": must be less than or equal to cpu limit of 250m` Signed-off-by: David Xia <[email protected]> [LimitRange]: https://kubernetes.io/docs/concepts/policy/limit-range/ * [RayJob] Deflaky RayJob e2e tests (#2963) Signed-off-by: kaihsun <[email protected]> * [RayService] Deflaky RayService envtest (#2962) Signed-off-by: kaihsun <[email protected]> * [release][9/N] Update text summarizer RayService to Ray 2.41 (#2961) * [RayService] adapter vllm 0.6.1.post2 (#2823) * adapter vllm 0.6.1.post2 * fix var define * Unify the cpu Settings in serve.py in service.yaml, all set to 1 * Maintain the configuration with vllm0.5x * [Release] Upgrade ray-job.batch-inference.yaml image to 2.41 (#2971) Signed-off-by: Chi-Sheng Liu <[email protected]> * [chore][docs] enable Markdownlint rule MD010 (#2975) * [CI] Change Pre-commit-shellcheck-to-shellcheck-py (#2974) Signed-off-by: owenowenisme <[email protected]> * [release] Update Yunikorn YAML file to Ray 2.41 (#2969) Signed-off-by: Cheng-Yeh Chung <[email protected]> * [release] Update YuniKorn YAML files to Ray 2.41 (#2976) Signed-off-by: win5923 <[email protected]> * [chore][docs] enable Markdownlint rule MD004 (#2973) [Unordered list style][1] [1]: https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md004---unordered-list-style * [Test] Use GcsFaultToleranceOptions in test and backward compatibility (#2972) * Update samples to use Ray 2.41.0 images (#2964) Signed-off-by: Andrew Sy Kim <[email protected]> * Update TPU Ray CR manifests to use Ray 2.41.0 (#2965) * [Refactor] Use constants for image tag, image repo, and versions in golang to avoid hard-coded strings (#2978) --------- Signed-off-by: Rueian <[email protected]> Signed-off-by: kaihsun <[email protected]> Signed-off-by: David Xia <[email protected]> Signed-off-by: Chi-Sheng Liu <[email protected]> Signed-off-by: 400Ping <[email protected]> Signed-off-by: LeoLiao123 <[email protected]> Signed-off-by: win5923 <[email protected]> Signed-off-by: owenowenisme <[email protected]> Signed-off-by: Cheng-Yeh Chung <[email protected]> Signed-off-by: Andrew Sy Kim <[email protected]> Co-authored-by: Rueian <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: David Xia <[email protected]> Co-authored-by: Chi-Sheng Liu <[email protected]> Co-authored-by: David Xia <[email protected]> Co-authored-by: Mykhailo Bobrovskyi <[email protected]> Co-authored-by: Ping <[email protected]> Co-authored-by: Owen Lin <[email protected]> Co-authored-by: Leo Liao <[email protected]> Co-authored-by: Blocka <[email protected]> Co-authored-by: zrant <[email protected]> Co-authored-by: kenchung285 <[email protected]> Co-authored-by: fscnick <[email protected]> Co-authored-by: ryanaoleary <[email protected]>
412 lines
14 KiB
Go
412 lines
14 KiB
Go
package log
|
|
|
|
import (
|
|
"archive/tar"
|
|
"bytes"
|
|
"context"
|
|
"errors"
|
|
"fmt"
|
|
"io"
|
|
"log"
|
|
"math"
|
|
"net/url"
|
|
"os"
|
|
"path"
|
|
"path/filepath"
|
|
"strings"
|
|
|
|
"github.com/spf13/cobra"
|
|
corev1 "k8s.io/api/core/v1"
|
|
v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
|
"k8s.io/cli-runtime/pkg/genericclioptions"
|
|
clientgoscheme "k8s.io/client-go/kubernetes/scheme"
|
|
"k8s.io/client-go/rest"
|
|
"k8s.io/client-go/tools/remotecommand"
|
|
cmdutil "k8s.io/kubectl/pkg/cmd/util"
|
|
"k8s.io/kubectl/pkg/util/templates"
|
|
|
|
"github.com/ray-project/kuberay/kubectl-plugin/pkg/util"
|
|
"github.com/ray-project/kuberay/kubectl-plugin/pkg/util/client"
|
|
"github.com/ray-project/kuberay/kubectl-plugin/pkg/util/completion"
|
|
)
|
|
|
|
const filePathInPod = "/tmp/ray/session_latest/logs/"
|
|
|
|
type ClusterLogOptions struct {
|
|
configFlags *genericclioptions.ConfigFlags
|
|
ioStreams *genericclioptions.IOStreams
|
|
Executor RemoteExecutor
|
|
outputDir string
|
|
nodeType string
|
|
ResourceName string
|
|
ResourceType util.ResourceType
|
|
}
|
|
|
|
var (
|
|
logLong = templates.LongDesc(`
|
|
Download logs from a Ray cluster and save them to a directory
|
|
`)
|
|
|
|
logExample = templates.Examples(`
|
|
# Download logs from a Ray cluster and save them to a directory with the Ray cluster's name. Retrieves 'all' logs
|
|
kubectl ray log my-raycluster
|
|
|
|
# Download logs from a Ray cluster and save them to a directory named /path/to/dir
|
|
kubectl ray log my-raycluster --out-dir /path/to/dir
|
|
|
|
# Download logs from a Ray cluster, but only for the head node
|
|
kubectl ray log my-raycluster --node-type head
|
|
|
|
# Download logs from a Ray cluster, but only for the worker nodes
|
|
kubectl ray log my-raycluster --node-type worker
|
|
|
|
# Download all (worker node and head node) the logs from a Ray cluster
|
|
kubectl ray log my-raycluster --node-type all
|
|
`)
|
|
|
|
// flag to check if output directory is generated and needs to be deleted
|
|
deleteOutputDir = false
|
|
)
|
|
|
|
func NewClusterLogOptions(streams genericclioptions.IOStreams) *ClusterLogOptions {
|
|
return &ClusterLogOptions{
|
|
configFlags: genericclioptions.NewConfigFlags(true),
|
|
ioStreams: &streams,
|
|
Executor: &DefaultRemoteExecutor{},
|
|
}
|
|
}
|
|
|
|
func NewClusterLogCommand(streams genericclioptions.IOStreams) *cobra.Command {
|
|
options := NewClusterLogOptions(streams)
|
|
// Initialize the factory for later use with the current config flag
|
|
cmdFactory := cmdutil.NewFactory(options.configFlags)
|
|
|
|
cmd := &cobra.Command{
|
|
Use: "log (RAYCLUSTER | TYPE/NAME) [--out-dir DIR_PATH] [--node-type all|head|worker]",
|
|
Short: "Get Ray cluster logs",
|
|
Long: logLong,
|
|
Example: logExample,
|
|
Aliases: []string{"logs"},
|
|
SilenceUsage: true,
|
|
ValidArgsFunction: completion.RayClusterResourceNameCompletionFunc(cmdFactory),
|
|
RunE: func(cmd *cobra.Command, args []string) error {
|
|
if err := options.Complete(cmd, args); err != nil {
|
|
return err
|
|
}
|
|
if err := options.Validate(); err != nil {
|
|
return err
|
|
}
|
|
return options.Run(cmd.Context(), cmdFactory)
|
|
},
|
|
}
|
|
cmd.Flags().StringVar(&options.outputDir, "out-dir", options.outputDir, "directory to save the logs to")
|
|
cmd.Flags().StringVar(&options.nodeType, "node-type", options.nodeType, "type of Ray node from which to download log, supports 'worker', 'head', or 'all'")
|
|
options.configFlags.AddFlags(cmd.Flags())
|
|
return cmd
|
|
}
|
|
|
|
func (options *ClusterLogOptions) Complete(cmd *cobra.Command, args []string) error {
|
|
if len(args) != 1 {
|
|
return cmdutil.UsageErrorf(cmd, "%s", cmd.Use)
|
|
}
|
|
|
|
if *options.configFlags.Namespace == "" {
|
|
*options.configFlags.Namespace = "default"
|
|
}
|
|
|
|
typeAndName := strings.Split(args[0], "/")
|
|
if len(typeAndName) == 1 {
|
|
options.ResourceType = util.RayCluster
|
|
options.ResourceName = typeAndName[0]
|
|
} else {
|
|
if len(typeAndName) != 2 || typeAndName[1] == "" {
|
|
return cmdutil.UsageErrorf(cmd, "invalid resource type/name: %s", args[0])
|
|
}
|
|
|
|
switch strings.ToLower(typeAndName[0]) {
|
|
case string(util.RayCluster):
|
|
options.ResourceType = util.RayCluster
|
|
case string(util.RayJob):
|
|
options.ResourceType = util.RayJob
|
|
case string(util.RayService):
|
|
options.ResourceType = util.RayService
|
|
default:
|
|
return cmdutil.UsageErrorf(cmd, "unsupported resource type: %s", typeAndName[0])
|
|
}
|
|
|
|
options.ResourceName = typeAndName[1]
|
|
}
|
|
|
|
if options.nodeType == "" {
|
|
options.nodeType = "all"
|
|
} else {
|
|
options.nodeType = strings.ToLower(options.nodeType)
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
func (options *ClusterLogOptions) Validate() error {
|
|
// Overrides and binds the kube config then retrieves the merged result
|
|
config, err := options.configFlags.ToRawKubeConfigLoader().RawConfig()
|
|
if err != nil {
|
|
return fmt.Errorf("Error retrieving raw config: %w", err)
|
|
}
|
|
if !util.HasKubectlContext(config, options.configFlags) {
|
|
return fmt.Errorf("no context is currently set, use %q or %q to select a new one", "--context", "kubectl config use-context <context>")
|
|
}
|
|
|
|
if options.outputDir == "" {
|
|
fmt.Fprintln(options.ioStreams.Out, "No output directory specified, creating dir under current directory using resource name.")
|
|
options.outputDir = options.ResourceName
|
|
err := os.MkdirAll(options.outputDir, 0o755)
|
|
if err != nil {
|
|
return fmt.Errorf("could not create directory with cluster name %s: %w", options.outputDir, err)
|
|
}
|
|
deleteOutputDir = true
|
|
}
|
|
|
|
switch options.nodeType {
|
|
case "all":
|
|
fmt.Fprintln(options.ioStreams.Out, "Command set to retrieve both head and worker node logs.")
|
|
case "head":
|
|
fmt.Fprintln(options.ioStreams.Out, "Command set to retrieve only head node logs.")
|
|
case "worker":
|
|
fmt.Fprintln(options.ioStreams.Out, "Command set to retrieve only worker node logs.")
|
|
default:
|
|
return fmt.Errorf("unknown node type `%s`", options.nodeType)
|
|
}
|
|
|
|
info, err := os.Stat(options.outputDir)
|
|
if os.IsNotExist(err) {
|
|
return fmt.Errorf("Directory does not exist. Failed with: %w", err)
|
|
} else if err != nil {
|
|
return fmt.Errorf("Error occurred will checking directory: %w", err)
|
|
} else if !info.IsDir() {
|
|
return fmt.Errorf("Path is not a directory. Please input a directory and try again")
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
func (options *ClusterLogOptions) Run(ctx context.Context, factory cmdutil.Factory) error {
|
|
clientSet, err := client.NewClient(factory)
|
|
if err != nil {
|
|
return fmt.Errorf("failed to retrieve Kubernetes client set: %w", err)
|
|
}
|
|
|
|
// Retrieve RayCluster name for the non RayCluster type node
|
|
var clusterName string
|
|
switch options.ResourceType {
|
|
case util.RayCluster:
|
|
clusterName = options.ResourceName
|
|
case util.RayJob:
|
|
rayJob, err := clientSet.RayClient().RayV1().RayJobs(*options.configFlags.Namespace).Get(ctx, options.ResourceName, v1.GetOptions{})
|
|
if err != nil {
|
|
return fmt.Errorf("failed to retrieve Ray job info for %s: %w", options.ResourceName, err)
|
|
}
|
|
clusterName = rayJob.Status.RayClusterName
|
|
case util.RayService:
|
|
rayService, err := clientSet.RayClient().RayV1().RayServices(*options.configFlags.Namespace).Get(ctx, options.ResourceName, v1.GetOptions{})
|
|
if err != nil {
|
|
return fmt.Errorf("failed to retrieve Ray job info for %s: %w", options.ResourceName, err)
|
|
}
|
|
clusterName = rayService.Status.ActiveServiceStatus.RayClusterName
|
|
default:
|
|
return fmt.Errorf("unsupported resource type: %s", options.ResourceType)
|
|
}
|
|
|
|
// set the list options for the specified nodetype
|
|
var listopts v1.ListOptions
|
|
switch options.nodeType {
|
|
case "all":
|
|
listopts = v1.ListOptions{
|
|
LabelSelector: fmt.Sprintf("ray.io/cluster=%s", clusterName),
|
|
}
|
|
case "head":
|
|
listopts = v1.ListOptions{
|
|
LabelSelector: fmt.Sprintf("ray.io/node-type=head, ray.io/cluster=%s", clusterName),
|
|
}
|
|
case "worker":
|
|
listopts = v1.ListOptions{
|
|
LabelSelector: fmt.Sprintf("ray.io/node-type=worker, ray.io/cluster=%s", clusterName),
|
|
}
|
|
default:
|
|
return fmt.Errorf("Unknown ray resource node type: %s", options.nodeType)
|
|
}
|
|
|
|
// Get list of nodes that are considered the specified node type
|
|
rayNodes, err := clientSet.KubernetesClient().CoreV1().Pods(*options.configFlags.Namespace).List(ctx, listopts)
|
|
if err != nil {
|
|
return fmt.Errorf("failed to retrieve head node for Ray cluster %s: %w", clusterName, err)
|
|
}
|
|
if len(rayNodes.Items) == 0 {
|
|
// Clean up the empty directory if the directory was generated. Since it will always be in current dir, only Remove() is used.
|
|
if deleteOutputDir {
|
|
os.Remove(options.outputDir)
|
|
}
|
|
return fmt.Errorf("No Ray nodes found for resource %s", clusterName)
|
|
}
|
|
|
|
// Get a list of logs of the Ray nodes.
|
|
var logList []*bytes.Buffer
|
|
for _, rayNode := range rayNodes.Items {
|
|
// Since the first container is always the Ray container, we will retrieve the first container logs
|
|
containerName := rayNode.Spec.Containers[0].Name
|
|
request := clientSet.KubernetesClient().CoreV1().Pods(rayNode.Namespace).GetLogs(rayNode.Name, &corev1.PodLogOptions{Container: containerName})
|
|
|
|
podLogs, err := request.Stream(ctx)
|
|
if err != nil {
|
|
return fmt.Errorf("Error retrieving log for Ray cluster node %s: %w", rayNode.Name, err)
|
|
}
|
|
defer podLogs.Close()
|
|
|
|
// Get current logs:
|
|
buf := new(bytes.Buffer)
|
|
_, err = io.Copy(buf, podLogs)
|
|
if err != nil {
|
|
return fmt.Errorf("Failed to get read current logs for Ray cluster Node %s: %w", rayNode.Name, err)
|
|
}
|
|
|
|
logList = append(logList, buf)
|
|
}
|
|
|
|
// Pod file name format is name of the Ray node
|
|
for ind, logList := range logList {
|
|
curFilePath := filepath.Join(options.outputDir, rayNodes.Items[ind].Name, "stdout.log")
|
|
dirPath := filepath.Join(options.outputDir, rayNodes.Items[ind].Name)
|
|
err := os.MkdirAll(dirPath, 0o755)
|
|
if err != nil {
|
|
return fmt.Errorf("failed to create directory within path %s: %w", dirPath, err)
|
|
}
|
|
file, err := os.OpenFile(curFilePath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o644)
|
|
if err != nil {
|
|
return fmt.Errorf("failed to create/open file for kuberay-head with path: %s: %w", curFilePath, err)
|
|
}
|
|
defer file.Close()
|
|
|
|
_, err = logList.WriteTo(file)
|
|
if err != nil {
|
|
return fmt.Errorf("failed to write to file for kuberay-head: %s: %w", rayNodes.Items[ind].Name, err)
|
|
}
|
|
|
|
containerName := rayNodes.Items[ind].Spec.Containers[0].Name
|
|
req := clientSet.KubernetesClient().CoreV1().RESTClient().
|
|
Get().
|
|
Namespace(rayNodes.Items[ind].Namespace).
|
|
Resource("pods").
|
|
Name(rayNodes.Items[ind].Name).
|
|
SubResource("exec").
|
|
Param("container", containerName).
|
|
VersionedParams(&corev1.PodExecOptions{
|
|
Command: []string{"tar", "--warning=no-file-changed", "-cf", "-", "-C", filePathInPod, "."},
|
|
Stdin: true,
|
|
Stdout: true,
|
|
Stderr: true,
|
|
TTY: false,
|
|
}, clientgoscheme.ParameterCodec)
|
|
|
|
restconfig, err := factory.ToRESTConfig()
|
|
if err != nil {
|
|
return fmt.Errorf("failed to get restconfig: %w", err)
|
|
}
|
|
|
|
exec, err := options.Executor.CreateExecutor(restconfig, req.URL())
|
|
if err != nil {
|
|
return fmt.Errorf("failed to create executor with error: %w", err)
|
|
}
|
|
|
|
err = options.downloadRayLogFiles(ctx, exec, rayNodes.Items[ind])
|
|
if err != nil {
|
|
return fmt.Errorf("failed to download ray head log files with error: %w", err)
|
|
}
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// RemoteExecutor creates the executor for executing exec on the pod - provided for testing purposes
|
|
type RemoteExecutor interface {
|
|
CreateExecutor(restConfig *rest.Config, url *url.URL) (remotecommand.Executor, error)
|
|
}
|
|
|
|
type DefaultRemoteExecutor struct{}
|
|
|
|
// CreateExecutor returns the executor created by NewSPDYExecutor
|
|
func (dre *DefaultRemoteExecutor) CreateExecutor(restConfig *rest.Config, url *url.URL) (remotecommand.Executor, error) {
|
|
return remotecommand.NewSPDYExecutor(restConfig, "POST", url)
|
|
}
|
|
|
|
// downloadRayLogFiles will use to the executor and retrieve the logs file from the inputted Ray head
|
|
func (options *ClusterLogOptions) downloadRayLogFiles(ctx context.Context, exec remotecommand.Executor, rayNode corev1.Pod) error {
|
|
outreader, outStream := io.Pipe()
|
|
go func() {
|
|
defer outStream.Close()
|
|
err := exec.StreamWithContext(ctx, remotecommand.StreamOptions{
|
|
Stdin: options.ioStreams.In,
|
|
Stdout: outStream,
|
|
Stderr: options.ioStreams.ErrOut,
|
|
Tty: false,
|
|
})
|
|
if err != nil {
|
|
log.Fatalf("Error occurred while calling remote command: %v", err)
|
|
}
|
|
}()
|
|
|
|
// Goes through the tar and create/copy them one by one into the destination dir
|
|
tarReader := tar.NewReader(outreader)
|
|
header, err := tarReader.Next()
|
|
if err != nil && !errors.Is(err, io.EOF) {
|
|
return fmt.Errorf("error will extracting head tar file for Ray head %s: %w", rayNode.Name, err)
|
|
}
|
|
|
|
fmt.Fprintf(options.ioStreams.Out, "Downloading log for Ray Node %s\n", rayNode.Name)
|
|
for !errors.Is(err, io.EOF) {
|
|
if err != nil {
|
|
return fmt.Errorf("Error reading tar archive: %w", err)
|
|
}
|
|
|
|
// Construct the full local path and a directory for the tmp file logs
|
|
localFilePath := filepath.Join(path.Clean(options.outputDir), path.Clean(rayNode.Name), path.Clean(header.Name))
|
|
|
|
switch header.Typeflag {
|
|
case tar.TypeDir:
|
|
if err := os.MkdirAll(localFilePath, 0o755); err != nil {
|
|
return fmt.Errorf("Error creating directory: %w", err)
|
|
}
|
|
case tar.TypeReg:
|
|
// Check for overflow: G115
|
|
if header.Mode < 0 || header.Mode > math.MaxUint32 {
|
|
fmt.Fprintf(options.ioStreams.Out, "file mode out side of accceptable value %d skipping file", header.Mode)
|
|
}
|
|
// Create file and write contents
|
|
outFile, err := os.OpenFile(localFilePath, os.O_CREATE|os.O_RDWR, os.FileMode(header.Mode)) //nolint:gosec // lint failing due to file mode conversion from uint64 to int32, checked above
|
|
if err != nil {
|
|
return fmt.Errorf("Error creating file: %w", err)
|
|
}
|
|
defer outFile.Close()
|
|
// This is to limit the copy size for a decompression bomb, currently set arbitrarily
|
|
for {
|
|
n, err := io.CopyN(outFile, tarReader, 1000000)
|
|
if err != nil {
|
|
if errors.Is(err, io.EOF) {
|
|
break
|
|
}
|
|
return fmt.Errorf("failed while writing to file: %w", err)
|
|
}
|
|
if n == 0 {
|
|
break
|
|
}
|
|
}
|
|
default:
|
|
fmt.Printf("Ignoring unsupported file type: %b", header.Typeflag)
|
|
}
|
|
|
|
header, err = tarReader.Next()
|
|
if header == nil && err != nil && !errors.Is(err, io.EOF) {
|
|
return fmt.Errorf("error while extracting tar file with error: %w", err)
|
|
}
|
|
}
|
|
|
|
return nil
|
|
}
|