Commit Graph
10 Commits
Author SHA1 Message Date
Stefan WangandGitHub c3153f3f63 fix(cache): use status code (not identity) to detect Service AlreadyExists (#3507)
The `cache` dataset initializer compared the exception instance to the
`ConflictError` class using `is`, which is always False:

    except ApiException as e:
        if e is ConflictError:  # always False

`core_v1.create_namespaced_service()` raises a plain `ApiException` with
`status=409`, not a `ConflictError`, so this branch was unreachable. When
the Service already existed (e.g., re-reconcile after a crash between
Service creation and LWS readiness), the code re-raised, fell into the
outer cleanup, and deleted the ServiceAccount — leaving the cache
cluster in a broken state.

Match the pattern used a few lines above for the ServiceAccount and
check `e.status == 409` instead.

Add a unit test that mocks `create_namespaced_service` to raise 409 and
asserts `download_dataset` completes without calling
`delete_namespaced_service_account`.

Signed-off-by: 1fanwang <[email protected]>
2026-06-15 14:58:28 +00:00
Tariq HasanandGitHub 9183dfd24f chore(scripts): add copyright boilerplate verification (#3359)
* chore(scripts): add copyright boilerplate verification system

Signed-off-by: tariq-hasan <[email protected]>

* chore(ci): add boilerplate verification to CI pipeline

Signed-off-by: tariq-hasan <[email protected]>

* chore: regenerate files with year-less copyright headers

Signed-off-by: tariq-hasan <[email protected]>

* refactor(boilerplate): simplify verification logic

Signed-off-by: tariq-hasan <[email protected]>

* chore(boilerplate): add missing copyright headers

Signed-off-by: tariq-hasan <[email protected]>

* ci(boilerplate): move verifier to code-quality workflow and run unit tests

Signed-off-by: tariq-hasan <[email protected]>

* feat(boilerplate): accept license headers with any year and remove year validation logic

Signed-off-by: tariq-hasan <[email protected]>

* feat(boilerplate): add copyright headers to generated Python API

Signed-off-by: tariq-hasan <[email protected]>

* refactor(boilerplate): use block-comment header for generated Go files

Signed-off-by: tariq-hasan <[email protected]>

* feat(boilerplate): verify copyright headers in Rust files

Signed-off-by: tariq-hasan <[email protected]>

---------

Signed-off-by: tariq-hasan <[email protected]>
2026-05-29 10:50:56 +00:00
e8ce5b0a28 fix(initializer): add missing glob wildcard to .pt and .pth ignore p… (#3307)
* fix(initializers): add missing glob wildcard to .pt and .pth ignore patterns

Signed-off-by: Vahagn <[email protected]>

* Apply suggestion from @Copilot

Signed-off-by: ghazariann <[email protected]>

* fix(initializer): use fnmatch glob matching in S3Storage ignore_patterns

Signed-off-by: Vahagn <[email protected]>

---------

Signed-off-by: Vahagn <[email protected]>
Signed-off-by: ghazariann <[email protected]>

Co-authored-by: Hanzo Dev <[email protected]>
2026-03-18 12:31:17 +00:00
Fiona WatersandGitHub 04b222787d fix: Defer kubernetes imports to method level for use with local mode (#3167)
* fix: Defer kubernetes imports to method level for use with local mode

Signed-off-by: Fiona-Waters <[email protected]>

* fix: Update init.py with lazy imports in case block

Signed-off-by: Fiona-Waters <[email protected]>

* fix: Update unit tests

Signed-off-by: Fiona-Waters <[email protected]>

---------

Signed-off-by: Fiona-Waters <[email protected]>
2026-02-05 14:44:21 +00:00
4e41a4f83f feat(cache): KEP-2655 - Supporting readiness probes on cache nodes (#2904)
Signed-off-by: Akshay Chitneni <[email protected]>
Co-authored-by: Akshay Chitneni <[email protected]>
2025-11-05 18:48:30 +00:00
Yuchen ChengandGitHub 9b198974a7 feat(initializer): add s3 model and dataset initializers (#2728)
* feat(initializer): add s3 model and dataset initializers

Signed-off-by: rudeigerc <[email protected]>

* chore: refactor with opendal

Signed-off-by: rudeigerc <[email protected]>

* chore: support `role_arn` and add `ignore_patterns` field in the Initializers configs

Signed-off-by: rudeigerc <[email protected]>

---------

Signed-off-by: rudeigerc <[email protected]>
2025-10-31 15:28:12 +00:00
4361def5e3 feat(cache): KEP-2655: Adding cache initializer (#2793)
Signed-off-by: Akshay Chitneni <[email protected]>
Co-authored-by: Akshay Chitneni <[email protected]>
2025-10-13 19:53:06 +00:00
Akash JaiswalandGitHub cbb653e620 feat: qwen 2.5 1.5b runtime, example and fix gpu e2e test (#2835)
* add: qwen 3 1.7b CTR

Signed-off-by: Akash Jaiswal <[email protected]>

* fix: pvc and ctr

Signed-off-by: Akash Jaiswal <[email protected]>

* feat: qwen 2, 1.5b model as example

Signed-off-by: Akash Jaiswal <[email protected]>

* fix: allow pattern and config update

Signed-off-by: Akash Jaiswal <[email protected]>

* fix: test and typo

Signed-off-by: Akash Jaiswal <[email protected]>

* update: qwen3 model

Signed-off-by: Akash Jaiswal <[email protected]>

* fix: add qwen 2.5, fix gpu e2e test and use local images

Signed-off-by: Akash Jaiswal <[email protected]>

* fix: whitespace

Signed-off-by: Akash Jaiswal <[email protected]>

* fix: review suggestions

Signed-off-by: Akash Jaiswal <[email protected]>

* fix

Signed-off-by: Akash Jaiswal <[email protected]>

---------

Signed-off-by: Akash Jaiswal <[email protected]>
2025-09-20 21:02:59 +00:00
Shao WangandGitHub fc7d3cca19 KEP-2401: Support mutating dataset preprocessing config in SDK (#2638)
* feat(sdk): Add InstructDataset and dataset_preprocess_config.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(doc): update dataset preprocessing API definition in KEP.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(sdk): Add get_args_in_dataset_preprocess_config func.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(sdk): Add the prefix path for dataset.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(manifests): Load local datasets.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(sdk): Add TorchTune prefix to dataset class.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(initializer): Update HF dataset initializer to support data_dir and data_files.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(sdk): remove data_files and data_dir definition.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(sdk): extract data_files and data_dir from storage_uri.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(sdk): fix errors in UT.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(manifest): Update dataset.data_dir in torchtune CTRs.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(sdk): fix dataset_uri.

Signed-off-by: Electronic-Waste <[email protected]>

---------

Signed-off-by: Electronic-Waste <[email protected]>
2025-06-04 18:08:15 +00:00
Andrey VelichkevichandGitHub 239b27d311 feat(controller): Refactor the Initializer APIs of TrainJob (#2523)
* feat(controller): Refactor the Initializer APIs of TrainJob

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix go unit test

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix integration test

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
2025-03-17 07:15:51 +00:00