mirror of
https://github.com/hanzo-ml/trainer.git
synced 2026-09-24 00:31:34 +00:00
Adds terminationGracePeriodSeconds field to PodSpecPatch so users can configure pod termination grace period per TrainJob via RuntimePatches. This is needed for distributed training with PyTorch Elastic (torchrun) where large models (70B+ parameters) require more than the default 30s to complete JIT checkpointing before SIGKILL on node drain or TrainJob pause. No changes to merge logic in trainingruntime.go are required since the existing StrategicMergePatch applied at batchv1.JobTemplateSpec level already handles this field automatically. Closes #3285 Signed-off-by: krishdef7 <[email protected]>
804 KiB
Generated
804 KiB
Generated