* [Docs] Add the draft description about feature intro, configurations, and usecases Signed-off-by: justinyeh1995 <[email protected]> * [Fix] Update the retry walk-through Signed-off-by: justinyeh1995 <[email protected]> * [Doc] rewrite the first 2 sections Signed-off-by: justinyeh1995 <[email protected]> * [Doc] Revise documentation wording and add Observing Retry Behavior section Signed-off-by: justinyeh1995 <[email protected]> * [Fix] fix linting issue by running pre-commit run berfore commiting Signed-off-by: justinyeh1995 <[email protected]> * [Fix] fix linting errors in the Markdown linting Signed-off-by: justinyeh1995 <[email protected]> * [Fix] Clean up the math equation Signed-off-by: justinyeh1995 <[email protected]> * Update the math formula of Backoff calculation. Co-authored-by: Nary Yeh <[email protected]> Signed-off-by: JustinYeh <[email protected]> * [Fix] Explicitly mentioned exponential backoff and removed the customization parts Signed-off-by: justinyeh1995 <[email protected]> * [Docs] Clarify naming by replacing “APIServer” with “KubeRay APIServer” Co-authored-by: Cheng-Yeh Chung <[email protected]> Signed-off-by: JustinYeh <[email protected]> * [Docs] Rename retry-configuration.md to retry-behavior.md for accuracy Signed-off-by: justinyeh1995 <[email protected]> * Update Title to KubeRay APIServer Retry Behavior Co-authored-by: Cheng-Yeh Chung <[email protected]> Signed-off-by: JustinYeh <[email protected]> * [Docs] Add a note about the limitation of retry configuration Signed-off-by: justinyeh1995 <[email protected]> --------- Signed-off-by: justinyeh1995 <[email protected]> Signed-off-by: JustinYeh <[email protected]> Co-authored-by: Nary Yeh <[email protected]> Co-authored-by: Cheng-Yeh Chung <[email protected]>
1.3 KiB
KubeRay APIServer Retry Behavior
The KubeRay APIServer automatically retries failed requests to the Kubernetes API when transient errors occur.
This built-in mechanism uses exponential backoff to improve reliability without requiring manual intervention.
As of v1.5.0, the retry configuration is hard-coded and cannot be customized.
This guide describes the default retry behavior.
Default Retry Behavior
The KubeRay APIServer automatically retries with exponential backoff for these HTTP status codes:
- 408 (Request Timeout)
- 429 (Too Many Requests)
- 500 (Internal Server Error)
- 502 (Bad Gateway)
- 503 (Service Unavailable)
- 504 (Gateway Timeout)
Note that non-retryable errors (4xx except 408/429) fail immediately without retries.
The following default configuration explains how retry works:
- MaxRetry: 3 retries (4 total attempts including the initial one)
- InitBackoff: 500ms (initial wait time)
- BackoffFactor: 2.0 (exponential multiplier)
- MaxBackoff: 10s (maximum wait time between retries)
- OverallTimeout: 30s (total timeout for all attempts)
which means \text{Backoff}_i = \min(\text{InitBackoff} \times \text{BackoffFactor}^i, \text{MaxBackoff})
where i is the attempt number (starting from 0).
The retries will stop if the total time exceeds the OverallTimeout.