mirror of
https://github.com/hanzoai/tasks.git
synced 2026-10-30 14:56:19 +00:00
bf23e93d68
## What changed? <!-- Describe what has changed in this PR --> Workflow Update: abort updates with non-retryable error if WFT fails unexpectedly. ## Why? <!-- Tell your future self why have you made these changes --> If WFT fails while worker is trying to complete it (because of different size limit violations, or bug in SDK, or other validation errors), server clears Update registry and aborts Updates with retryable `Unavailable` error. Internal retries recreate Update and new WFT. If workers keeps sending same malformed/invalid response then WFT fails again and all of these repeats in the loop until client timeout. Things are getting worse if Update message is the cause of the WFT failure. In this case, because retries of Update and WFT are not synced, not every WFT got Update message, and empty WFT completes successfully. This means that it starts to jump back and forth between completed and failed state and never become transient, which leads to very fast history grow. Failing Update with non-retryable `WorkflowNotReady` error stops retries right away and returns error response to the client quicker. ## How did you test it? <!-- How have you verified this change? Tested locally? Added a unit test? Checked in staging env? --> Modified existing functional test. ## Potential risks <!-- Assuming the worst case, what can be broken when deploying this change to production? --> No risks. ## Documentation <!-- Have you made sure this change doesn't falsify anything currently stated in `docs/`? If significant new behavior is added, have you described that in `docs/`? --> Needs to be updated. ## Is hotfix candidate? <!-- Is this PR a hotfix candidate or does it require a notification to be sent to the broader community? (Yes/No) --> Yes. --------- Co-authored-by: Stephan Behnke <[email protected]>