Alex ShtinandGitHub 03c5b1fd11 ScheduleToStart timeout for speculative WT on normal task queue (#5146) ...

<!-- Describe what has changed in this PR -->
**What changed?**
Add `ScheduleToStart` timeout for speculative WT on normal task queue.

<!-- Tell your future self why have you made these changes -->
**Why?**
`UpdateWorkflowExecution` API handler tries to add WT to matching
directly. This call must be made outside of workflow lock and might fail
(if matching is temporarily unavailable, for example). There is no good
way to handle this error after lock was released. This is why I
introduced this `ScheduleToStart` timeout for speculative WT on the
normal task queue. This timeout will ensure that WT will be eventually
added to matching and workflow won't stuck.

Also removed special flag which was used to skip transfer task creation
when speculative WT is converted to normal before MS with speculative WT
is persisted.

Both these speculative WT logic changes (timeout on normal and skip
transfer task) are extracted to special
`GenerateScheduleSpeculativeWorkflowTaskTasks` method.

<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
**How did you test it?**
Added functional test.

<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
**Potential risks**
Speculative WT times out once if there is no worker available.

<!-- Is this PR a hotfix candidate or require that a notification be
sent to the broader community? (Yes/No) -->
**Is hotfix candidate?**
No.

2023-11-30 03:19:41 +00:00

..

api.go

ScheduleToStart timeout for speculative WT on normal task queue (#5146)

2023-11-30 03:19:41 +00:00