Shahab TajikandGitHub ee00da67b9 Validate build id and reschedule tasks when redirect rule applies (#5681) ...

## What changed?
<!-- Describe what has changed in this PR -->
- Send BuildIdRedirectInfo from Matching to History on
Record*TaskStarted call containing information about redirect intention.
- History validates the redirect info against current MS and fails the
request if the redirect does not have source build if equal to the
current assigned build of the workflow.
- If redirect is valid, workflow is assigned to the new build id and all
pending but not started task are rescheduled to be sent to the new build
id.

## Why?
<!-- Tell your future self why have you made these changes -->
To prevent the following problems from happening for workflows with
concurrent tasks when redirect rules are deleted or not fully propagated
yet:
1. Assign wf back to an old build id after processing task using a newer
build id
2. Interleaved Starts: for some duration tasks are dispatched to a mix
of old or new build IDs
3. New activity output being fed to old wf
4. Execution gets stuck after being (partially) redirected

## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Functional test. More unit tests to be added in a followup PR.

## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
As it is right now, in rare situations when a redirect rule is applied
to a WF with concurrent activities and some of them are in backoff
period due to failure, we may schedule (and start) them on the newer
build without waiting for the backoff to finish. This is planed to be
improved in the futuer.

## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
None.

## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.

2024-05-17 15:49:29 -07:00

..

v1

Validate build id and reschedule tasks when redirect rule applies (#5681)

2024-05-17 15:49:29 -07:00