## What changed?
Use the new dynamic config instead of leaving it as a placeholder and
update its documentation.
Also bumps the server version constant to `1.32.0`. This was supposed to
be done after the `1.31.0` release and was missed.
## Why?
Remove duplication as we migrate the code to the CHASM backed
implementation.
## Potential risks
This config is only relevant for external endpoints which are
experimental or older server versions, needs to be called out in the
release notes.
## What
Adds an architecture doc for the worker commands framework — how
commands flow from history through matching to workers, and how to add
new command types.
---------
Co-authored-by: Sean Kane <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
## What changed?
Deleted `"system.enableNexus"`.
## Why?
Nexus has been GA since Dec 2024.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Re-does #9138 which was incidentally merged.
Include transient and speculative WFT events in
`GetWorkflowExecutionHistoryReponse` response, unless UI or CLI made
request.
* Adds `transient_or_speculative_events` back to
`GetMutableStateResponse`
* Reserve `transient_workflow_task` in `HisotryCOntinuation` token
* Add validation helpers
* Add query-compare-query for transient events at request start and end
Re-implements #7732
## Why?
Fix "premature end of stream" errors when workers request history after
cache eviction w/ transient/speculative workflow tasks present. This
adds transient & speculative WFT events in `GetWorkflowExecutionHistory`
(already in `PollWorkflowTask`). Worker cache eviction w/ speculative
workflow tasks causes the expected and actual event counts to be
different. #7732 passed transient events through continuation tokens,
which could become stale during pagination. This PR implements mutable
state querying at both start and end of pagination and compares
transient event IDs to detect if WFT state changed during pagination and
return a retryable error.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Same risks from #7732
## What changed?
Include transient and speculative WFT events in
`GetWorkflowExecutionHistoryReponse` response, unless UI or CLI made
request.
* Adds `transient_or_speculative_events` back to
`GetMutableStateResponse`
* Reserve `transient_workflow_task` in `HisotryCOntinuation` token
* Add validation helpers
* Add query-compare-query for transient events at request start and end
Re-implements #7732
## Why?
Fix "premature end of stream" errors when workers request history after
cache eviction w/ transient/speculative workflow tasks present. This
adds transient & speculative WFT events in `GetWorkflowExecutionHistory`
(already in `PollWorkflowTask`). Worker cache eviction w/ speculative
workflow tasks causes the expected and actual event counts to be
different. #7732 passed transient events through continuation tokens,
which could become stale during pagination. This PR implements mutable
state querying at both start and end of pagination and compares
transient event IDs to detect if WFT state changed during pagination and
return a retryable error.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Same risks from #7732
## What changed?
- Puts an upper bound on how many concurrent backfillers can be active
in a scheduler.
## Why?
- Prevents a case of unbounded growth.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- I've set too low a limit, and on rollout, we break someone expecting a
higher number of concurrent backfillers. But I'd be pretty surprised if
anyone has a use case for 100+ backfillers..
## What changed?
- Fix issues in architecture doc diagrams
## Why?
- Fix issues in architecture doc diagrams
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Documentation Only. Update Nexus Readme to remove experimental language
and improve callback configuration recommendation.
## Why?
Nexus is Generally Available it is no longer experimental
Want to improve security posture for recommended configurations
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Roey Berman <[email protected]>
Co-authored-by: Roey Berman <[email protected]>
## What changed?
Context cancelations are getting incorrectly translated to `Unavailable`
service errors, which lead to Visibility Availability Alert false
positives.
This code change adds branches to check context error types before
casting to the resulting `serviceerror` for both ElasticSearch and SQL
visibility stores.
## Why?
Separate client and service side errors.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Rodrigo Zhou <[email protected]>
## What changed?
Improve the error message for an Update that is aborted by a closing
workflow.
## Why?
To make it easier for users to understand what happend; and for devs to
debug it easier.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Very low as the service error _type_ is still the same (NotFound).
## What changed?
Workflow Update: update doc to reflect recent changes in #7296.
## Why?
Keep internal doc up to date.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
<!-- Describe what has changed in this PR -->
Workflow Update: fix doc on when `WorkflowUpdateAbortedErr` is returned.
## Why?
<!-- Tell your future self why have you made these changes -->
Make doc in-sync with code.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
N/A
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
N/A
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
It is it.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
<!-- Describe what has changed in this PR -->
Drop speculative WFT even if it had events.
## Why?
<!-- Tell your future self why have you made these changes -->
Special compatibility flag was
[added](https://github.com/temporalio/api/pull/467) to
`RespondWorkflowTaskCompletedRequest`. If this flag is set to true, then
server can drop speculative WFT with update rejections, even if it
shipped events to the worker. SDK support will be added later.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Added new unit tests.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
No risks. Compatibility flag is used to support backward compatibility
and don't drop speculative WFT if worker doesn't support it (old go
SDKs).
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
Yes, updated.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
<!-- Describe what has changed in this PR -->
Delay Update abortion with effects package.
## Why?
<!-- Tell your future self why have you made these changes -->
To address race condition case when Update is accepted and WF is
completed on the same WFT. In this scenario Update result waiters should
have similar experience as if Update was failed with error on the same
WFT, meaning that even waiters for `Accepted` stage should get Update
result (which is abort failure). Previous implementation had race
condition and those waiter might get Update failure, or "Update is
accepted" response.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Added unit tests.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
No risks.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
Yes, updated.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
<!-- Describe what has changed in this PR -->
Abort updates with failure if update was accepted but workflow
completes.
## Why?
<!-- Tell your future self why have you made these changes -->
Accepted Updates are failed with special server failure because if a
client knows that Update has been accepted, it expects any following
requests to return an Update result (or failure) but not an error. There
can be different types of Update failures coming from worker and a
client must handle them anyway. It is easier and less error-prone for a
client to handle only Update failures instead of both failures and not
obvious `NotFound` errors in case if the Workflow completes before the
Update completes.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Modified existing tests.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
It is some sort of breaking change for very narrow case though.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
TBD
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
<!-- Describe what has changed in this PR -->
Workflow Update: update docs with recent code changes in #6477 and
#6485. Also fixed some minors.
## Why?
<!-- Tell your future self why have you made these changes -->
Keep doc up to date.
## What changed?
<!-- Describe what has changed in this PR -->
Move versioning docs to the previous place as it is referenced in the
public docs.
We can move/delete this file once versioning goes to public preview.
## Why?
<!-- Tell your future self why have you made these changes -->
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
Document Workflow Update internals.
## Why?
<!-- Tell your future self why have you made these changes -->
For better visibility on how Workflow Update is implemented.
---------
Co-authored-by: Stephan Behnke <[email protected]>
…oaded
## What changed?
MatchingService supports gRPC ForceLoadTaskQueuePartition. Whenever
matching engine loads a root partition, the RPC is called for all
non-root, non-sticky partitions of the same taskQueue.
## Why?
This addresses a unique situation where a TaskQueue has acquired a
backlog, and has been unloaded because there are no workers polling the
TaskQueue. When a worker comes back online, it can take a series of
polls before it is load balanced to all partitions, forcing them to
load. This can delay non-root partitions from attempting a sync-match
with the poller. Forcing the partitions to load when the root partition
is loaded can prevent this entirely because the root partition will be
forced to load when the poll triggers a sync-match.
## How did you test it?
Start workflow with sleep. Allow worker to handler workflow initially,
then kill it. Wait for partitions to be unloaded. Start worker back up,
watch task get picked up immediately.
## Potential risks
The new gRPC method and triggering code could be implemented incorrectly
and cause a cascade of calls. Worst case scenario would be perpetual
loop from root-partition to root-partition.
Additionally, we could have an unforeseen case where we call this RPC
far more often than necessary. Metrics have been installed to help us
keep an eye on that.
## Documentation
Matching Service documentation has been expanded to cover task queue
partitions.
## Is hotfix candidate?
nope.
## What changed?
<!-- Describe what has changed in this PR -->
Added a diagram to the retries.md docs.
## Why?
<!-- Tell your future self why have you made these changes -->
To make it clearer what's going on.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
Adding a short doc to give a basic overview of the gRPC retry behavior.
Note that this could/should also include the task processing retry
behavior, but I'm not familiar enough with it to write that right now.
## What changed?
<!-- Describe what has changed in this PR -->
I reorganized our docs into `docs/{admin,architecture,development}`.
## Why?
<!-- Tell your future self why have you made these changes -->
We discussed this internally, but I believe this is a better
segmentation for the different audiences looking at our docs. I plan on
adding more stuff to docs/admin like metrics and dynamic config docs.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
I'm mainly looking for broken links.
- I went through each markdown file in docs manually.
- I looked through all inspection errors in the IDE.
- I looked at all references to "docs/" or "develop/" in our code.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
zeekay
