## What changed?
Use the new dynamic config instead of leaving it as a placeholder and
update its documentation.
Also bumps the server version constant to `1.32.0`. This was supposed to
be done after the `1.31.0` release and was missed.
## Why?
Remove duplication as we migrate the code to the CHASM backed
implementation.
## Potential risks
This config is only relevant for external endpoints which are
experimental or older server versions, needs to be called out in the
release notes.
- GitHub Actions: parallel amd64 + arm64 on self-hosted runners
(lqd-build-amd64/arm64), multi-arch manifest, push to GAR
- Dockerfile: CGO-enabled Go build with sqlite support
- embedded-sqlite.yaml: file-based WAL persistence at /data/
- Build tags on test utility files to fix production builds
- Removed luxfi/zap local replace, using published v0.2.1
## What changed?
- Added `development-jwt.yaml` config that runs Temporal with JWT
authorization enabled using a JWKS file loaded from local disk. Loading
from local disk is added in
https://github.com/temporalio/temporal/pull/9590.
- Added `config/jwt/setup-keys.sh` to generate an RSA key pair and JWKS
file in `/tmp/temporal-jwt-test/`.
- Added `config/jwt/generate-token.sh` to create signed JWTs for
testing.
- Added `make start-jwt` target.
## Why?
Testing JWT authentication currently requires an external identity
provider or manual key/token setup. This provides a self-contained local
workflow: `make start-jwt` generates keys automatically and starts the
server with auth enforced, and `generate-token.sh` produces tokens on
demand.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
Tested this in conjunction with
https://github.com/temporalio/temporal/pull/9582. Was able to list the
workflows by passing the token and dumping their history. When the token
is not used, the auth fails with "Request denied".
## Potential risks
Keys are stored in /tmp/temporal-jwt-test/ which is world-readable. This
is acceptable for local development but the scripts should not be used
in shared or production environments.
## What changed?
Replace deprecated tctl comment with temporal cli
## Why?
tctl is deprecated
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
No. This is only local test config.
## What changed?
Deleted `"system.enableNexus"`.
## Why?
Nexus has been GA since Dec 2024.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Adds and integrates `yamlfmt` (Go-based, by Google).
Anything not annotated was changed by `yamlfmt`.
## Why?
Consistent YAML formatting. Reduce unrelated diffs caused by formatting
in PRs.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Add dynamic config to toggle standalone activity functionality.
Refactored frontend dynamic config into activity config. Removed
standalone activity dc prefix `chasm`
## Why?
We need the ability to toggle standalone activity functionality as we go
to prod. We also agreed to remove the `chasm` prefix from any chasm
originated dynamic config keys
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Change dev environment to enable Chasm
## Why?
Chasm should be enabled for development by default so devs don't have to
explicitly set the config.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Add revision number to Worker Deployment Version Data that is synced to
Task Queues.
## Why?
This allows making task queue registration also async, but more
importantly, it prevent's race conditions between concurrent
registrations, setCurrent/Ramping, drainage, and deletion.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None.
## What changed?
WISOTT
## Why?
Prevent incompatible changes in the future.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None
## What Changed
This PR introduces a new `--config-file` flag (and the
`TEMPORAL_SERVER_CONFIG_FILE_PATH ` environment variable) to remove the
dependency on `dockerize` in the Temporal server Docker image.
When a configuration file is specified using either the CLI flag or the
environment variable, the server will load configuration **only** from
that file.
Users who want templating behavior similar to `dockerize` can enable it
by adding the comment `# enable-template` at the top of the
configuration file.
---
### Key Changes
1. **New `--config-file` flag:**
* Adds a global `--config-file` flag that accepts a path to a single
configuration file (absolute or relative to the project root).
* Can also be set via the `TEMPORAL_SERVER_CONFIG_FILE_PATH `
environment variable.
2. **Deprecated legacy flags:**
* The `--config`, `--env`, and `--zone` flags are now marked as
**deprecated** in CLI help text.
* These flags still work for backward compatibility.
3. **Embedded config template:**
* The `config_template.yaml` file is now embedded in the binary to
support loading configuration from environment variables.
* Templating is supported if the file includes the `# enable-template`
comment at the top.
4. **Templating support:**
* Configuration files can use templating by including `#
enable-template` at the beginning of the YAML file.
---
### Configuration Loading Priority (Highest to Lowest)
1. **`--config-file` specified** → Load that specific file
2. **`--config`, `--env`, or `--zone` specified** → Load from
configuration directory (**deprecated**)
3. **No configuration specified** → Load from embedded template using
environment variables (default)
---
### Expected Behavior
The following examples illustrate how the new configuration loading
logic behaves:
* **Default behavior:**
Running `temporal start` without flags loads configuration from
environment variables only using the embedded template.
* **Using `--config-file`:**
`temporal --config-file=/path/to/config.yaml start` loads configuration
from the specified file path.
* **Using `TEMPORAL_SERVER_CONFIG_FILE_PATH`:**
Setting `TEMPORAL_SERVER_CONFIG_FILE_PATH=/path/to/config.yaml temporal
start` has the same effect as using the flag.
* **Validation and error handling:**
The CLI returns clear error messages when conflicting flags or
environment variables are used, or when a specified file does not exist.
---
## Breaking Change
The default behavior of `temporal start` has changed.
It now loads configuration **from environment variables** instead of
using a default template path.
---------
Co-authored-by: Alex Stanfield <[email protected]>
Co-authored-by: michaely520 <[email protected]>
Co-authored-by: Yichao Yang <[email protected]>
Co-authored-by: David Reiss <[email protected]>
## What changed?
Add custom grouping logic at replication low priority task scheduler
## Why?
To allow custom logic to group replication task.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
No risk.
## What changed?
- When a deactivated (drained/draining) version gets rollbacked to
becoming active again, the drainage information that was associated with
it will get cleared.
- This is required because when this version were to get deactivated
again, we need the code to start the drainage process on it again. Right
now, it was not doing that since we were not clearing the drainage
information which did not start a fresh drainage tracker for this
version!
*NOTE*: This change requires a patch to be present. This is because when
this scenario is replicated in production, it will result in two new
history calls (syncSummary as well as a CAN of the version workflow).
This will specifically happen when the rolled-back version gets
deactivated eventually, over
[here](https://github.com/temporalio/temporal/blob/d85e67120957a16e0c14278bfb00e80412f16f5c/service/worker/workerdeployment/version_workflow.go#L546)
- note how previously, since the drainage information was not cleared,
[this](https://github.com/temporalio/temporal/blob/d85e67120957a16e0c14278bfb00e80412f16f5c/service/worker/workerdeployment/version_workflow.go#L261)
was never executed!)
*How did I verify that this requires a patch*:
- Ran the replay test script before the fix introduced in this PR and
noticed:
<img width="793" height="79" alt="image"
src="https://github.com/user-attachments/assets/bb975d26-4ad1-4847-b477-b74c7d2f55e4"
/>
- Ran the replay test script after the fix introduced in this PR and
noticed:
<img width="911" height="99" alt="image"
src="https://github.com/user-attachments/assets/ad3ff3e1-666a-49b4-bda4-e05f3412d984"
/>
In other words, the total number of our entity workflows went up after
introducing this change and thus requires a patch gate!
## Why?
- we found a bug in the versioning entity workflows and this is an
effort to fix this!
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- yes, there is a risk of NDE's happening in production but I mitigated
that risk by placing the fix against a version gate.
## What changed?
Added support for parsing permissions from JWT claim using regular
expression
## Why?
Default JWT Claim Mapper expects permission in form `namespace:role`. If
it's not possible to configure JWT issuer to follow `namespace:role`
permissionsPattern can be set to regular expression with named groups to
parse permission. More details in issue gh-7560
## How did you test it?
- Unit tests
- Local tests
- Self hosted environment (ongoing)
## Potential risks
This change is only activated if new configuration is provided
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
WIP
## Is hotfix candidate?
No
## What changed?
Enable activity commands - remove feature flag.
## Why?
preparing for GA
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Added support for flags to restore activity options to reset activity
and update activity commands.
## Why?
Users should be able to reset/restore activity options.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## What changed?
- Enable transition history in dev env and tests
- Reset state transition count when reporting nexus operation completion to a reset run
## Why?
- Exercise transition history and state-based replication path more
often
- Make chasm development and testing easier since it relies on
transition history
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Support for text/template was added to the config loading system. Sprig
is used for some useful templating functions, as used by dockerize and
helm.
## Why?
This avoids the need to use dockerize to render config templates when
deploying in containers. Without the need to write the intermediate
config to disk, the image can be immutable. Any volumes used to copy
over an alternative template can be read-only also, enabling more secure
deployment.
## How did you test it?
Built a `FROM scratch` image using the updated temporal-server binary
and confirmed that it would work when deployed using (slightly adjusted)
version of our helm charts.
## Potential risks
None.
## Documentation
Documentation should be updated to explain this new feature, but given
it's so close to the standard deployment setup with dockerize, this
doesn't need to block release of the feature imho.
## Is hotfix candidate?
No
---------
Co-authored-by: David Reiss <[email protected]>
## What changed?
<!-- Describe what has changed in this PR -->
- Enable execution timeout timer by default
## Why?
<!-- Tell your future self why have you made these changes -->
- it's been several server releases since this feature is merged, it's
safe to enable it by default now without any backward compatibility
concerns
- Will remove the flag after next oss release if not issue reported
about it.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- Existing tests
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
- Very low. Already enabled in all production envs
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
- A better more robust replay tester covering the different API's made
possible in worker-versioning 3.1
## Why?
<!-- Tell your future self why have you made these changes -->
- To ensure that NDE changes made in these entity wf's can be caught in
the near future
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- These are tests
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
UserDataManger in child partitions constantly fetches data from the
parent partition. There is a safely mechanism to prevent busy loop in
case the parent returns result immediately instead of only when the data
is updated.
The safety mechanism did not work well with Versioning 3 because many
updates are involved in a single tests. With this update we relax the
safety check to only apply when the user data version remains the same
and continue to fetch the data asap as long as each time it gets new
data.
Here, also I'm relaxing the "requested task queue user data for version
greater than known version" errors to be thrown only for the root
partition, as they can normally happen for child partitions.
## Why?
<!-- Tell your future self why have you made these changes -->
So that back-to-back Versioning 3 APIs do not have to wait for a long
time.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Existing tests.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
Yes.
## What changed?
<!-- Describe what has changed in this PR -->
The lib version accidentally went back in a previous PR. Fixing it here.
## Why?
<!-- Tell your future self why have you made these changes -->
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
Add DrainageStatus child workflow to worker deployment system
## Why?
To periodically update the version workflow with results from
visibility.
## How did you test it?
Functional tests. Currently broken (see todo comment in code)
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
---------
Co-authored-by: Shivam Saraf <[email protected]>
## What changed?
Fixed README formatting
## Why?
It didn't render correctly, which made it confusing to read.
## How did you test it?
n/a
## Potential risks
n/a
## Documentation
✅
## Is hotfix candidate?
No
## What changed?
<!-- Describe what has changed in this PR -->
Enable Update Workflow (sync and async) by default.
Note that I'm leaving the flag and the code checks for now, in case we
need to disable it again.
## Why?
<!-- Tell your future self why have you made these changes -->
For public preview.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
- changed ES visibilityStore struct and some of its functions to public
- added namespaceRegistry, searchAttributesProvider,
searchAttributesMapperProvider arguments to custom visibility store
factory interface
- added field `closeTime` in DeleteExecutionVisibilityTask
## Why?
<!-- Tell your future self why have you made these changes -->
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
Added a new per-namespace dynamic config for allowed callback addresses.
This dynamic config is a list of maps that currently supports two
entries:
* `Pattern:string` The host:port that this rule applies to. It is a
pattern because wildcard (`*`) matching is supported.
* `AllowInsecure:bool` Indicates whether `https` is required for this
address.
Added validation when processing StartWorkflow requests to ensure that
any attached callback URLs have a matching allowed address entry and
that the scheme matches the `AllowInsecure` setting.
## Why?
<!-- Tell your future self why have you made these changes -->
To be able to white-list allowed callback endpoints.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Updated functional tests.
## What changed?
Add round robin on replication task execution
## Why?
With multiple clusters connect together, it is possible one busy cluster
flush the replication stack and cause replication lag on other clusters.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
This is a re-revert of https://github.com/temporalio/temporal/pull/5797
now that all changes moving forward will be part of 1.25 release.
## Why?
The new logic to proxy all FE history read operations to the history
service has been part of release 1.23 and it is now safe (w.r.t. FE and
history services deployment ordering) as part of release 1.25 to start
deprecating the legacy code.
Also changed the semantics of cluster info's HTTPAddress to just be the
address, and not include the scheme.
We now use the TLS configuration to derive the URL scheme.
## What changed?
<!-- Describe what has changed in this PR -->
- added EnableShadowReadMode config (default to false)
- if shadow read mode is enabled, perform shadow read from the secondary
visibility store for visibility read requests
## Why?
<!-- Tell your future self why have you made these changes -->
- to measure latency metrics from both visibility stores
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- tested locally, and verified that visibility read requests from both
visibility stores are getting reasonably same latencies
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
We had added a configuration flag to make MutableState cache a
host-level cache.
Changing dynamic config to enable this flag by default and adjusting
cache size to 128K. We found that this size is appropriate for most
workloads we see.
Also setting host-level cache size to be 8K in development dynamic
config.
## Why?
Host level cache is better for cache utilization and it prevents the
problem of a smaller shard-level cache getting full with pinned
elements.
## How did you test it?
Running the cluster with load.
## Potential risks
None
## Documentation
## Is hotfix candidate?
No
## What changed?
<!-- Describe what has changed in this PR -->
- Properly update workflow transition history.
- Only update transition history when closing transaction as active
- Only update updates to replicated application state as state
transition (NOTE: this is a different concept from the existing state
transition counter we have in executionInfo, which is simply how many
times the mutable state object got updated.)
- When sub state machine tree is dirty
- When history event got generated.
- When there's sync activity
- All other changes are either system level field update or not
replicated (e.g. buffer events or transient workflow task)
## Why?
<!-- Tell your future self why have you made these changes -->
- State-based replication rely on all clusters agree on the same
transition history. and replication task processing logic relies on the
transition history to detect missing or conflicting state transitions.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- Existing tests
- Added unit tests
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
- State transition history is behind a feature flag so no risk.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
- N/A
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
- No
- Combined all Nexus related feature flags into a single
`system.enableNexus` dynamic config.
Dynamic configs that were _removed_ (some were namespace specific):
- `frontend.enableNexusAPIs`
- `frontend.enableCallbackAttachment`
- `history.enableMutableStateTransitionHistory`
- `history.outboundProcessorEnabled`
- `component.nexusoperations.enabled`
- Added support for callback header attachment and propagation
- WIP Support for `Worker` endpoint target - localhost only
- Added dynamic config for `cass` and `sql`:
```
- component.nexusoperations.callback.endpoint.template:
- value:
http://localhost:7243/api/v1/namespaces/{{.NamespaceName}}/nexus/callback
```
- Properly handle gRPC errors coming from matching in the frontend nexus
handler
- Consider invalid Nexus responses a downstream error
- Fix panic when trying to get event ID for `StateMachineTimerTask`
(used for logging)
- Don't return `errNoRecentPoller` for nexus tasks
- Remove namespace wait in functional tests (not Nexus specific)
## What changed?
<!-- Describe what has changed in this PR -->
Added new fault injection error `ExecuteAndTimeout`.
While working on this, I also figured out that fault injection code
required refactoring and renames. I will do it in separate PR.
## Why?
<!-- Tell your future self why have you made these changes -->
`ExecuteAndTimeout` is a special error which emulates case, when client
got a `Timeout` error, but operation actually reached persistence and
was executed successfully.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Run locally.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
No risks.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
No.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## Why?
The new logic to proxy all FE history read operations to the history
service has been part of release 1.23 and it is now safe (w.r.t. FE and
history services deployment ordering) as part of release 1.24 to start
deprecating the legacy code.
## What changed?
<!-- Describe what has changed in this PR -->
- Add new timer task type for workflow execution timeout & carry over
execution timeout timer state across runs in a workflow chain.
- This change depends on
https://github.com/temporalio/temporal/pull/5531 so that only one timer
task will be created on the standby side as well.
- New behavior is controlled via a feature flag.
## Why?
<!-- Tell your future self why have you made these changes -->
- For a workflow chain, we only need one execution timeout timer task
for the entire chain. Before this change, each run in the chain will
create a run timeout timer. If user set runTimeout == execution timeout
(which is the default if only execution timeout is specified!), then the
timeout timer for all the runs will fire at the exact same time (because
run timeout will be shortened to respect the execution timeout). This
can result in a timer burst for a single workflowID.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- Added unit tests
- Tested locally with cron workflows.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
- In worst case, workflow may not timeout properly.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
- N/A, no change on behavior.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
- No.
## What changed?
<!-- Describe what has changed in this PR -->
- When workflow in the source cluster completes and starts a new run in
the same transaction. Replication should also apply that in one
transaction in standby cluster. Today this only happens when the current
workflow closes with continue as new event, but there are other cases
like cron and retry where workflow can close with
completed/timedout/failed event and also has a new run.
## Why?
<!-- Tell your future self why have you made these changes -->
- Transactional guarantee should be preserved by replication
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- New unit tests
- Run canary using global namespace.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
- Replication task may go to dlq in worst case
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
- N/A
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
- No.
zeekay
