## What changed?
Migrate `tests/acquire_shard_test.go` away from deprecated
`testcore.FunctionalTestBase` and to `parallelsuite`
Adds the supporting `testcore` plumbing the new pattern was missing:
- `WithLogger(log.Logger)` — injects a custom cluster logger via
`NewEnv` (wraps a new `WithClusterLogger` `TestClusterOption` + a
`Logger` field on `TestClusterParams`, honored in `setupCluster`).
- `WithHistoryShardCount(int32)` — sets the history shard count via
`NewEnv`.
## Why?
Part of our ongoing test flakes and migration work.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Could introduce additional flakes by parallelizing tests
## What changed?
Migrate `tests/signal_with_start_from_workflow.go` away from deprecated
`testcore.FunctionalTestBase` and to `parallelsuite`
## Why?
Part of our ongoing test flakes and migration work.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Could introduce additional flakes by parallelizing tests
## What changed?
Migrate `tests/cancel_workflow_test.go` away from deprecated
`testcore.FunctionalTestBase` and to `parallelsuite`
## Why?
Part of our ongoing test flakes and migration work.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Could introduce additional flakes by parallelizing tests
## What changed?
WISOTT
## Why?
Part of our migration away from testcore.FunctionalTestBase to
parallelsuite and TestEnv
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Tests only, could introduce flakiness
## What changed?
WISOTT
## Why?
Part of our migration to `testcore.TestEnv` for improved reliability and
performance.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA
## What changed?
This PR adds `SignalWithStartWorkflowExecution` as a synchronous Nexus
operation exposed via `__temporal_system endpoint`, allowing workflows
to signal-with-start other workflows through the CHASM Nexus operation
framework.
Key changes:
- `chasm/lib/workflow/nexus_service.go`: `workflowServiceNexusHandler`
implements the `SignalWithStartWorkflowExecution` Nexus sync operation
by resolving the namespace and delegating to the History service. A
`SignalWithStartOperationProcessor` handles input enrichment (namespace,
request ID, links) and routing via CHASM's
`NexusOperationProcessorResult`.
- `chasm/lib/workflow/library.go` — library now holds the
`workflowServiceNexusHandler`, the workflow `Config`, the SA mapper
provider, and the SA validator. `newLibrary` (used by fx) takes those
dependencies, while the public `NewLibrary` keeps its old signature for
external callers. Adds `NexusServices()` so the library registers its
Nexus service via CHASM.
- `chasm/lib/workflow/validator.go`: `RequestValidator` consolidates the
`SignalWithStartWorkflowExecution` validation logic (previously inlined
in `WorkflowHandler`) into a reusable, injectable struct. This same
validator is used by both the frontend handler and the new CHASM
processor.
- common/dynamicconfig/constants.go — adds
`EnableSignalWithStartFromWorkflow` (namespace-scoped, default false).
- `service/frontend/workflow_handler.go`: Removed the `SignalWithStart`
validation block
- `service/history/fx.go`: Provides a `HistoryServiceServerProvider` so
the CHASM workflow library can call the history handler directly.
- `temporal/fx.go`: Removes the now-redundant `ChasmLibraryOptions`
grouping; each service module registers its own CHASM libraries.
- `components/nexusoperations/workflow/commands.go`: `NotFound` and
`InvalidArgument` errors during Nexus command handling are now surfaced
as workflow task failures instead of being treated as transient handler
errors.
- `common/payloads`: Adds `EncodeSingle`, `MustEncodeSingle`, and
`MustEncode` helpers used in tests.
- `cmd/tools/getproto`: Adds support for nexus-proto-annotations proto
imports.
- `tests/signal_with_start_from_workflow_test.go`: Functional test suite
covering the happy path, duplicate detection, conflict policies, and
validation rejection for the new Nexus operation.
## Why?
This functionality is one of our most requested GitHub issues.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
The history service now directly exposes a `HistoryServiceServer`
interface via `fx` for injection into the CHASM workflow library. This
tight coupling between the CHASM workflow library and the history
handler could complicate future layering — callers outside the history
service should not adopt this pattern. The feature is gated by
`history.enableSignalWithStartFromWorkflow` for rollout.
---------
Co-authored-by: Roey Berman <[email protected]>
## What changed?
WISOTT
## Why?
Part of our migration process to testcore.TestEnv for reliability and
speed purposes.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None, test file changes only
## What changed?
WISOTT
## Why?
Part of our migration to `TestEnv` to speed up tests and reduce flakes.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Tests only
## What changed?
In the `RespondWorkflowTaskCompleted` api added a
`!ms.IsWorkflowExecutionPaused` to the gate creating a new workflow
task, without this check the workflow will schedule new work while
keeping `PauseInfo` on the workflow making it show up as `Paused` in the
UI but unable to be unpaused.
## Why?
Fix#10239
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
Minimal, pause is behind a dynamic config that is defaulted to false
## What changed?
Increasing the "short context" in workflow update tests where short
contexts were consistently erroring out with `context deadline exceeded`
in CI, causing flakes.
For `TestSpeculativeWorkflowTask_QueryFailureClearsWFContext`: the two
`QueryWorkflow` callers share a context that will cancel the shared
context when one of the queries returns failed and clears the workflow
context. The query buffer error will clear the workflow context and
clear the update registry. Then poll the workflow again to validate the
update is sent on a new speculative workflow task, this last portion is
unchanged.
## Why?
Reduce flakiness.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA, tests only.
## What changed?
Add `WithTags(metrics.CommandType(Unspecified))` to the call sites for
Workflow Update event_blob_size metric emission, which was missing in
the pathch #10253 to the problem introduced in #10223
## Why?
Fixes a warning and dropped metric when the metrics handler has
different shapes
## How did you test it?
- [ ] built
- [X] run locally and tested manually: Ran server with `make start`,
created a simple workflow with an update handler, start the worker,
start a workflow, and send the update. Grepped server logs for `error in
prometheus reporter` and `event_blob_size` and curled the metrics
endpoint `curl -s http://127.0.0.1:8000/metrics > /tmp/snapshot.txt`
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Minimal, fixing bug
## What changed?
Use a metrics handler without `header_callsite` tag because Prometheus
rejects re-registering the same metric with a different label set and
logs `error in prometheus reporter ... event_blob_size ... has different
label names`.
## Why?
Fix found from regression
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Fixes a regression introduced in #10223
## Summary
- Emit the `EventBlobSize` for `UpdateWorkflowExecution` requess, tagged
with `namespace`.
---------
Co-authored-by: Hanzo Dev <[email protected]>
## What changed?
Two test fixes targeting consistent CI failures across all recent runs,
plus a migration of `NexusApiTestSuite` to `parallelsuite`.
## Why?
Both failures reproduced in 9–10/10 of the last 10 CI runs:
**1.
`TestNexusStartOperation_WithNamespaceAndTaskQueue_SupportsVersioning`**
(10/10 runs)
The test calls `UpdateWorkerBuildIdCompatibility`, which is the v0.1
(Version Set / Build ID compat) API gated by
`FrontendEnableWorkerVersioningDataAPIs`. The test only enabled
`FrontendEnableWorkerVersioningRuleAPIs` (the v0.2 Rules API), so the
v0.1 call was rejected with a `PermissionDenied` error.
**2. `TestQueryWorkflow_NonStickyMultiPageHistory`** (9/10 runs)
The test asserted `NextPageToken != nil` after fetching workflow
history. The shared functional test cluster sets
`SendRawHistoryBetweenInternalServices=true` (in
`tests/testcore/dynamic_config_overrides.go`), which causes the history
service to return all events in one shot — making `NextPageToken` always
nil regardless of the `MatchingHistoryMaxPageSize=2` override.
## How did you test it?
**Fix 1 — nexus versioning test (`tests/nexus_api_test.go`):**
Added
`env.OverrideDynamicConfig(dynamicconfig.FrontendEnableWorkerVersioningDataAPIs,
true)` alongside the existing `RuleAPIs` override so the v0.1 API is
available for the test.
Also migrated `NexusApiTestSuite` from the custom `NexusTestBaseSuite`
(which used a `useTemporalFailures` field mutated before `suite.Run`) to
`parallelsuite.Suite[*NexusApiTestSuite]`. The `useTemporalFailures`
flag is now a typed parameter passed through `parallelsuite.Run`,
eliminating shared mutable state between the two suite invocations.
**Fix 2 — query pagination test (`tests/query_workflow_test.go`):**
Extracted `TestQueryWorkflow_NonStickyMultiPageHistory` from
`QueryWorkflowSuite` into a standalone top-level test that uses
`testcore.NewEnv` with:
- `testcore.WithDedicatedCluster()` — isolated cluster not affected by
the global config override
-
`testcore.WithDynamicConfig(dynamicconfig.SendRawHistoryBetweenInternalServices,
false)` — disables raw history so pagination happens normally
- `testcore.WithDynamicConfig(dynamicconfig.MatchingHistoryMaxPageSize,
2)` — forces multi-page history
- [x] built
- [x] added new functional test(s) (standalone query test)
- [x] covered by existing tests (nexus suite)
## Potential risks
The `NexusApiTestSuite` migration to `parallelsuite` changes how
`useTemporalFailures` is threaded through tests. The two existing
top-level runners (`TestNexusApiTestSuiteWithLegacyErrorPaths` /
`WithTemporalFailures`) are preserved; the flag is now a
compile-time-checked parameter rather than a struct field.
## What changed?
Added a `SendToChannel(ctx, ch)` helper to `FunctionalTestBase`,
mirroring the existing `WaitForChannel`, and replaced all bare `ch <-
struct{}{}` sends in functional test bodies with calls to it.
## Why?
Bare channel sends in test bodies silently hang when the receiving
goroutine has stalled or died, causing tests to block until the overall
test timeout rather than failing fast with a useful message.
`WaitForChannel` already existed to solve this for receives,
`SendToChannel` closes the gap for sends.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None, this is a pure test-helper addition.
## What changed?
1. The report was flagging commits that fixed a flake as culprits
despite showing fewer failures, this change fixes false positives by
making sure the commit caused an increase in flakes
2. Sort the table by bayesian probability instead of overall failure
rate.
New run is here:
https://github.com/temporalio/temporal/actions/runs/24151267149
## Why?
The Bayesian model in runBisect is direction-agnostic: it identifies
statistical transition points where the failure rate changes
significantly, but treats a 10%→0% transition identically to a 0%→10%
one. Commits that fixed a flake therefore appeared as high-confidence
"suspects", generating misleading and unactionable report entries.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Minimal, still getting confidence in this reporting method.
## What changed?
Four subtests in TestStandaloneActivityTestSuite/TestRequestCancel —
StaleToken, StaleAttemptToken, MismatchedTokenNamespace, and
MismatchedTokenComponentRef — now each create their own fresh
context.WithTimeout(t.Context(), 10*time.Second) instead of reusing the
parent test's shared context.
Why?
TestRequestCancel created a single 10-second context at the top of the
function and shared it across all subtests. The subtests run
sequentially; the earlier ones each do a PollActivityTaskQueue call (a
long-poll) plus additional RPC round trips. On slower CI runners (e.g.
mysql8 on ARM), the combined time consumed by the earlier subtests left
less than MinLongPollTimeout (2 seconds) in the shared context by the
time the four failing subtests started. The server correctly rejects
poll requests whose context deadline is below MinLongPollTimeout,
returning "Context timeout is too short.", which caused these subtests
to fail intermittently in CI.
Failure run here:
https://github.com/temporalio/temporal/actions/runs/24060174422/job/70177640247
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None, this is test-only
## What changed?
Added a Bayesian commit bisect analysis to the `flakereport` tool that
automatically identifies which commits most likely introduced test
flakiness. Inspired by this Python tool:
https://github.com/hauntsaninja/git_bayesect
Sample run of the report can be found
[here](https://github.com/temporalio/temporal/actions/runs/23914956307).
This change uses the set of test runs which are mapped to commit history
to infer with Bayesian statistics the probability that a commit
introduced a flaky test or tests. There is a fuller explanation in
`doc.go`. A tl;dr: The algorithm ranks each commit (as a transition) in
the history by assigning equal p_before and p_after and using a
Beta-Binomial marginal likelihood to determine the posterior
probability. There is a heuristic to lower the prior probability for
commits that only touch documentation or ci configuration.
## Why?
This can help us identify the actual commit that introduced a problem
and also help us get commit authors to validate their work instead of
engineers without all of the context for a change.
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Minimal, reporting only
## What changed?
`StartBatchOperation` with a malformed visibility query fails
immediately with `InvalidArgument`, today this will start a batch
operation that retries forever. There are two changes here:
* Validate the query at the frontend, if malformed return
`InvalidArgument`
* In `fetchPage` and in the `CountWorkflow` estimate call
`InvalidArgument` errors are wrapped in
`temporal.NewNonretryableApplicationError`. This protects previous
submissions from continuing to retry.
## Why?
When a batch job is started with a malformed visibility query, the batch
activity received an `InvalidArgument` error from the visibility store
but returned it as a plain Go error. The Temporal SDK wrapped this as a
retryable `ApplicationError`, causing the activity to retry forever (no
`MaxAttempts`, 20yr `StartToCloseTimeout`). `DescribeBatchOperation`
always returned `RUNNING` — the job would never complete or fail on its
own. The only recovery was a manual `StopBatchOperation` call.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
Addresses #9782
## Potential risks
The sqlparser.Parse runs on every `StartBatchOperation` but it is a pure
in-memory parse, it's used in the SQL parser later on so this does run
twice but it gives the ability to fail quickly. The activity-layer fix
covers the same issues as a backstop but changing activity code always
presents a danger.
## What changed?
Remove guards to allow breaking changes in two files.
## Why?
Safety.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA
## What changed?
Fix flaky test
`TestUpdateWithStartSuite/TestWorkflowStartConflict/workflow_id_conflict_policy_fail:_use-existing`
handling both possible orderings of the race condition it tests.
## Why?
The test injects a hook (`UpdateWithStartInBetweenLockAndStart`) that
fires a concurrent `StartWorkflowExecution` between the
update-with-start lock acquisition and start attempt to simulate a race
condition. The test assumes the update always lands in a second
speculative WFT:
1. WFT #1 polled with no messages → completed (empty)
2. UWS retry creates speculative WFT #2 → polled with update →
accepted/completed
This assumption broke after the `parallelsuite` migration, which gives
each test its own isolated `testcore.NewEnv` running in parallel. With a
dedicated env, the retryable history client's immediate retry of the
Unavailable error races tightly with `RecordWorkflowTaskStarted`:
- Ordering A (retry wins the lock first): the update is admitted while
WFT #1 is still scheduled, so it attaches to WFT #1's messages when the
worker polls it. No WFT #2 is ever created.
- Ordering B (RecordWorkflowTaskStarted wins): WFT #1 starts and
completes with no messages, then the retry runs with no pending WFT,
creating speculative WFT #2 with the update.
The original two-poll design panicked in ordering A (index out of range
[0] with length 0 on task.Messages[0] in the empty-response first poll).
The initial single-poll fix panicked in ordering B for the same reason.
Neither ordering is
guaranteed, so the test must handle both.
## How did you test it?
- [ ] built
- [X] run locally and tested manually - tested with `-count=50` locally
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None, test only
## What changed?
Replace plain `int` with `atomic.Int32` in two VersionWorkflowSuite unit
tests.
## Why?
CI caught this as a crash in the unit test job
Failure
[here](https://github.com/temporalio/temporal/actions/runs/23856744785)
```
WARNING: DATA RACE
Read at 0x00c000b005e8 by goroutine 320:
go.temporal.io/server/service/worker/workerdeployment.(*VersionWorkflowSuite).Test_MultipleSyncStates_BlocksCaNUntilAllComplete.func2()
version_workflow_test.go:1240 +0x2c
Previous write at 0x00c000b005e8 by goroutine 326:
...version_workflow_test.go:1240 +0x3c
```
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA
## What changed?
WISOTT
## Why?
Safety and runtime increases from parallelizing tests.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA
## What changed?
Add new review guidelines for test reliability and fix instances found
in the current codebase.
## Why?
I have found these to be the most common patterns across the last two
weeks of targeting flaky tests.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Minimal
## What changed?
* Upgrade the `actions/create-github-app-token` and `actions/setup-go`
because of deprecations notices for NodeJS
* Change the test summary to `!cancelled` so successful runs will still
show the failed tests in Summary.
## Why?
Deprecations warnings for the first bullet point. For the second bullet
point it's helpful to see the flaky tests on a successful run.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Tests only
## What changed?
testrunner writes intermediate JUnit XML reports after each completed
test attempt and enforces its own total deadline derived from the go
test -timeout flag.
## Why?
When a test run exceeds the GitHub Actions job timeout (40 min), the
entire process group is killed before the testrunner can write its final
merged XML, producing a synthetic junit.crash.xml showing
`functional-test (crash)`, with no visibility into actual test failures.
[Sample
run](https://github.com/temporalio/temporal/actions/runs/23567876357)
that actually had 108 failures on manual inspection
There are two distinct failure modes this addresses:
1. Killed between attempts — attempt 1 completes with failures, the
process is killed during attempt 2. Previously: no XML on disk. Now:
attempt 1's results survive in the intermediate file.
2. Go test timeout doesn't fire — the test binary hangs past its own
`-timeout=35m` (e.g. deadlocked goroutines). Previously: GitHub kills at
40 min with no XML. Now: the testrunner enforces the same 35m deadline
itself, kills `gotestsum`, extracts any `--- FAIL:` lines captured from
stdout, writes a report, and exits cleanly within the 5-minute buffer
before GitHub's kill.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
The testrunner's context deadline fires at the same time as go test's
own -timeout. If both fire simultaneously, parseTestTimeouts (existing
path) and the new `ctx.Err()` check (new path) can both trigger. The
`ctx.Err()` check is guarded to only activate if the normal
timeout-panic path didn't already break the loop, so there is no
double-processing.
## What changed?
1. tests/task_queue_stats_test.go — Remove the `t *testing.T` field from
`taskQueueStatsSuite` and replace all `s.t` usages with `s.T()`.
Critically, fix `validateDescribeWorkerDeploymentVersion` to use
`a.NoError(err)` instead of `require.NoError(s.T(), err)` inside the
callback.
2. tests/xdc/history_replication_signals_and_updates_test.go — Fix
`pollWorkflowResult` to handle errors from `GetWorkflowExecutionHistory`
gracefully: return the error from the inner function, retry with a nil
page token on transient errors (e.g. CurrentBranchChanged after conflict
resolution), and abort cleanly when the context expires.
## Why?
TestTaskQueueStats panic: Calling require.NoError(s.T(), err) inside an
EventuallyWithT callback is unsafe. EventuallyWithT runs its callback in
a separate goroutine on a ticker. If the outer test context expires and
the test completes while this callback is still running, the
require.NoError call invokes t.Fail() on a completed test, which panics
the entire test binary with panic: Fail in goroutine after TestXxx has
completed. The fix uses the CollectT-scoped a.NoError(err) so assertions
are buffered and safe.
TestConflictResolutionGetResult hang/crash: pollWorkflowResult was
calling responseInner.History.Events without checking if responseInner
was nil. When GetWorkflowExecutionHistory returns an error (e.g.
CurrentBranchChanged after conflict resolution resets the branch token),
the response is nil, causing a nil pointer dereference that panics the
goroutine. Because the goroutine crashed before sending to
workflowResultCh, the main test goroutine blocked on <-workflowResultCh
for the entire 30-minute CI timeout. The fix returns the error from the
inner function and retries with a nil page token on transient errors,
which allows the poll to succeed on the updated branch.
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
The XDC fix changes `pollWorkflowResult` to retry indefinitely on any
non-context error. If a non-transient error were to occur repeatedly,
this could loop until the context expires rather than failing
immediately. This is acceptable since the context timeout provides a
hard bound, and the test failure message via `c.t.s.NoError(ctx.Err(),
...)` will make the root cause clear.
## What changed?
Add instrumentation for the activity operator commands which previously
had no metrics. `activity_targeting_method` is `id` when targetting a
specific ID, else `type`. UpdateOptions also has a structured log line
for changed fields.
## Why?
Add visibility into the usage of the activity operator commands.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Minimal, cardinality is low for these
## What changed?
WISOTT
## Why?
Our CI changes the working directory to the `tests/` directory and this
causes the relative path to resolve differently.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None
## What changed?
Fix four async poller helpers in `versioning_3_test.go` to avoid calling
`s.NoError()` inside goroutines that can outlive the test:
- `doPollWftAndHandle `
- `pollWftAndHandleQueries`
- `pollNexusTaskAndHandle`
- `doPollActivityAndHandle`
Each helper had this pattern: a closure `f()` that called
`s.NoError(err)`, then in the async path the closure was invoked from a
goroutine. Moving `s.NoError `out of the closure into the sync-only path
eliminates the assertion-in-goroutine risk.
## Why?
These goroutines poll with up to 30–60 second timeouts, but the
enclosing tests use 15-second contexts. When the test context expires
and the test completes first, the goroutine is still running. When its
poll eventually times out and it calls s.NoError(err), Go panics:
`panic: Fail in goroutine after
TestVersioning3FunctionalSuiteV2/TestWorkflowWithPinnedOverride_NoSticky
has completed`
This crash kills the entire test binary, causing `gotestsum` to write
only `junit.crash.xml` with no individual test results. Observed in CI
run #23502599693 on multiple backends (sqlite shard2, postgres12_pgx
shard0, cass_os3 shard0).
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None, test only change.
## What changed?
Fix `TestDescribeWorkerDeployment_TwoVersions_Sorted` in
`TestWorkerDeploymentSuiteV2`, which was consistently failing with
`build_id_1` at index 0 instead of `build_id_2`.
Sample failure:
https://github.com/temporalio/temporal/actions/runs/23466355731/job/68279366267
## Why?
In V2 (VersionDataRevisionNumber) mode, each version's `CreateTime` is
derived from the deployment workflow's `workflow.Now(ctx)` when the
registration update is processed — unlike V0, where each version has its
own version workflow that independently sets its `CreateTime` via a sync
signal.
The old test waited for a wall-clock `time.Since(startTime) > 1ms`
before starting the second poller. However, this didn't guarantee that
the deployment workflow had actually processed `build_id_1`'s
registration before `build_id_2`'s registration arrived. If both
`RegisterWorkerInWorkerDeployment` updates reached the deployment
workflow within the same Temporal workflow task millisecond, both
versions received identical `CreateTime` values. The `sort.Slice` in
`deploymentStateToDeploymentInfo` was a no-op with equal timestamps, so
the result depended on Go's map iteration order — which consistently
returned `build_id_1` before `build_id_2`, producing an ascending
(wrong) result. This explains the consistent, non-flaky failure.
V0 is unaffected because version workflow `CreateTimes` are set at each
version workflow's own start time (separate workflows with distinct
history clocks), which are always different.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None. The change only affects test setup ordering.
`ensureCreateVersionInDeployment` is already used throughout the test
suite for exactly this purpose and has a 1-minute timeout, well within
the test's 30-second overall context.
## What changed?
WISOTT
## Why?
Data race was detected in a [functional
test](https://github.com/temporalio/temporal/actions/runs/23487536498/job/68347191553).
In the body of the `MutableStateToGetResponse` function is this comment:
```
// NOTE: fields of GetMutableStateResponse (returned value of this func)
// are accessed outside of workflow lock, and, therefore,
// ***MUST*** be copied by value from mutableState fields.
```
VersioningInfo is a proto pointer so it must be cloned, not aliased.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
`common.CloneProto` handles nil safety so very minimal risks
## What changed?
Functional tests write debug-level logs to a file rather than inline in
the job. Info and above are still visible and the debug logs are
available for download at the end of each test job.
## Why?
Debug level logs make parsing the functional test run logs very
difficult, sometimes too large to even download the logs. This makes
debugging flaky test runs even more difficult.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
- [X] manually verified in CI
## Potential risks
Minimal, debug logs are still available to download and parse. If there
are logs that are missing for debugging we should move those from
`debug` to `info` level in the future
## What changed?
The test previously used `InactiveChannelDeletionDelay: 0` with the
default `RealTimeSource`, and did not assign `s.scheduler.ts = s.ts`.
Two changes fix this:
1. Set `s.scheduler.ts = s.ts` to use the suite's `EventTimeSource`
instead of `RealTimeSource`
2. Change `InactiveChannelDeletionDelay` from `0` to `time.Nanosecond`,
and advance fake time by `2 * time.Nanosecond` after each
`taskWG.Wait()` to trigger cleanup asynchronously
## Why?
With `InactiveChannelDeletionDelay: 0` and `RealTimeSource`,
`cleanupLoop` called `time.NewTimer(0)` which fires instantly, infinite
tight loop that continuously allocated OS timer objects and acquired the
scheduler's write lock.
Dispatcher busy-wait: `numInflightTask` is manually set to 1 to force
the weighted-dispatch path. This makes `hasRemainingTasks()` permanently
true, so `dispatchTasksWithWeight`() never returns to the event loop's
select. The dispatcher spins indefinitely.
Sample:
https://github.com/temporalio/temporal/actions/runs/23311641483/job/67800610136
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Unit test changes only
## What changed?
WISOTT
## Why?
Codifying the interaction between workflow unpause w/ paused activities.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
Test only
## What changed?
Implements `DeleteActivityExecution` for standalone activities.
Implements the `Activity.Terminate()` function that was a no-op. Add the
`DeleteActivityExecution` RPC to the activity service proto, frontend
and internal handler. Delete will terminate the activity if it is still
running then schedule it for deletion.
## Why?
Necessary for standalone activities, maintain parity with workflows.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
Minimal, not enabled by default yet
## What changed?
Fix a panic in `TestTaskQueueStats_{Classic,Pri}_Suite` when tests run
in parallel. The outer `t *testing.T` was passed to the suite and a
panic occurred outside of the testing context.
Sample: https://github.com/temporalio/temporal/actions/runs/23262976802
## Why?
Previously enabled parallel execution which caused this bug more
frequently.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Test only
## What changed?
WISOTT
## Why?
Failed in a previous test run:
https://github.com/temporalio/temporal/actions/runs/23258145881/job/67619316218
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA, tests only
## What changed?
When a workflow is reset, `ApplyWorkflowExecutionStartedEvent`
re-registers the start-event callbacks using the reset operation's
request ID. `HandleNexusCompletion` cannot find a matching
`BufferedStart` and discards the completion. The `resetRequestID` param
is removed from `WorkflowResetter.ResetWorkflow` and the original
request ID is used. `findStartRequestID` reads the original request ID
back from `WorkflowExecutionInfo.RequestIds` by finding the
`EVENT_TYPE_WORKFLOW_EXECUTION_STARTED` entry.
## Why?
CHASM scheduler relies on callback `request_id` to match WF completions
to originating `BufferedStart` entries. When it cannot be found the
scheduler is permanently stuck with the workflow marked as still
running.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
-
`TestScheduledWorkflowDoubleReset_SchedulerSeesCompletion_{HSM,CHASM}Callbacks`:
create a schedule, trigger immediately, reset the workflow twice, signal
the completion to complete, poll `ListSchedules` until scheduler shows
`COMPLETED`.
## Potential risks
CHASM scheduler has not been enabled in production yet, the blast radius
should be minimal.
## What changed?
WISOTT
## Why?
Part of a piece by piece migration of functional tests to use `TestEnv`
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Tests only
## What changed?
WISOTT
## Why?
Panics after the sub test finished were causing failures and CI issues
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Test only
## What changed?
WISOTT
## Why?
Data race is causing CI to break
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA
## What changed?
WISOTT
## Why?
Planned migration away from `FunctionalTestBase` to `TestEnv`
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Tests only
## What changed?
Fix a race condition in GetWorkflowExecutionHistory that caused SDK
workers to receive incomplete history and fail with "premature end of
stream". On the last page of a paginated `GetWorkflowExecutionHistory`
response, re-query mutable state to detect events that were committed to
the DB between the first and last page fetches. If `freshNextEventId >
continuationToken.NextEventId`, the gap is fetched from persistence and
appended to the response before transient/speculative events are added.
The continuation token is updated with the fresh boundary so
`appendTransientTasks` validates against the correct `NextEventId`. If
the re-query itself fails, the request returns an error so the client
retries.
Also adds a `nil` check in `ValidateTransientWorkflowTaskEvents`,
preventing a possible nil-pointer dereference.
## Why?
A speculative WFT is converted to normal (e.g., by an incoming signal),
committing 1–2 new events. The continuation token from page 1 points to
NextEventId=8; the DB range [6, 8) on page 2 returns only events 6–7,
missing the newly-committed events 8 and 9. `appendTransientTasks` finds
no transient tasks (speculative was committed), so the assembled history
is missing 2 events.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
- The re-query on the last history page adds one extra GetMutableState
RPC per paginated `GetWorkflowExecutionHistory` call. This is bounded to
final-page responses only and the existing path already made this call
inside `appendTransientTasks`, so the net overhead is one additional
call specifically when a gap is detected.
- Returning an error when the fresh mutable-state re-query fails changes
the previous behavior of silently continuing. Clients will retry, which
is correct, but retry storms are possible if persistence is consistently
unavailable mitigated by the client's existing backoff.
## What changed?
Add a default 90s timeout to individual tests using the
`testcore.NewEnv` for setup. Tests should use the `context.Context`
returned from `testcore.NewEnv(...)` to fail early when a test exceeds
the timeout.
## Why?
Go only allows to set a timeout for an entire test run via `go test`,
but a single test case can make the entire run be stuck for the full
test timeout time. This change will make those tests exit early speeding
up the dev process.
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Test change only, could introduce flakes if tests did not have previous
context timeouts.
---
> [!NOTE]
> **Medium Risk**
> Touches shared test infrastructure and changes how contexts/timeouts
are created and propagated across functional tests, which could
introduce new test flakes or premature cancellations if timeouts are too
aggressive.
>
> **Overview**
> Functional tests created via `testcore.NewEnv` now get a *test-scoped*
timeout context (default **90s**, configurable via
`TEMPORAL_TEST_TIMEOUT` or per-test `testcore.WithTimeout`) that is
canceled on timeout and fails the test during cleanup.
>
> `testcore.NewContext` is updated to optionally derive from a parent
context, and key functional tests are migrated to pass
`s.Context()`/`env.Context()` into RPC calls and
`context.WithTimeout(...)` to respect cancellations. Linting/docs are
updated to discourage `context.Background()` in tests and document the
new timeout knob.
>
> <sup>Written by [Cursor
Bugbot]() for commit
98852a5e43d62aec769af4caae7de9783f314065. This will update automatically
on new commits. Configure
[here]().</sup>
## What changed?
Re-does #9138 which was incidentally merged.
Include transient and speculative WFT events in
`GetWorkflowExecutionHistoryReponse` response, unless UI or CLI made
request.
* Adds `transient_or_speculative_events` back to
`GetMutableStateResponse`
* Reserve `transient_workflow_task` in `HisotryCOntinuation` token
* Add validation helpers
* Add query-compare-query for transient events at request start and end
Re-implements #7732
## Why?
Fix "premature end of stream" errors when workers request history after
cache eviction w/ transient/speculative workflow tasks present. This
adds transient & speculative WFT events in `GetWorkflowExecutionHistory`
(already in `PollWorkflowTask`). Worker cache eviction w/ speculative
workflow tasks causes the expected and actual event counts to be
different. #7732 passed transient events through continuation tokens,
which could become stale during pagination. This PR implements mutable
state querying at both start and end of pagination and compares
transient event IDs to detect if WFT state changed during pagination and
return a retryable error.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Same risks from #7732
## What changed?
* Convert the flake report we use to monitor functional test reliability
from invoking the `tringa` cli tool and a Python script to only a Go
script that fetches the artifacts, parses the JUnit, and creates the
report.
* This also fixes an issue where `tringa` only found the last 20 tests,
the Go script will iterate through all failures in the last `N` days
## Why?
Everything is in a single executable making it easier to test and
extend.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Minimal, this is only a tooling change.
## What changed?
Removed temporary ignores from buf.yaml related to specific proto files.
## Why?
Catch proto changes
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA
## What changed?
Remove `invalid_state_transition_workflow_update_message` and
`workflow_update_registry_size_limited` metrics. Add the `namespace` tag
to the logs/softasserts in the `instrumentation` methods the metrics are
used for.
## Why?
The logs will give us the same information and the namespace without
cardinality concerns. These metrics are not expected to fire frequently.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Minimal, metrics changes only.
## What changed?
WISOTT
## Why?
A few tests have issues not yielding in the allotted 40 minute test
time, causing timeouts and CI failures. This addresses two of those.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA, tests only
## What changed?
Include transient and speculative WFT events in
`GetWorkflowExecutionHistoryReponse` response, unless UI or CLI made
request.
* Adds `transient_or_speculative_events` back to
`GetMutableStateResponse`
* Reserve `transient_workflow_task` in `HisotryCOntinuation` token
* Add validation helpers
* Add query-compare-query for transient events at request start and end
Re-implements #7732
## Why?
Fix "premature end of stream" errors when workers request history after
cache eviction w/ transient/speculative workflow tasks present. This
adds transient & speculative WFT events in `GetWorkflowExecutionHistory`
(already in `PollWorkflowTask`). Worker cache eviction w/ speculative
workflow tasks causes the expected and actual event counts to be
different. #7732 passed transient events through continuation tokens,
which could become stale during pagination. This PR implements mutable
state querying at both start and end of pagination and compares
transient event IDs to detect if WFT state changed during pagination and
return a retryable error.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Same risks from #7732
## What changed?
Remove `(retry 1/2)` from test name for more accurate reporting.
## Why?
More accurate reporting.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Reporting only
## What changed?
Increase the time out and retries for TestTransientWorkflowTaskTimeout
which has been flaking consistently. Ran this 50 times on my local
machine and had 0 failures.
## Why?
Prevent future flakes.
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Slightly longer test times.
## What changed?
WISOTT
## Why?
Cleaner report
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None, reporting only
## What changed?
WISOTT
## Why?
Help reduce the friction in our CI
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
No
## What changed?
Reduce `dynamicconfig.HistorySizeSuggestContinueAsNew` from 2 MB -> 10KB
for a flaky test.
## Why?
Hopefully reduce flakiness of test
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA, test only
## What changed?
Added `Unwrap` function to `updateError` which will preserve the
`serviceerrors.ResourceExhausted` error in the
`serviceerror.MultiOperationExecutionError`
## Why?
Preserving error details
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
No, giving more information
## What changed?
Made the "unable to find any scheduled or completed task" error
non-retryable in the batcher activities by converting it to a
`temporal.NonRetryableApplicationError` with type "NoWorkflowTaskFound".
## Why?
This error is a workflow with no task history, which will not resolve
with retries.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Low risk, should only affect workflows that already cannot be processed.
## What changed?
Improving test coverage for degraded wft visibility
## Why?
Missed comments in #8769
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Tests only
## What changed?
When buffered events are applied, like when applying signals after
workflow task failures occur, preserve the workflow task attempt so the
`TemporalReportedProblems` search attribute is still properly added.
## Why?
Without this change the `TemporalReportedProblems` search attribute will
never be added if a workflow is consistently queried.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
Minimal, this adds a new int to the workflow task execution, but it's
only used in one place.
## What changed?
Added TemporalPauseInfo search attribute updates for workflow pause and
unpause operations.
Implementation (service/history/workflow/mutable_state_impl.go):
- Added `buildWorkflowPauseInfoEntries()` and
`buildActivityPauseInfoEntries() `helper functions
- Refactored `updatePauseInfoSearchAttribute()` to combine workflow and
activity pause entries
- Updated `ApplyWorkflowExecutionPausedEvent()` and
`ApplyWorkflowExecutionUnpausedEvent()` to call search attribute update
## Why?
Enables querying and filtering paused workflows using the
`TemporalPauseInfo` search attribute, a requirement for M0.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
Minimal, visibility will have another search attribute changing.
## What changed?
Changed `DecodeWorkerDeploymentMemo` to return an error instead of nil
on decode failures. Added error handling in `ListWorkerDeployments` to
log and skip deployments with corrupted memos.
## Why?
Production nil pointer panic at client.go:610 when
`DecodeWorkerDeploymentMemo` silently returned nil on decode errors.
Previously only checked if memo exists, not if decode succeeded.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
List operations will silently skip deployments with malformed memos
instead of failing the entire request. Error is logged for debugging.
## What changed?
Bumping `go.temporal.io/api` to `v1.60.0`
## Why?
Preparing for release
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Minimal
## What changed?
Alert company slack when the main build fails after a PR merge
## Why?
Help us combat our issues with flaky tests.
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
No, not running in production
## What changed?
Use the `commonnexus.PayloadSerializer` for deserializing the nexus
payloads. This is applied to both CHASM and HSM
## Why?
Fixes a bug with payload deserialization in both CHASM and HSM
implementations.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Minimal, this is not enabled out for CHASM and was an existing bug for
HSM impl.
## What changed?
Add `WorkflowPause` to `NamespaceCapabilities` for future UI work
## Why?
Will be used on the UI side
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA
## What changed?
unit tests have been timing out recently at 20 minutes but in the
`Makefile` we use a 35 minute timeout. This PR matches the Makefile
default and allows for an extra five minutes so the test-runner program
can properly cleanup and provide output on why the timeout occurred.
## Why?
Aligning withe the `Makefile`
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Longer unit-test runs
When processing callbacks in mutable state use a read only context to
validate there are callbacks to process, if there are get the writeable
context to process the callbacks.
## What changed?
Use a read context to check for callbacks
## Why?
Removes a write of the root workflow component persisted / replicated
upon close.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA, not rolled out for users yet
## What changed?
Implement `LifecycleState` method for chasm.Callback which was always
showing Running
## Why?
Addresses a TODO
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA
## What changed?
Remove duplicated `DestinationDownError`
## Why?
Proper error handling
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA
## What changed?
Integrated chasm/lib/callback into MutableState and updated
callback-related APIs to use CHASM implementation. Added integration
tests following the scheduler pattern.
## Why?
Porting callback functionality from HSM to CHASM as part of the ongoing
HSM-to-CHASM migration effort.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
This will need to be tested more, but is not being turned on with this
PR
---
> [!NOTE]
> Integrates CHASM-based callbacks into mutable state and history APIs,
adds configs and error types, updates routing/state transitions, and
includes migration tests.
>
> - **CHASM Callback Integration**:
> - Add CHASM callback module (FX wiring, HTTP caller provider, request
routing) and metrics shared with HSM implementation.
> - Implement `ToAPICallback`, invocation flow changes, and refined
error wrapping.
> - **History/MutableState**:
> - Expose `ChasmEnabled`, `ChasmWorkflowComponent{,ReadOnly}`; use
CHASM for adding/processing completion callbacks when enabled.
> - Describe API now builds callback info from both CHASM and HSM trees.
> - Continue-as-new/retry path aggregates callbacks from both
implementations.
> - **State Machine/Executors**:
> - Switch to `queues/errors` types; move `NamespaceIDAndDestination` to
`queues/common`.
> - Adjust transitions: generate `BackoffTask` with scheduled time;
include destination on reschedule; no tasks on success/fail.
> - Simplify retry result (policy passed separately) and validation
signatures.
> - **Config**:
> - Add `EnableCHASMCallbacks` and `MaxCHASMCallbacksPerWorkflow`; wire
through history configs.
> - **Tests**:
> - Add migration and CHASM-enabled functional tests; update unit tests
to new error/types and behaviors.
>
> <sup>Written by [Cursor
Bugbot]() for commit
e3b51a931d3ea2dd7f18fb2f9c55fc880ba828d8. This will update automatically
on new commits. Configure
[here]().</sup>
---------
Co-authored-by: Roey Berman <[email protected]>
## What changed?
Set `system.numConsecutiveWorkflowTaskProblemsToTriggerSearchAttribute`
to 5 by default following full rollout.
## Why?
validated in latest rollout
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
No, this has already been rolled out
## What changed?
Fix the slack report formatting, removed the emojis from the response,
and added a reporting of the total failures.
## Why?
Better reporting
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Not production
## What changed?
- Removed duplicate workflow context tags (`wf-namespace-id`, `wf-id`,
`wf-run-id`) in
`service/history/workflow/mutable_state_impl.go:6034-6064`
## Why?
- `MutableStateImpl.logger` is wrapped with `LazyLogger` in
[`context.go:48-71`](https://github.com/temporalio/temporal/blob/main/service/history/workflow/context.go#L48-L71)
that automatically adds these tags
- Explicit tag additions in log calls caused duplication because
`withLogger.prependTags()`
([`with_logger.go:27-33`](https://github.com/temporalio/temporal/blob/main/common/log/with_logger.go#L27-L33))
does not deduplicate
- Deduplication only works with the `.With()` method
([`zap_logger.go:214-227`](https://github.com/temporalio/temporal/blob/main/common/log/zap_logger.go#L214-L227))
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
- [X] Verified logger chain: `ContextImpl` → `LazyLogger` wrapper →
`MutableStateImpl`
## Potential risks
- None. Tags still present via `LazyLogger`, just not duplicated
## What changed?
add the `WorkflowID` to the query to isolate tests run in parallel
## Why?
Fix flaky tests issue
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
No, test changes only
## What changed?
`fx.In` shouldn't be embedded in the `{Invocation,Backoff}TaskExecutor`
because these are output parameters.
## Why?
These will cause tests to fail once mutable state is integrated.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
No
## What changed?
CHASM library registration for callbacks
## Why?
pre-cursor to integration w/ mutable state
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None, not in running code yet.
## What changed?
Adding an `InvocationTaskExecutor` and `BackoffTaskExecutor` to
`chasm/lib/callback`
## Why?
Second step of migrating callback from HSM -> CHASM
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None, this is not integrated.
## What changed?
Migrating the `component/callbacks/statemachine{_test}.go` to
`chasm/lib/callback` using the `CHASM` framework
## Why?
Part 2 of X for migrating callback to CHASM.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None
---------
Co-authored-by: Roey Berman <[email protected]>
## What changed?
Register the `BatchWorkflowProtobuf` as both `BatchWFTypeName` and
`BatchWFTypeNameProtobuf`, picking up the legacy batch operation in
addition to the new one.
## Why?
Now in the version after 1.30 the frontend can be changed to call the
new workflow function.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None in this iteration.
This reverts commit b0d241c43e.
## What changed?
Revert PR #8451 because of deployment failures.
## Why?
Causing deployment failures.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Further dependency on legacy batch code.
## What changed?
Create the chasm placeholder structs for `Callback` port from HSM ->
CHASM
## Why?
Port HSM -> CHASM
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
- [X] No testing needed
## Potential risks
None, this code is not used yet
## What changed?
When resolving search attributes allow users to query for `TemporalXXX`
search attributes w/out the `Temporal` prefix. The server will default
to the user namespace search attributes, then system/predefined search
attributes, then system/predefined `TemporalXXX` search attributes.
## Why?
This will unblock future chasm workstreams and allow users to create
search attributes that might conflict with existing system/predefined
SAs.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
...
---------
Co-authored-by: Rodrigo Zhou <[email protected]>
## What changed?
Remove legacy batch pathway, this has been migrated to use an internal
protobuf and not need protobuf -> struct -> protobuf serialization
## Why?
Deadcode
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
We've checked other internal locations for this dependency but we
haven't found any, there's a chance we missed some.
## What changed?
Add a new search attribute `TemporalReportedProblems` when a workflow
task fails or timeouts N consecutive times
## Why?
Enables users to easily discover workflows that are not making progress.
After a workflow task fails or times out N consecutive times a Search
Attribute, `TemporalReportedProblems` a `KeywordList`, will be added
with two entries, a `cause` and a `category`. These search attributes
will be queryable by users with queries like:
* `TemporalReportedProblems IS NOT NULL`
* `TemporalReportedProblems IN ('category=WorkflowTaskFailed')` or
`TemporalReportedProblems IN ('category=WorkflowTaskTimedout')`
* `'TemporalReportedProblems IN ("cause=UnhandledApplicationFailure")'
OR 'TemporalReportedProblems IN ("cause=ScheduleToCloseTimeout")'`
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
- [x] added new xdc test(s)
## Potential risks
Flapping _could_ happen with this change, but is unlikely. There's a
possibility of overloading the visibility system, but these changes are
protected with a dynamic config.
---------
Co-authored-by: Roey Berman <[email protected]>
## What changed?
Change flaky tests to run only on Wednesdays
## Why?
Get a better signal and avoid alert fatigure.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
N/A
## What changed?
Add `on.workflow_dispatch` to Flaky Tests Report github action
## Why?
Allows us to run this test manually whenever we'd like
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA
## What changed?
Run our flaky test report on a cadence and send results in Slack
## Why?
Help oss team take action on our flaky alerts
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA
## What changed?
Removes the full test failure from the `Details` portion of a
`junit.Result` which was breaking XML formatting.
## Why?
Breaks our flaky test analysis
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Flaky tests reporting could be breaking again.
## What changed?
Expanding `testrunner` tool to add alerts for panics and data races.
Sample table with an injected data race (and real panic):
<table><tr><th><th>Tests</th><th>Passed ✅</th><th>Skipped
⏭️</th><th>Failed ❌</th></tr><tr><td>JUnit Test Report</td><td>721
ran</td><td>340 passed</td><td>370 skipped</td><td>11
failed</td></tr></table>
<table><tr><th>Test</th><th>Result</th></tr><tr><td
colspan="2"><strong>JUnit Test
Report</strong></td></tr><tr><td>functional-test (crash)</td><td>❌
failure</td></tr><tr><td>io/server/tests.TestActivityApiBatchResetClientTestSuite/TestActivityBatchReset_DataRace</td><td>❌
failure</td></tr><tr><td>io/server/tests.TestTaskQueueStats_Pri_Suite/TestNoTasks_ValidateStats</td><td>❌
failure</td></tr><tr><td>io/server/tests.TestVersioning3FunctionalSuite/TestUnpinnedQuery_NoSticky/ForceTaskForwardForcePollForwardForceAsync</td><td>❌
failure</td></tr><tr><td>io/server/tests.TestVersioningFunctionalSuite/TestDispatchQueryOld/ForceTaskForwardForcePollForwardForceAsync</td><td>❌
failure</td></tr><tr><td>io/server/tests.TestActivityApiBatchResetClientTestSuite/TestActivityBatchReset_DataRace
(retry 1)</td><td>❌
failure</td></tr><tr><td>io/server/tests.TestActivityApiBatchResetClientTestSuite/TestActivityBatchReset_DataRace
(retry 2)</td><td>❌ failure</td></tr><tr><td>func1.DATA RACE: Data race
detected — in
go.temporal.io/server/tests.(*ActivityApiBatchResetClientTestSuite).TestActivityBatchReset_DataRace.func1</td><td>❌
failure</td></tr><tr><td>PANIC: runtime error: invalid memory address or
nil pointer dereference — in
TestTaskQueueStats_Pri_Suite/TestAddMultipleTasks_MultiplePartitions_ValidateStats_Cached</td><td>❌
failure</td></tr><tr><td>func1.DATA RACE: Data race detected — in
go.temporal.io/server/tests.(*ActivityApiBatchResetClientTestSuite).TestActivityBatchReset_DataRace.func1</td><td>❌
failure</td></tr><tr><td>func1.DATA RACE: Data race detected — in
go.temporal.io/server/tests.(*ActivityApiBatchResetClientTestSuite).TestActivityBatchReset_DataRace.func1</td><td>❌
failure</td></tr></table>
Source:
https://github.com/temporalio/temporal/actions/runs/17870152469?pr=8351
## Why?
This change will help the temporal team find bugs sooner.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA
## What changed?
Remove the `WorkflowRunTimeout` from test workflow
## Why?
Potential cause of flaky tests
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Testing only
## What changed?
When a user resets a workflow to not the base workflow, we should still
trigger nexus callbacks.
## Why?
Consistency in user experience, fixing a bug in the implementation.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
Could have an increase in the number of callbacks.
## What changed?
Propagate nexus headers on CancelOperation
## Why?
proper header handling
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA
## What changed?
Revert the changes in #8135
## Why?
Tests have been flaky since this PR merged.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Unknown changes to task queue rate limiting
## What changed?
`BatchWorkflow` is not yet deprecated, fix was not properly applied on
previous PR
## Why?
nilptr exception
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
na
## What changed?
Batch workflows were panicking because executions can be nil, but there
is no check to prevent nil pointer exception.
## Why?
Prevent nil-ptrs
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA
## What changed?
optimize the batch operation processing in `BatchActivity` and
`BatchActivityWithProtobuf` by removing the need to wait for entire
pages to complete before fetching the next page.
- Implemented proactive page fetching once a worker becomes available
- common `processWorkflowsWithProactiveFetching` function to reduce code
duplication
## Why?
The previous implementation had workers wait for entire pages to
complete. This optimization improves resource utilization. The
refactoring also eliminates duplicated functions in the `BatchParams`
struct and `BatchOperation` protobuf.
Addresses issue #8098.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
The changes maintain backward compatibility.
## Potential risks
While this change improves performance, it does modify the concurrency
model of batch processing:
1. **Timing changes**: The optimization changes when pages are fetched
relative to task completion, which could expose edge cases in error
handling or heartbeat timing
2. **Memory usage**: Pages may be fetched earlier, potentially
increasing peak memory usage if the next page is large
3. **Rate limiting interaction**: The more aggressive task scheduling
could interact differently with rate limiting, though the same
per-worker limits are maintained
4. **Heartbeat behavior**: heartbeats track the progress of an entire
page and are applied after an entire page finishes
The changes preserve all existing error handling, retry logic, and rate
limiting behavior, but the different execution timing could surface
previously hidden race conditions.
---------
Co-authored-by: Roey Berman <[email protected]>
