The social-orchestrator migration off upstream Temporal onto the embedded
Hanzo Tasks engine needs five workflow primitives its digest/sendEmail/post
workflows depend on. Implemented server-side (event-sourced, replay-safe) plus
the @hanzoai/tasks TS SDK (the worker social runs):
- Signal → running workflow: engine.signalWorkflow now re-dispatches a workflow
task after appending WORKFLOW_EXECUTION_SIGNALED, so the stateless worker
replays the extended history and fires the handler (digest/sendEmail are
entirely signal-driven). TS decider read the wrong history attr key
("signal" vs "signalName") — fixed.
- continueAsNew (kind=4): closes the run CONTINUED_AS_NEW + starts a successor
under the same workflowId carrying input/search-attrs/memo. TS continueAsNew
emits the command instead of throwing ContinueAsNewNotSupported.
- startChild (kind=5): detached (ABANDON) child, seq-keyed + idempotent; retires
the 0x006D 501 for the command path. TS startChild/executeChild added.
- Typed search attributes: carried on StartWorkflow/SignalWithStart wire, stored
on the execution, visibility-queryable (already read by fieldValue). TS adds
TypedSearchAttributes/defineSearchAttributeKey/SearchAttributeType.
- workflowIdConflictPolicy USE_EXISTING/TERMINATE_EXISTING/FAIL.
- ActivityFailure wraps ApplicationFailure (with .type) so workflow error
matching (err.cause.type === 'refresh_token') works.
Also fixes a pre-existing wire defect: the server registered
SignalWithStartWorkflow at 0x0069 while every client sends 0x0066 (schema +
Go/TS SDK) — signalWithStart 404'd against the real engine. Swapped with the
HTTP-only GetWorkflowHistory (no ZAP client).
DescribeWorkflow(workflowId, "") now resolves the OPEN run (Temporal's current
execution) so a continueAsNew successor is the one signalled/described.
Tests: engine unit (search attrs, conflict policy, continueAsNew, startChild)
+ TS e2e vs real tasksd (signals-across-continueAsNew, startChild, search attrs
run to COMPLETED). go test -race ./pkg/... green; TS vitest green.
Adds every symbol hanzoai/base/plugins/tasks and hanzoai/commerce
call today that was missing from pkg/sdk/*: SignalWithStartWorkflow,
GetWorkflow, QueryWorkflow, CheckHealth, WorkflowRunGetOptions +
WorkflowRun.GetWithOptions, ExecuteChildWorkflow + ChildWorkflowFuture,
LocalActivityOptions + WithLocalActivityOptions + ExecuteLocalActivity
(Phase-1 remote-path), worker.InterruptCh(), worker.Options rate-limit
fields wired via golang.org/x/time/rate.
New opcodes 0x0066 signalWithStartWorkflow / 0x0067 queryWorkflow
land here; frontend-side handlers were wired in PR-A via the
translate.go dispatch table.
Closes red-review §2 gap table. Dry-run base migration compiles.
Replaces StubEnv use in dispatch.go with workerEnv, a real
CoroutineEnv implementation. Activity dispatch now goes over the
wire via ScheduleActivity (opcode 0x006B) + long-poll
WaitActivityResult (0x006C), not a pre-populated in-memory map.
NewTimer uses time.NewTimer in a per-timer goroutine; Select uses
a fan-in wake channel with a 20ms watcher poll (no per-iteration
allocation). Root cancel scope propagates to timers, activity polls,
and child scopes on return.
Workflow return errors are no longer swallowed — non-nil fn returns
get encoded via temporal.Encode into a single FailureCommand entry
in the commands envelope so the frontend can fail the execution
per its retry policy.
Phase-1 note documented in struct comment: every dispatch re-runs
the workflow from the top with the same input — activities must be
idempotent. Phase 2 replaces this with event-sourced replay
without changing the public surface.
Adds WorkerTransport.ScheduleActivity / WaitActivityResult /
StartChildWorkflow with v1 JSON-in-envelope bodies. Schema
schema/tasks.zap gains the 0x006B/0x006C struct definitions.
schema/tasks.zap — one wire contract for all task RPCs:
* Namespace, Workflow lifecycle, Schedule, TaskQueue (matching)
* 20 RPCs as the v1 surface, mirroring what base/commerce/ta
actually use today. Opcodes are appended on extension; never
reused. Request/response types carry only the fields active
handlers need — no pass-through of Temporal internals.
pkg/sdk/doc.go — commits the migration contract:
go.temporal.io/sdk/{client,worker,workflow,activity,temporal}
→ github.com/hanzoai/tasks/pkg/sdk/{same}
Implementation fans out across blue/red agent pairs per package
(client, worker, workflow, activity, temporal). Each pair ports
one facet of the surface with no temporal.io imports. CI check
to enforce this lands after the first two packages merge.
## What changed?
Added TemporalExternalPayloadSizeBytes and TemporalExternalPayloadCount
to Visibility Schema as pre-defined search attribute.
## Why?
We are planning to expose external payload size and count as the search
attributes. They are pre-defined, rather than system, because they won't
be set on every single workflow, but only ones which store large
payloads externally (e.g. in S3).
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Incorrectly modified schema
## What changed?
- Support separate ID spaces for different archetypes in SQL
persistence.
## Why?
- Separate ID space workstream
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Add `TemporalUsedWorkerDeploymentVersions` search attribute to contain
all the Deployment Versions that completed at least one workflow tasks
for the execution.
This Search Attribute replaces, now deprecated, `BuildIds` Search
attribute.
## Why?
Needed a replacement for `BuildIds` Search attribute which holds
internal values and prefixes.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
None.
Support CHASM Count "Group By" Filtering.
## What changed?
* Adds implementation of CHASM Count Executions `Group By` Filter
support.
* Unifies `visibility_store.go` implementation to get count execution
info.
* Adds functional tests for validation.
## Why?
Needed to unblock standalone activities, and fully support CHASM Count
Execution requirements.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## What changed?
Update Visibility Schema with new TemporalLowCardinalityKeyword01
column.
## Why?
Allocate additional column/ES mapping for CHASM Low Cardinality Keyword
search attribute, meant for CHASM Archetype execution status.
TemporalLowCardinalityKeyword fields are stored no differently than
Keyword fields, but have a user requirement that values are from a low
cardinal set, eg. Enums, since they support GROUP BY filtering. Normal
Keyword fields do not support GROUP BY query filters.
Each archetype can track its own execution status as any CHASM Low
Cardinality Keyword field.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Remove transient dependencies on persistence in the chasm package.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Adds Archetype search attributes schema for CHASM Visibility. Functions
similarly to custom search attributes, where pre-allocated column names
or explicit properties, called field names, will map to user aliased
Archetype search attributes.
Adds Archetype search attributes to ClusterMetadata to save all
pre-allocated field name to type mappings in the Cassandra backed
ClusterMetadata store.
## Why?
Required for CHASM Visibility support. Component authors will register a
component with tagged attribute values for querying (List, Count APIs
for now), and executions of the registered component will be tagged with
these attributes.
Archetype name used for querying is left out of this schema change, may
require further discussion.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Add a new search attribute `TemporalReportedProblems` when a workflow
task fails or timeouts N consecutive times
## Why?
Enables users to easily discover workflows that are not making progress.
After a workflow task fails or times out N consecutive times a Search
Attribute, `TemporalReportedProblems` a `KeywordList`, will be added
with two entries, a `cause` and a `category`. These search attributes
will be queryable by users with queries like:
* `TemporalReportedProblems IS NOT NULL`
* `TemporalReportedProblems IN ('category=WorkflowTaskFailed')` or
`TemporalReportedProblems IN ('category=WorkflowTaskTimedout')`
* `'TemporalReportedProblems IN ("cause=UnhandledApplicationFailure")'
OR 'TemporalReportedProblems IN ("cause=ScheduleToCloseTimeout")'`
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
- [x] added new xdc test(s)
## Potential risks
Flapping _could_ happen with this change, but is unlikely. There's a
possibility of overloading the visibility system, but these changes are
protected with a dynamic config.
---------
Co-authored-by: Roey Berman <[email protected]>
## What changed?
Reverts to urlfav/cli v1 as per other tools and uses shared flag aliases
where appropriate for more consistency with the other tools.
Works around Go embed's lack of support for symlinks by hard coding the
path to the index template to use. Adds a test to ensure this doesn't
get out of sync with the latest.
## Why?
Better consistency with other tools and being able to actually find the
index template :)
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Add a top-level Visibility config in the server config file.
Option to overwrite the default number of preallocated custom search
attributes in the server config
(`visibility.persistenceCustomSearchAttributes`).
Register the preallocated custom search attributes when using custom
Visibility store.
The config can be changed at any point: increasing the number of custom
search attributes will register additional custom search attributes in
the cluster metadata; decreasing the number of custom search attributes
is no-op.
Replace struct validator library with
`github.com/go-playground/validator`.
## Why?
Be able to add more custom search attributes when using SQL Visibility
store.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
- Revive and update original PR #2977 for Elasticsearch CLI tool
Replaces manual curl invocations with a proper CLI tool that leverages
Temporal's built-in Elasticsearch auth providers and provides better
error handling and logging.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None, net new.
---
> [!NOTE]
> Introduces `temporal-elasticsearch-tool` for ES schema/index
management and ping, extends ES client APIs, embeds ES schema, and
updates Makefile to use the tool.
>
> - **Tools**:
> - New `temporal-elasticsearch-tool` CLI with commands: `setup-schema`,
`update-schema`, `create-index`, `drop-index`, `ping`; supports AWS auth
and uses embedded schema files.
> - Adds entrypoint `cmd/tools/elasticsearch`, README, and basic tests.
> - **Elasticsearch Client**:
> - Extends `CLIClient` with `ClusterPutSettings`, `IndexPutTemplate`,
`IndexPutMapping`, `Ping` and implements them (v7) using raw requests
where needed; allows custom HTTP client from config (e.g., AWS-signed).
> - **Schema**:
> - Embeds ES v7 cluster settings and index template
(`schema.Embedded...` accessors).
> - **Build/Makefile**:
> - Adds build target and binary cleanup for
`temporal-elasticsearch-tool`; updates `install-schema-es` and
`install-schema-xdc` to use the CLI instead of curl.
> - Includes binary in `.goreleaser.yml`; excludes it in
`.dockerignore`.
>
> <sup>Written by [Cursor
Bugbot]() for commit
b37864809ac95a427e74b2a424852992aac2442b. This will update automatically
on new commits. Configure
[here]().</sup>
## What changed?
`CREATE EXTENSION` sql query is replaced with a pgsql script that checks
for the existence of the extension programatically.
## Why?
`CREATE EXTENSION` can cause issues in environment where creating
extensions are limited to certain users. (e.g. Azure)
## How did you test it?
I've ran the updated schema setup in both environments where btree_gin
is installed and and not installed.
## Potential risks
Regression in visibility database setup.
## Documentation
Not needed.
## Is hotfix candidate?
No
---------
Co-authored-by: Rodrigo Zhou <[email protected]>
## What changed?
Add SQL support for matching task fairness.
Based on #8043 with some changes to reduce code duplication (can be
applied to cassandra later).
## Why?
Support all persistence implementations.
## How did you test it?
- [x] covered by existing tests
- [x] added new unit test(s)
---------
Co-authored-by: sivagirish81 <[email protected]>
## What changed?
Added new flag to tdbg's `list-tasks` command to query the fairness
table instead.
## Why?
Allow operation on fairness table, too.
## How did you test it?
- [ ] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
```
./tdbg taskqueue list-tasks --task-queue-type TASK_QUEUE_TYPE_WORKFLOW --task-queue test --min-pass 1
```
It's late ... I didn't manage to see a result, though. But it didn't
fail and I saw the query being executed in the debugger.
## What changed?
- Add FairLevel to some protos.
- Implement pass handling in task store v2.
- Remove TTLs from task store v2.
- Add some error checking to prevent misuse.
- Add support for using LIMIT in the GetTasks query.
- Rename some things.
## Why?
Start implementing matching task fairness.
## How did you test it?
- [x] built
- [x] covered by existing tests
## What changed?
Added new schema for task table that supports fairness scheduling.
* split `matching_task_store.go` into v1 (pre-fairness) and v2
(fairness) - clean copy seemed safest
* inject two task stores, `PhysicalTaskQueueManager` picks the one it
needs
## What changed?
- Consolidate task ids into `writeTaskRequest` instead of separate
slice.
- Factor out some common code in `newPhysicalTaskQueueManager`.
- Factor out `taskResponse.err()`.
- Move code out of select in `getTasksPump`.
- Add a `UseCassandraPersistence` test helper.
- Remove some obsolete structs and functions.
- Rename some test functions.
- Update some comments.
## Why?
These are a bunch of small changes from the fairness feature branch that
can apply to main immediately, which will make reviewing/merging that
branch easier.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
## What changed?
Removes the `encodingType` from the `Serializer` interface.
## Why?
The parameter is superfluous since it always set to
`enumspb.ENCODING_TYPE_PROTO3`. There are no plans to move to a
different encoding scheme. An encoding parameter for existing/new
methods can be added (back) as needed.
The next steps will be:
1. a codec for the serializer to allow customizing the encoding globally
1. an env variable to customize the encoding to be JSON (for
debugging/developing purposes)
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
<!-- Describe what has changed in this PR -->
Remove license header from every file. Because it is really hard to
follow in this PR here is the summary:
1. License header is removed from all `.go` and `.proto` files
:fireworks::fireworks:🎆.
2. `LICENSE` file in the root directory has only Temporal and Uber
copyrights.
3. 5 other `LICENSE` files added to the packages which have copyrights
different from Temporal and Uber: Datadog, Xargin, "Mat Ryer, Tyler
Bunnell and contributors".
4. `license_file` flag is removed from all code generation tools.
5. `copyright_file` flag is removed from `go:generate mockgen`
directive.
6. All copyright related targets are removed from `Makefile`.
7. Updated Temporal copyright year to 2025 everywhere.
## Why?
<!-- Tell your future self why have you made these changes -->
I double checked with legal department that it is not needed to have
license header in every file. One file per repo is enough. I put all
copyrights to the root `LICENSE` file and removed header from all other
files. Also updated tools and `Makefile`.
## What changed?
Use `path.Join()` instead of `filepath.Join()`
## Why?
#7454
## How did you test it?
Tested locally
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
None
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
None
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No
## What changed?
- CHASM nodes are now internally represented as split distinctly between
Metadata and Data parts.
- Additional columns were added to SQL plugins, bumping the SQL schema
- No change was made to the Cassandra columns; instead, ExecutionManager
ensures that when running on Cassandra, the entire node continues to be
represented in a single blob, to avoid lowering Cass performance
post-change.
## Why?
Before, a ChasmNode contained both its metadata fields, and the user
data blob. Decoding a ChasmNode would cause an additional (often large,
relative to the metadata) allocation to store the inner data blob, as
well as additional memcpys while encoding/decoding. With this change,
the data field can be passed along as-is, opaquely, while the metadata
field can be decoded without additional penalty from a large data blob.
## How did you test it?
- Updated tests
---------
Co-authored-by: Yichao Yang <[email protected]>
## What changed?
SQL persistence support has been added for CHASM nodes.
- A new `chasm_node_maps` table has been added to SQL schemas
- SQL test suites updated
- SQL execution manager/util updated
- SQL plugins for `chasm_node_map` table
## Why?
- We've landed Cassandra persistence support for CHASM nodes, this
brings SQL support up to parity.
## How did you test it?
- New tests ran on all SQL plugins
- `go test -v ./...`
## Potential risks
- No application paths write chasm_node_maps yet, but change does
involve a schema change
---------
Co-authored-by: Yichao Yang <[email protected]>
## What changed?
CHASM persistence support for Cassandra.
- Cassandra schema version bumped for new fields
- New queries for read/update/delete CHASM nodes on the executions table
- ExecutionManager/Serializer updates for CHASM nodes
- Updates to `persistence_interface`
- Wired in `chasm.Registry` under history's config
- Commented MutableStateImpl callsites, some stuff relies on tree
implementation being merged
- common/persistence tests updated for chasm nodes
## Why?
## How did you test it?
- `make unit-test && make lint`
## Potential risks
- Includes a schema change
- CHASM applications not in Prod (fields won't be exercised in persistence layer)
## What changed?
<!-- Describe what has changed in this PR -->
Fix `data_encoding` column in `current_executions` table in SQL
database.
The column must be `NOT NULL`.
## Why?
<!-- Tell your future self why have you made these changes -->
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
1. Start Temporal at v1.26.2
2. Upgrade schemas in v1.27.0
3. Restart Temporal at v1.27.0 <-- at this point, Temporal is throwing
errors
4. Rollback Temporal to v1.26.2
5. Upgrade schemas in this PR
6. Restart Temporal at v1.27.0 <-- everything works
Tested with both MySQL and PostgreSQL.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
Add version column to SQLlite, MySQL, and Postgress VisibilityStores
## Why?
There is no built in support for versioning in SQLlite, MySQL, and
Postgress, which leaves open the possibility for out-of-order updates.
We need to explicitly have a column for version to prevent this.
## How did you test it?
Manually tested that Server can run normally (still some bugs). There
will be more tests with nightly pipelines and mid pipeline version
changes to test backwards compatibility.
## Potential risks
High. A failure during table alteration or in compatibility guarantees
would entirely halt all visibility updates.
## Documentation
Not that I am aware of. We will have to close several github issues
though!
## Is hotfix candidate?
Not for cloud. Aiming for upcoming OSS release.
---------
Co-authored-by: rodrigozhou <[email protected]>
## What changed?
Add TemporalWorkerDeploymentVersion, TemporalWorkerDeployment, and
TemporalWorkflowVersioningBehavior Search Attributes
## Why?
To be able to list workflows based on their behavior, deployment, and
most recently used deployment version.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
---------
Co-authored-by: Shivam Saraf <[email protected]>
Co-authored-by: Rodrigo Zhou <[email protected]>
## What changed?
<!-- Describe what has changed in this PR -->
Add `OnConflictOptions` to `StartWorkflowExecution`.
- if the workflow id conflict policy is not `USE_EXISTING`, then no-op;
- if the field is nil, then no-op;
- otherwise, then it will add a `WorkflowExecutionOptionsUpdatedEvent`
to the existing running workflow.
## Why?
<!-- Tell your future self why have you made these changes -->
Ability to attach completion callbacks to existing running workflows.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
WIP: writing tests
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
Removed dependency of `common/persistence` from `common/namespace`
package.
And removed dependency of `common/namespace` on
`common/namespace/nsreplication`.
## Why?
<!-- Tell your future self why have you made these changes -->
Last step towards using `common/namespace` from `common/dynamicconfig`.
(circular dependency right now)
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
No behavior changes here.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
change the attributes of upgrade.sh
## Why?
<!-- Tell your future self why have you made these changes -->
Making this script executable allows automation. Most likely this is not
used, but just in case.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
No as I aware
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
N/A
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
Yes
## What changed?
Add PausedInfo search attribute
## Why?
To allow users find workflows with paused activities
## How did you test it?
make unit-test.
---------
Co-authored-by: Rodrigo Zhou <[email protected]>
## What changed?
<!-- Describe what has changed in this PR -->
Add search attributes param to sqlite.NewNamespaceConfig.
## Why?
<!-- Tell your future self why have you made these changes -->
Ability to start Temporal from the CLI with custom search attributes.
Address https://github.com/temporalio/temporal/issues/6195
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Modified the CLI to not call `registerSearchAttributes`, pass it
directly to `NewNamespaceConfig`.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
Make import formatting a part of the build, and test validity with
"ensure-no-changes" target
## Why?
So we never discuss import order again (hopefully)
## How did you test it?
its a build change, so the build will test it!
## Potential risks
As long as build works, the only thing I risk is the ire of coworker who
want imports to be different ;)
## Documentation
N/A
## Is hotfix candidate?
No
## What changed?
Add "start_time" nullable column to the current_executions table.
Some refactoring of the existing code - send row as a parameter rather
then every single field of that row.
## Why?
Adding column - to allow fast checking workflow creation time to avoid
noisy workflow restarts. Without it we need to get workflow start time
from a mutable state, which can affect latency.
## Is hotfix candidate?
No
## What changed?
<!-- Describe what has changed in this PR -->
Restored schema scripts for `nexus_incoming_services`
Added new schema revisions to drop `nexus_incoming_services` tables and
create `nexus_endpoint` tables
## Why?
<!-- Tell your future self why have you made these changes -->
OSS v1.24 was released using an older schema change from before incoming
services were renamed to endpoints, so we need to keep that to avoid
breaking schema upgrades
## What changed?
<!-- Describe what has changed in this PR -->
Fix SQL visibility root workflow fields in schema: they are supposed to
be not null.
## Why?
<!-- Tell your future self why have you made these changes -->
The schema set the root workflow fields as nullable, but the code was
assuming to be not null.
GH Issue: https://github.com/temporalio/temporal/issues/6050
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Start Temporal v1.23.1, create some workflows, update the schemas in
v1.24.0, upgrade Temporal v1.24.0:
- reproduced the error
- applied this fix, and the error was gone.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
Both our PostgreSQL and MySQL database backends will now automatically
reconnect to the database when certain errors occur: all errors chosen
have been experienced when testing this behavior through an AWS Aurora
RDS failover of either MySQL or PostgreSQL.
For both backends we will reconnect when we see:
- `ECONNRESET`
- `ECONNABORTED`
- `ECONNREFUSED`
- `io.EOF`
- `io.ErrUnexpectedEOF`
- `database/sql/driver.ErrBadConn`
for postgres we will also reconnect on the following SQLStates:
- `25006` read-only transaction
- `57P03` cannot connect now
- `0A000` feature not supported, but ONLY when the message is `cannot
set transaction read-write mode during recovery`
for mysql we will also reconnect when we see the following error codes:
- `1040` too many connections
- `1792` read-only transaction (SQLstate `25006`)
- `1836` running in read-only mode
This logic is easily extensible should we discover more failure modes
over time
## Why?
We've had multiple community reports of Temporal problems during RDS
failover. One part of this is the fact that we wouldn't necessarily
reconnect; we were at the whims of our chosen SQL abstraction's
connection pooling logic.
## How did you test it?
I manually tested this functionality in the presence of repeated RDS
failovers:
- [x] postgres12 plugin with pq driver
- [x] postgres12 plugin with pgx driver
- [x] mysql plugin
Automated testing will be added to our regular testing pipelines once
our infrastructure friends have added the support I need (it's in
progress)
## Potential risks
We're concerned there's a correctness issue in our PostgreSQL backend
that's related to our behavior during an RDS failover. If we merge this
before I figure out what's going on, we could hide the issue and make it
harder to reproduce.
## Documentation
N/A
## Is hotfix candidate?
Yes?
## What changed?
<!-- Describe what has changed in this PR -->
- Send BuildIdRedirectInfo from Matching to History on
Record*TaskStarted call containing information about redirect intention.
- History validates the redirect info against current MS and fails the
request if the redirect does not have source build if equal to the
current assigned build of the workflow.
- If redirect is valid, workflow is assigned to the new build id and all
pending but not started task are rescheduled to be sent to the new build
id.
## Why?
<!-- Tell your future self why have you made these changes -->
To prevent the following problems from happening for workflows with
concurrent tasks when redirect rules are deleted or not fully propagated
yet:
1. Assign wf back to an old build id after processing task using a newer
build id
2. Interleaved Starts: for some duration tasks are dispatched to a mix
of old or new build IDs
3. New activity output being fed to old wf
4. Execution gets stuck after being (partially) redirected
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Functional test. More unit tests to be added in a followup PR.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
As it is right now, in rare situations when a redirect rule is applied
to a WF with concurrent activities and some of them are in backoff
period due to failure, we may schedule (and start) them on the newer
build without waiting for the backoff to finish. This is planed to be
improved in the futuer.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
None.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
- Refactored Nexus Registry code to merge incoming and outgoing into one
registry
- Added separate concept of "endpoints" - which is roughly analogous to
what was previously called "services"
- Upgraded the Nexus Go SDK and API dependencies accordingly
- Command handler interface was modified to accept a context given new
requirements
- Note that invoking a worker endpoint is not yet supported, there will
be some minor restructuring required to enable that
- Requires this unmerged API PR
https://github.com/temporalio/api/pull/403. The two PRs should be merged
together.
## Why?
Part of late redesign. See also https://github.com/nexus-rpc/api/pull/5
## How did you test it?
Modified existing and added new tests.
## What changed?
<!-- Describe what has changed in this PR -->
Add root workflow execution to Visibility
## Why?
<!-- Tell your future self why have you made these changes -->
Follow up of https://github.com/temporalio/temporal/pull/5520.
Be able to list all workflows based on the root execution.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Modified existing tests to check the root workflow execution.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
No.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
This PR is intended to merge 4 months of work in the `nexus` feature
branch into `main`.
The functionality it brings is:
- Dispatching Nexus Tasks by namespace and task queue
- Internal implementation of the Incoming Service Registry - not yet
exposed
- Attaching workflow close callbacks on `StartWorkflowExecutionRequest`
and processing of those callbacks
There's more to come and some of the callback code will be refactored
into a new plugin architecture
(https://github.com/temporalio/temporal/pull/5446) soon.
---------
Co-authored-by: PJ Doerner <[email protected]>
## What changed?
<!-- Describe what has changed in this PR -->
Remove support for standard visibility
## Why?
<!-- Tell your future self why have you made these changes -->
Standard visibility is deprecated.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
Remove support for Cassandra as visibility store.
Updated integration tests that used Cassandra visibility to use
Elasticsearch.
## Why?
<!-- Tell your future self why have you made these changes -->
Deprecating Cassandra visibility.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Updated tests.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
No.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
This change adds a flag to `temporal-sql-tool` and
`temporal-cassandra-tool`'s `update-schema` and `setup-schema` to
specify an embedded schema instead of a path to the schema directory or
schema file.
```
NAME:
temporal-cassandra-tool update-schema - update cassandra schema to a specific version
USAGE:
temporal-cassandra-tool update-schema [command options] [arguments...]
OPTIONS:
--version value, -v value target version for the schema update, defaults to latest
--schema-dir value, -d value path to directory containing versioned schema
--schema-name value, -s value name of embedded versioned schema, one of: [cassandra/temporal cassandra/visibility]
```
```
NAME:
temporal-sql-tool update-schema - update sql schema to a specific version
USAGE:
temporal-sql-tool update-schema [command options] [arguments...]
OPTIONS:
--version value, -v value target version for the schema update, defaults to latest
--schema-dir value, -d value path to directory containing versioned schema
--schema-name value, -s value name of embedded versioned schema, one of: [mysql/v57/temporal mysql/v57/visibility mysql/v8/temporal mysql/v8/visibility postgresql/v96/temporal postgresql/v96/visibility postgresql/v12/temporal postgresql/v12/visibility]
```
## Why?
We want to embed the db schemas in the `temporal-sql-tool` and
`temporal-cassandra-tool` with `go:embed` so that customers can simply
specify which schema they want using a flag instead of having to copy
the schema files to wherever they're running the db tool and passing the
path to the schema.
https://github.com/temporalio/temporal/issues/2059
## How did you test it?
I wrote a new unit test for reading the embedded file system, and the
existing codepath of the changed functions are all tested by the
existing unit tests. I also updated the unit test for config validation
to reflect the new changes.
I tested this locally with a locally running Postgres. Since the more
complex changes are in `updatetask.go`, which is shared by the sql and
the cassandra tool, I think there is no need to also test with a real
cassandra db. I also built the cassandra command line tool and confirmed
that the flags and validation checks behaved as expected.
## Potential risks
Schema updates could fail. But the changes are pretty contained to the
new `-s` flag which isn't used anywhere in production yet, so the risk
is very low. The code for the `-d` flag remained basically the same.
## Is hotfix candidate?
No
zeekay
