## What changed?
- This PR adds revision number mechanics to handle task dispatch
inconsistencies that could arise since our versioning API's are becoming
eventually-consistent.
## Why?
- Making our versioning API's eventually consistent.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
- There are risks but they are gated by a dynamic config.
---
> [!NOTE]
> Adds revision-number semantics to task dispatch/versioning and
introduces a new per-deployment TQ user-data schema, updating APIs,
persistence, and server logic (DC-gated) with comprehensive tests.
>
> - **APIs (proto changes)**:
> - Add `revision_number` to `taskqueue.v1.TaskVersionDirective` and
propagate `task_dispatch_revision_number` in History
`Record*TaskStarted` requests.
> - Matching `SyncDeploymentUserDataRequest`: new fields
`deployment_name`, `update_routing_config`, `upsert_versions_data`,
`forget_versions`; response includes `routing_config_changed`. Deprecate
old `update_version_data`/`forget_version` oneof.
> - Deployment API: add `WorkerDeploymentVersionData`; mark
`DeploymentVersionData` deprecated where applicable.
> - **Persistence (task queue user data)**:
> - Extend `DeploymentData` with `deployments_data` (map of deployment →
`WorkerDeploymentData` holding `RoutingConfig` and per-build version
data). Deprecate legacy `versions` and `unversioned_ramp_data` fields.
> - **Server logic**:
> - Matching/History: compute target version using revision numbers;
start workflow deployment transitions using revision-aware decisions;
support mixed old/new schemas when calculating current/ramping.
> - Matching: new helpers to migrate/clean old-format entries, apply
routing-config updates atomically, and prevent query blackholes using
status.
> - Task dispatch carries and records revision number; internal task
struct/plumbing updated.
> - **Dynamic config**:
> - Add `system.useRevisionNumberForWorkerVersioning` (plumbed through
History/Matching) to gate new behavior.
> - **Tests**:
> - Add/expand unit and functional tests covering new schema,
routing-config updates, revision-number behavior, and propagation.
> - **Misc**:
> - Describe/Stats paths updated to read both schemas; minor build/deps
update (api-go replace).
>
> <sup>Written by [Cursor
Bugbot]() for commit
e84f4845067edbb8f65d2b7bac1f579b620db41c. This will update automatically
on new commits. Configure
[here]().</sup>
---------
Co-authored-by: ShahabT <[email protected]>
## What changed?
Allow draining backlog from the other table when moving from tasks v1 to
v2 and back (i.e. classic to fairness-enabled).
Exiting draining mode when draining is complete is not implemented yet.
## Why?
To allow migrating to fairness without losing tasks.
## How did you test it?
- [x] added new functional test(s)
## Potential risks
We need to get the rangeid lock on both tables when migrating, which
slows down initialization. This is only the first time and while
migration is active, after the old tasks are drained then we won't need
to lock the old table. That part isn't implemented yet, so this is off
by default.
## What changed?
Adding a breakdown of stats per-priority levels to the two APIs that
return task queue stats already.
## Why?
New API is defined here: https://github.com/temporalio/api/pull/611.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## What changed?
- Add FairLevel to some protos.
- Implement pass handling in task store v2.
- Remove TTLs from task store v2.
- Add some error checking to prevent misuse.
- Add support for using LIMIT in the GetTasks query.
- Rename some things.
## Why?
Start implementing matching task fairness.
## How did you test it?
- [x] built
- [x] covered by existing tests
## What changed?
<!-- Describe what has changed in this PR -->
Remove license header from every file. Because it is really hard to
follow in this PR here is the summary:
1. License header is removed from all `.go` and `.proto` files
:fireworks::fireworks:🎆.
2. `LICENSE` file in the root directory has only Temporal and Uber
copyrights.
3. 5 other `LICENSE` files added to the packages which have copyrights
different from Temporal and Uber: Datadog, Xargin, "Mat Ryer, Tyler
Bunnell and contributors".
4. `license_file` flag is removed from all code generation tools.
5. `copyright_file` flag is removed from `go:generate mockgen`
directive.
6. All copyright related targets are removed from `Makefile`.
7. Updated Temporal copyright year to 2025 everywhere.
## Why?
<!-- Tell your future self why have you made these changes -->
I double checked with legal department that it is not needed to have
license header in every file. One file per repo is enough. I put all
copyrights to the root `LICENSE` file and removed header from all other
files. Also updated tools and `Makefile`.
## What changed?
- Don't call `allowForwarding` on a root partition
- Use absolute times for backlog head age so we only call `time.Since`
once
- Return a slice of `taskqueuespb.InternalTaskQueueStatus` for subqueues
## Why?
Small optimizations, more detailed internal info
## How did you test it?
updated tests
## What changed?
<!-- Describe what has changed in this PR -->
Updated code to use latest API changes in which Deployment Versions are
represented by string fields rather than structs.
`WorkerDeploymentVersion` proto message still exists but only for the
internal APIs.
## Why?
<!-- Tell your future self why have you made these changes -->
See https://github.com/temporalio/api/pull/547.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
---------
Co-authored-by: Carly de Frondeville <[email protected]>
## What changed?
<!-- Describe what has changed in this PR -->
Using new Deployment Options fields sent by SDK in Versioning 3
functionality. Old fields are still used when new fields are absent.
Implementation did not change, both new and old fields sent in polls and
task responses are still converted to old `Deployment` object and used
as before. Later, code will be refactored to change the `Deployment`
usages to `DeploymentVersion`.
Also added new fields to replace `Deployment` with `DeploymentVersion`
fields in internal protos where needed. Matching<->History communication
happens via these new fields, only new internal fields are written but
both new and old fields are read.
## Why?
<!-- Tell your future self why have you made these changes -->
Incorporating latest renames in Versioning APIs.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Existing tests changed to use new fields (or both old and new depending
on the test).
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
None.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
None yet.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
<!-- Describe what has changed in this PR -->
Add versioning v3 fields to the internal APIs between History and
Matching so they can send needed info when tasks are scheduled or
started.
## Why?
<!-- Tell your future self why have you made these changes -->
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
Added a new `tdbg` subcommand : --describe-task-queue-partition which
shall output the following statistics, per partition, for better
visibility:
- [TQ statistics
](https://github.com/temporalio/api/blob/2ed0a18b7ad6b6052a15da6d9d81bb97e05bc359/temporal/api/taskqueue/v1/message.proto#L92)
- List of pollers for each build ID
- More metrics defined via `InternalTaskQueueStatus` struct (see this PR
itself)
## Usage
``` ./tdbg taskqueue describe-task-queue-partition --task-queue "hello-world" --partition-id 2 ```
## Options/Flags for the command
```
OPTIONS:
--namespace-id value NamespaceId (default: "default")
--task-queue value Task Queue name
--task-queue-type value Task Queue type: activity, workflow, nexus
(experimental) (default: "TASK_QUEUE_TYPE_WORKFLOW")
--partition-id value Partition ID (default: 0)
--sticky-name value Sticky Name for a task queue partition, if present
--select-build-id value Build IDs (accepts multiple inputs)
--select-unversioned Unversioned task queue partition (default: true)
--select-all-active All active task queue versions (default: true)
--help, -h show help (default: false)
```
## Sample output
```
{
"versionsInfoInternal": {
"": {
"physicalTaskQueueInfo": {
"taskQueueStats": {
"approximateBacklogCount": "1",
"approximateBacklogAge": "81.937463s"
},
"internalTaskQueueStatus": {
"readLevel": "1",
"taskIdBlock": {
"startId": "2",
"endId": "100000"
}
}
}
}
}
}
```
## Why?
<!-- Tell your future self why have you made these changes -->
- better visibility into TQ metrics per partition
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit test? Checked in staging env? -->
- 100% unit test coverage
- Added more tests for DescribeTaskQueuePartition specifically
- Tested the command locally
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this change to production? -->
None, this is an internal tool.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to be sent to the broader community? (Yes/No) -->
nope
## What changed?
<!-- Describe what has changed in this PR -->
When a query is forwarded to parent partition, allow the parent to chose
build ID itself rather than using the one selected by child partition.
## Why?
<!-- Tell your future self why have you made these changes -->
This fixes a race condition in which query comes to the child right
after User Data is updated but before the child partition has the latest
User Data. In that case child may select a stale build ID that no longer
has workers. When forwarding the request to parent, we want to give the
parent a second chance to fix the child's mistake.
This fixes the flakes in the following SDK test:
https://github.com/temporalio/sdk-go/blob/master/test/worker_versioning_test.go#L885
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Made sure the SDK test passes 10 consecutive times.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
None.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
None.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
<!-- Describe what has changed in this PR -->
- This PR aims to enable the metric `approximateBacklogCounter` which
shall serve to be an approximation of the number of tasks in the task
queue backlog.
**Note**
- The following commit will not be merged to main:
https://github.com/temporalio/temporal/pull/5915
- The main purpose of this commit was to provide an exact count for SQL
databases since they don't persist the value of the backlog counter
during creation of tasks. Thus, if a TQ were to shut down ungracefully,
the backlog counter value read from persistence on restart would not be
accurate. This commit added support for running a `Count(*)` query for
those scenarios.
- However, it was found out that it was not altogether exact. `task_gc`
only goes on and deletes tasks from the db if there has been a second's
difference between the previous batch's delete. In the case where there
hasn't, as was found in some of my test cases, the task_gc will not run.
Thus, the `Count(*)` query would return an over-estimate and it would be
better to use the in-memory counter in this case.
- There are *no* consequences of not including this PR for this project.
The counter is meant to be an estimate of the number of tasks in the
backlog and will continue to do so. The only difference would be that it
would be less accurate for SQL databases than others in certain
occasions.
## Why?
<!-- Tell your future self why have you made these changes -->
So that new and helpful metrics can be used by customers:
1. `ApproximateBacklogCount`
## How did you test it?
- Suite of tests brought in by each individual PR
- CI tests
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
Documentation has been added in appropriate places.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No
---------
Co-authored-by: ShahabT <[email protected]>
## What changed?
<!-- Describe what has changed in this PR -->
- Send BuildIdRedirectInfo from Matching to History on
Record*TaskStarted call containing information about redirect intention.
- History validates the redirect info against current MS and fails the
request if the redirect does not have source build if equal to the
current assigned build of the workflow.
- If redirect is valid, workflow is assigned to the new build id and all
pending but not started task are rescheduled to be sent to the new build
id.
## Why?
<!-- Tell your future self why have you made these changes -->
To prevent the following problems from happening for workflows with
concurrent tasks when redirect rules are deleted or not fully propagated
yet:
1. Assign wf back to an old build id after processing task using a newer
build id
2. Interleaved Starts: for some duration tasks are dispatched to a mix
of old or new build IDs
3. New activity output being fed to old wf
4. Execution gets stuck after being (partially) redirected
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Functional test. More unit tests to be added in a followup PR.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
As it is right now, in rare situations when a redirect rule is applied
to a WF with concurrent activities and some of them are in backoff
period due to failure, we may schedule (and start) them on the newer
build without waiting for the backoff to finish. This is planed to be
improved in the futuer.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
None.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
Add `DescribeTaskQueuePartition` to be used by enhanced
`DescribeTaskQueue` API
Use new `TaskQueueVersionGroup` message in api repo to specify which
versions are of interest in a way that can be reused by server protos.
## Why?
So that we can ask partitions about internal details, and then pick a
subset of those internal details to share with the public-facing
`DescribeTaskQueue` API
## How did you test it?
It builds
## Potential risks
None
## Documentation
New message documented in API repo
https://github.com/temporalio/api/pull/379
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
---------
Co-authored-by: Shahab Tajik <[email protected]>
## What changed?
<!-- Describe what has changed in this PR -->
This is the history side changes needed for the Worker Versioning
Assignment Rules to work. It includes:
- add versioning stamp to task started events based on the poller
- persisting the assigned build IDs in mutable state for new workflows
and independently-versioned activities. This is done in two places:
- when build ID is returned by matching as an output of Add*Task API
- from the versioning stamp in task started events to cover the
following cases:
- Sync-match
- MS rebuild based on history events
- When we failed to update build ID at schedule time
- Matching does not re-evaluate the version directive for spooled tasks
for WV2, instead it uses the assignment decision made at spool time
(which is the build ID of the physical task queue in which the task was
spooled to).
## Why?
<!-- Tell your future self why have you made these changes -->
Part of Worker Versioning new API.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Added unit test and functional tests where needed.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
One database write is added if all the following conditions are true for
a workflow task:
- assignment rules are added for the Task Queue (i.e. worker versioning
is enabled)
- sync-match did not happen for this task
- this is the first workflow task of the execution
- the workflow has not inherited a build ID (for child WF or CaN)
One database write is added if all the following conditions are true for
an activity task:
- assignment rules are added for the Task Queue (i.e. worker versioning
is enabled)
- sync-match did not happen for this task
- the activity is on a different task queue (or otherwise asked to be
independently assigned to a build ID)
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
None so far.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No
## What changed?
<!-- Describe what has changed in this PR -->
Assigning a task to the appropriate build ID based on the versioning
assignment rules.
For now, both old and new versioning data is supported, in the following
manner
- tasks for unassigned (new) workflows/activities -> first try
assignment rules, if there is no matching assignment rules fallback to
default version set.
- tasks of existing versioned executions -> if the build id is part of a
version set keep using the version set, otherwise, assume user intends
to use the new API and use a dedicated queue for the build ID.
- This means we no longer created "guessed" version sets for "unkown"
build IDs
- tasks of existing unversioned executions -> stays unversioned
## Why?
<!-- Tell your future self why have you made these changes -->
Part of new Worker Versioning API
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Unit tests, functional test will come when the API for adding rules is
implemented.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
Because we no longer created "guessed" version sets for "unkown" build
IDs, users of old WV API may "loose" tasks in extremely rare situations.
But that is expected for the old API given its private preview stage.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
I've upgraded the version of our Go API in order to allow invalid UTF-8
data in our proto strings.
Please review
https://github.com/temporalio/temporal/pull/5476/commits/5cf979058d0897638f0cf80efea5dbfd737c4a97
only
## Why?
The version of gogo/protobuf we used allowed invalid UTF-8 in strings
which is a violation of the proto3 spec. We were unaware of this until
our new google/protobuf based structs began failing to deserialize data
we'd used previously
## How did you test it?
I added a new end-to-end test to verify that we can create, describe,
and delete a namespace that contains invalid UTF-8 in multiple
attributes
## Potential risks
This is lower risk than commits prior to this as temporal DBs may
already contain invalid UTF-8 data
## Documentation
I adjusted our docs to mention this behavior and the `protolegacy` tag
## Is hotfix candidate?
Yes: this will be going out as part of our 1.23.0 release. In fact, the
release is blocked on it!
---------
Co-authored-by: David Reiss <[email protected]>
## What changed?
I replaced protoc.sh with
[protogen](https://github.com/temporalio/api-go/pull/139)
## Why?
I've copied some variant of this protoc.sh tool into three repos now.
This is unsustainable, so I've added a little build tool to api-go that
will handle proto generation and post-processing for us.
## How did you test it?
[I ran `make proto` in the PR stacked on this
one](https://github.com/temporalio/temporal/pull/5211)
## Potential risks
None
## Is hotfix candidate?
No
**What changed?**
I rebuilt all proto files and bumped the version of our protoc plugin
**Why?**
So as to not pollute the PRs of others
**How did you test it?**
CI
**Potential risks**
None
**Is hotfix candidate?**
No.
**What changed?**
I added the magic MIT License header to all proto files and regenerated
the code
**Why?**
Turns out we didn't include the header our licensegen script checked
for, so when the new protoc plugin prepending the license our licensegen
script would prepend it a second time
**What changed?**
gogo/protobuf has been replaced with Google's official go compiler.
**Why?**
gogo/protobuf has been deprecated for some time and the community is
moving on, building new tools (like vtproto) atop google's v2 compiler.
**How did you test it?**
`make test`
**Potential risks**
1. The change from embedded gogo-generated-structs to
google-generated-pointers-to-structs created a risk of nil pointer
exceptions. I've fixed all the ones our tests found but it's possible
there are more lurking in the new code.
2. This change may cause our performance to decrease. Certainly
encoding/deconding of proto objects will become slower, but the overuse
of pointers by the google compiler may negatively affect our overall
performance. We'll need to keep an eye on the GC stats
3. This breaks the HTTP API. We will not support [shortand payload
encoding](https://github.com/temporalio/proposals/blob/master/api/http-api.md#payload-formatting)
in this first pass; that will come once this initial work is in testing.
**Breaking changes for developers**
- `*time.Time` in proto structs will now be
[timestamppb.Timestamp](https://pkg.go.dev/google.golang.org/[email protected]/types/known/timestamppb#section-documentation)
- `*time.Duration` will now be
[durationpb.Duration](https://pkg.go.dev/google.golang.org/protobuf/types/known/durationpb)
- V2-generated structs embed locks, so you cannot dereference them. `go
vet` will scream at you about this. If you need a copy, use
`proto.Clone`.
- If the performance of this sucks then I will either update our code
generator to add shallow-clone methods or hand-roll the ones we need
- Proto enums will, when formatted to JSON, now be in
`SCREAMING_SNAKE_CASE` rather than `PascalCase`. We decided (in
discussion with the SDK team) that now was as good a time as any to rip
the bandage off.
- Proto objects, or objects embedding protos, cannot be compared using
`reflect.DeepEqual` or _anything_ that uses it. This includes `testify`
and `mock` equality testers!
- You will need to use the `common/testing/protorequire`,
`common/testing/protoassert`, or `common/testing/protomock` packages
instead. I've implemented proto-compatible matchers and assertions there
for all cases I've encountered
- If you need `reflect.DeepEqual` for any reason you can use
`go.temporal.io/api/temporalproto.DeepEqual` instead
Note that history loading will not be impacted by the JSON changes: I
rewrote history loading to dynamically fix incoming history JSON data
(like all our other sdks); you can find this code in [my fork of our go
API](https://github.com/tdeebswihart/temporal-api-go/blob/master/internal/temporalhistoryv1/load.go)
alongside its tests.
**🚨Sharp Edges Introduced🚨**
Beware `*timestamppb.Timestamp.AsTime()`. If you need to extract a time
value from a proto time (timestamppb) **always** make sure to check
whether it's nil first. When the proto object is `nil` `AsTime()` will
return a non-zero time at the proto epoch: UTC midnight on January 1,
1970.
I've made this mistake multiple times during this transition and each
time it's been a pain to debug
**Is hotfix candidate?**
No.
