## What changed?
<!-- Describe what has changed in this PR -->
When a query is forwarded to parent partition, allow the parent to chose
build ID itself rather than using the one selected by child partition.
## Why?
<!-- Tell your future self why have you made these changes -->
This fixes a race condition in which query comes to the child right
after User Data is updated but before the child partition has the latest
User Data. In that case child may select a stale build ID that no longer
has workers. When forwarding the request to parent, we want to give the
parent a second chance to fix the child's mistake.
This fixes the flakes in the following SDK test:
https://github.com/temporalio/sdk-go/blob/master/test/worker_versioning_test.go#L885
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Made sure the SDK test passes 10 consecutive times.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
None.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
None.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
<!-- Describe what has changed in this PR -->
- This PR aims to enable the metric `approximateBacklogCounter` which
shall serve to be an approximation of the number of tasks in the task
queue backlog.
**Note**
- The following commit will not be merged to main:
https://github.com/temporalio/temporal/pull/5915
- The main purpose of this commit was to provide an exact count for SQL
databases since they don't persist the value of the backlog counter
during creation of tasks. Thus, if a TQ were to shut down ungracefully,
the backlog counter value read from persistence on restart would not be
accurate. This commit added support for running a `Count(*)` query for
those scenarios.
- However, it was found out that it was not altogether exact. `task_gc`
only goes on and deletes tasks from the db if there has been a second's
difference between the previous batch's delete. In the case where there
hasn't, as was found in some of my test cases, the task_gc will not run.
Thus, the `Count(*)` query would return an over-estimate and it would be
better to use the in-memory counter in this case.
- There are *no* consequences of not including this PR for this project.
The counter is meant to be an estimate of the number of tasks in the
backlog and will continue to do so. The only difference would be that it
would be less accurate for SQL databases than others in certain
occasions.
## Why?
<!-- Tell your future self why have you made these changes -->
So that new and helpful metrics can be used by customers:
1. `ApproximateBacklogCount`
## How did you test it?
- Suite of tests brought in by each individual PR
- CI tests
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
Documentation has been added in appropriate places.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No
---------
Co-authored-by: ShahabT <[email protected]>
## What changed?
<!-- Describe what has changed in this PR -->
- Send BuildIdRedirectInfo from Matching to History on
Record*TaskStarted call containing information about redirect intention.
- History validates the redirect info against current MS and fails the
request if the redirect does not have source build if equal to the
current assigned build of the workflow.
- If redirect is valid, workflow is assigned to the new build id and all
pending but not started task are rescheduled to be sent to the new build
id.
## Why?
<!-- Tell your future self why have you made these changes -->
To prevent the following problems from happening for workflows with
concurrent tasks when redirect rules are deleted or not fully propagated
yet:
1. Assign wf back to an old build id after processing task using a newer
build id
2. Interleaved Starts: for some duration tasks are dispatched to a mix
of old or new build IDs
3. New activity output being fed to old wf
4. Execution gets stuck after being (partially) redirected
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Functional test. More unit tests to be added in a followup PR.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
As it is right now, in rare situations when a redirect rule is applied
to a WF with concurrent activities and some of them are in backoff
period due to failure, we may schedule (and start) them on the newer
build without waiting for the backoff to finish. This is planed to be
improved in the futuer.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
None.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
Add `DescribeTaskQueuePartition` to be used by enhanced
`DescribeTaskQueue` API
Use new `TaskQueueVersionGroup` message in api repo to specify which
versions are of interest in a way that can be reused by server protos.
## Why?
So that we can ask partitions about internal details, and then pick a
subset of those internal details to share with the public-facing
`DescribeTaskQueue` API
## How did you test it?
It builds
## Potential risks
None
## Documentation
New message documented in API repo
https://github.com/temporalio/api/pull/379
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
---------
Co-authored-by: Shahab Tajik <[email protected]>
## What changed?
<!-- Describe what has changed in this PR -->
This is the history side changes needed for the Worker Versioning
Assignment Rules to work. It includes:
- add versioning stamp to task started events based on the poller
- persisting the assigned build IDs in mutable state for new workflows
and independently-versioned activities. This is done in two places:
- when build ID is returned by matching as an output of Add*Task API
- from the versioning stamp in task started events to cover the
following cases:
- Sync-match
- MS rebuild based on history events
- When we failed to update build ID at schedule time
- Matching does not re-evaluate the version directive for spooled tasks
for WV2, instead it uses the assignment decision made at spool time
(which is the build ID of the physical task queue in which the task was
spooled to).
## Why?
<!-- Tell your future self why have you made these changes -->
Part of Worker Versioning new API.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Added unit test and functional tests where needed.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
One database write is added if all the following conditions are true for
a workflow task:
- assignment rules are added for the Task Queue (i.e. worker versioning
is enabled)
- sync-match did not happen for this task
- this is the first workflow task of the execution
- the workflow has not inherited a build ID (for child WF or CaN)
One database write is added if all the following conditions are true for
an activity task:
- assignment rules are added for the Task Queue (i.e. worker versioning
is enabled)
- sync-match did not happen for this task
- the activity is on a different task queue (or otherwise asked to be
independently assigned to a build ID)
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
None so far.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No
## What changed?
<!-- Describe what has changed in this PR -->
Assigning a task to the appropriate build ID based on the versioning
assignment rules.
For now, both old and new versioning data is supported, in the following
manner
- tasks for unassigned (new) workflows/activities -> first try
assignment rules, if there is no matching assignment rules fallback to
default version set.
- tasks of existing versioned executions -> if the build id is part of a
version set keep using the version set, otherwise, assume user intends
to use the new API and use a dedicated queue for the build ID.
- This means we no longer created "guessed" version sets for "unkown"
build IDs
- tasks of existing unversioned executions -> stays unversioned
## Why?
<!-- Tell your future self why have you made these changes -->
Part of new Worker Versioning API
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Unit tests, functional test will come when the API for adding rules is
implemented.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
Because we no longer created "guessed" version sets for "unkown" build
IDs, users of old WV API may "loose" tasks in extremely rare situations.
But that is expected for the old API given its private preview stage.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
I've upgraded the version of our Go API in order to allow invalid UTF-8
data in our proto strings.
Please review
https://github.com/temporalio/temporal/pull/5476/commits/5cf979058d0897638f0cf80efea5dbfd737c4a97
only
## Why?
The version of gogo/protobuf we used allowed invalid UTF-8 in strings
which is a violation of the proto3 spec. We were unaware of this until
our new google/protobuf based structs began failing to deserialize data
we'd used previously
## How did you test it?
I added a new end-to-end test to verify that we can create, describe,
and delete a namespace that contains invalid UTF-8 in multiple
attributes
## Potential risks
This is lower risk than commits prior to this as temporal DBs may
already contain invalid UTF-8 data
## Documentation
I adjusted our docs to mention this behavior and the `protolegacy` tag
## Is hotfix candidate?
Yes: this will be going out as part of our 1.23.0 release. In fact, the
release is blocked on it!
---------
Co-authored-by: David Reiss <[email protected]>
## What changed?
I replaced protoc.sh with
[protogen](https://github.com/temporalio/api-go/pull/139)
## Why?
I've copied some variant of this protoc.sh tool into three repos now.
This is unsustainable, so I've added a little build tool to api-go that
will handle proto generation and post-processing for us.
## How did you test it?
[I ran `make proto` in the PR stacked on this
one](https://github.com/temporalio/temporal/pull/5211)
## Potential risks
None
## Is hotfix candidate?
No
**What changed?**
I rebuilt all proto files and bumped the version of our protoc plugin
**Why?**
So as to not pollute the PRs of others
**How did you test it?**
CI
**Potential risks**
None
**Is hotfix candidate?**
No.
**What changed?**
I added the magic MIT License header to all proto files and regenerated
the code
**Why?**
Turns out we didn't include the header our licensegen script checked
for, so when the new protoc plugin prepending the license our licensegen
script would prepend it a second time
**What changed?**
gogo/protobuf has been replaced with Google's official go compiler.
**Why?**
gogo/protobuf has been deprecated for some time and the community is
moving on, building new tools (like vtproto) atop google's v2 compiler.
**How did you test it?**
`make test`
**Potential risks**
1. The change from embedded gogo-generated-structs to
google-generated-pointers-to-structs created a risk of nil pointer
exceptions. I've fixed all the ones our tests found but it's possible
there are more lurking in the new code.
2. This change may cause our performance to decrease. Certainly
encoding/deconding of proto objects will become slower, but the overuse
of pointers by the google compiler may negatively affect our overall
performance. We'll need to keep an eye on the GC stats
3. This breaks the HTTP API. We will not support [shortand payload
encoding](https://github.com/temporalio/proposals/blob/master/api/http-api.md#payload-formatting)
in this first pass; that will come once this initial work is in testing.
**Breaking changes for developers**
- `*time.Time` in proto structs will now be
[timestamppb.Timestamp](https://pkg.go.dev/google.golang.org/[email protected]/types/known/timestamppb#section-documentation)
- `*time.Duration` will now be
[durationpb.Duration](https://pkg.go.dev/google.golang.org/protobuf/types/known/durationpb)
- V2-generated structs embed locks, so you cannot dereference them. `go
vet` will scream at you about this. If you need a copy, use
`proto.Clone`.
- If the performance of this sucks then I will either update our code
generator to add shallow-clone methods or hand-roll the ones we need
- Proto enums will, when formatted to JSON, now be in
`SCREAMING_SNAKE_CASE` rather than `PascalCase`. We decided (in
discussion with the SDK team) that now was as good a time as any to rip
the bandage off.
- Proto objects, or objects embedding protos, cannot be compared using
`reflect.DeepEqual` or _anything_ that uses it. This includes `testify`
and `mock` equality testers!
- You will need to use the `common/testing/protorequire`,
`common/testing/protoassert`, or `common/testing/protomock` packages
instead. I've implemented proto-compatible matchers and assertions there
for all cases I've encountered
- If you need `reflect.DeepEqual` for any reason you can use
`go.temporal.io/api/temporalproto.DeepEqual` instead
Note that history loading will not be impacted by the JSON changes: I
rewrote history loading to dynamically fix incoming history JSON data
(like all our other sdks); you can find this code in [my fork of our go
API](https://github.com/tdeebswihart/temporal-api-go/blob/master/internal/temporalhistoryv1/load.go)
alongside its tests.
**🚨Sharp Edges Introduced🚨**
Beware `*timestamppb.Timestamp.AsTime()`. If you need to extract a time
value from a proto time (timestamppb) **always** make sure to check
whether it's nil first. When the proto object is `nil` `AsTime()` will
return a non-zero time at the proto epoch: UTC midnight on January 1,
1970.
I've made this mistake multiple times during this transition and each
time it's been a pain to debug
**Is hotfix candidate?**
No.
