## What changed?
Add `DescribeTaskQueuePartition` to be used by enhanced
`DescribeTaskQueue` API
Use new `TaskQueueVersionGroup` message in api repo to specify which
versions are of interest in a way that can be reused by server protos.
## Why?
So that we can ask partitions about internal details, and then pick a
subset of those internal details to share with the public-facing
`DescribeTaskQueue` API
## How did you test it?
It builds
## Potential risks
None
## Documentation
New message documented in API repo
https://github.com/temporalio/api/pull/379
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
---------
Co-authored-by: Shahab Tajik <[email protected]>
## What changed?
<!-- Describe what has changed in this PR -->
This is the history side changes needed for the Worker Versioning
Assignment Rules to work. It includes:
- add versioning stamp to task started events based on the poller
- persisting the assigned build IDs in mutable state for new workflows
and independently-versioned activities. This is done in two places:
- when build ID is returned by matching as an output of Add*Task API
- from the versioning stamp in task started events to cover the
following cases:
- Sync-match
- MS rebuild based on history events
- When we failed to update build ID at schedule time
- Matching does not re-evaluate the version directive for spooled tasks
for WV2, instead it uses the assignment decision made at spool time
(which is the build ID of the physical task queue in which the task was
spooled to).
## Why?
<!-- Tell your future self why have you made these changes -->
Part of Worker Versioning new API.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Added unit test and functional tests where needed.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
One database write is added if all the following conditions are true for
a workflow task:
- assignment rules are added for the Task Queue (i.e. worker versioning
is enabled)
- sync-match did not happen for this task
- this is the first workflow task of the execution
- the workflow has not inherited a build ID (for child WF or CaN)
One database write is added if all the following conditions are true for
an activity task:
- assignment rules are added for the Task Queue (i.e. worker versioning
is enabled)
- sync-match did not happen for this task
- the activity is on a different task queue (or otherwise asked to be
independently assigned to a build ID)
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
None so far.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No
## What changed?
<!-- Describe what has changed in this PR -->
Assigning a task to the appropriate build ID based on the versioning
assignment rules.
For now, both old and new versioning data is supported, in the following
manner
- tasks for unassigned (new) workflows/activities -> first try
assignment rules, if there is no matching assignment rules fallback to
default version set.
- tasks of existing versioned executions -> if the build id is part of a
version set keep using the version set, otherwise, assume user intends
to use the new API and use a dedicated queue for the build ID.
- This means we no longer created "guessed" version sets for "unkown"
build IDs
- tasks of existing unversioned executions -> stays unversioned
## Why?
<!-- Tell your future self why have you made these changes -->
Part of new Worker Versioning API
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Unit tests, functional test will come when the API for adding rules is
implemented.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
Because we no longer created "guessed" version sets for "unkown" build
IDs, users of old WV API may "loose" tasks in extremely rare situations.
But that is expected for the old API given its private preview stage.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
I replaced protoc.sh with
[protogen](https://github.com/temporalio/api-go/pull/139)
## Why?
I've copied some variant of this protoc.sh tool into three repos now.
This is unsustainable, so I've added a little build tool to api-go that
will handle proto generation and post-processing for us.
## How did you test it?
[I ran `make proto` in the PR stacked on this
one](https://github.com/temporalio/temporal/pull/5211)
## Potential risks
None
## Is hotfix candidate?
No
**What changed?**
I rebuilt all proto files and bumped the version of our protoc plugin
**Why?**
So as to not pollute the PRs of others
**How did you test it?**
CI
**Potential risks**
None
**Is hotfix candidate?**
No.
**What changed?**
I added the magic MIT License header to all proto files and regenerated
the code
**Why?**
Turns out we didn't include the header our licensegen script checked
for, so when the new protoc plugin prepending the license our licensegen
script would prepend it a second time
**What changed?**
gogo/protobuf has been replaced with Google's official go compiler.
**Why?**
gogo/protobuf has been deprecated for some time and the community is
moving on, building new tools (like vtproto) atop google's v2 compiler.
**How did you test it?**
`make test`
**Potential risks**
1. The change from embedded gogo-generated-structs to
google-generated-pointers-to-structs created a risk of nil pointer
exceptions. I've fixed all the ones our tests found but it's possible
there are more lurking in the new code.
2. This change may cause our performance to decrease. Certainly
encoding/deconding of proto objects will become slower, but the overuse
of pointers by the google compiler may negatively affect our overall
performance. We'll need to keep an eye on the GC stats
3. This breaks the HTTP API. We will not support [shortand payload
encoding](https://github.com/temporalio/proposals/blob/master/api/http-api.md#payload-formatting)
in this first pass; that will come once this initial work is in testing.
**Breaking changes for developers**
- `*time.Time` in proto structs will now be
[timestamppb.Timestamp](https://pkg.go.dev/google.golang.org/[email protected]/types/known/timestamppb#section-documentation)
- `*time.Duration` will now be
[durationpb.Duration](https://pkg.go.dev/google.golang.org/protobuf/types/known/durationpb)
- V2-generated structs embed locks, so you cannot dereference them. `go
vet` will scream at you about this. If you need a copy, use
`proto.Clone`.
- If the performance of this sucks then I will either update our code
generator to add shallow-clone methods or hand-roll the ones we need
- Proto enums will, when formatted to JSON, now be in
`SCREAMING_SNAKE_CASE` rather than `PascalCase`. We decided (in
discussion with the SDK team) that now was as good a time as any to rip
the bandage off.
- Proto objects, or objects embedding protos, cannot be compared using
`reflect.DeepEqual` or _anything_ that uses it. This includes `testify`
and `mock` equality testers!
- You will need to use the `common/testing/protorequire`,
`common/testing/protoassert`, or `common/testing/protomock` packages
instead. I've implemented proto-compatible matchers and assertions there
for all cases I've encountered
- If you need `reflect.DeepEqual` for any reason you can use
`go.temporal.io/api/temporalproto.DeepEqual` instead
Note that history loading will not be impacted by the JSON changes: I
rewrote history loading to dynamically fix incoming history JSON data
(like all our other sdks); you can find this code in [my fork of our go
API](https://github.com/tdeebswihart/temporal-api-go/blob/master/internal/temporalhistoryv1/load.go)
alongside its tests.
**🚨Sharp Edges Introduced🚨**
Beware `*timestamppb.Timestamp.AsTime()`. If you need to extract a time
value from a proto time (timestamppb) **always** make sure to check
whether it's nil first. When the proto object is `nil` `AsTime()` will
return a non-zero time at the proto epoch: UTC midnight on January 1,
1970.
I've made this mistake multiple times during this transition and each
time it's been a pain to debug
**Is hotfix candidate?**
No.
