Files
tasks/tools/tdbg/task_encoder.go
Tim Deeb-SwihartandGitHub 95169a53c3 Replace gogo/protobuf with google/protobuf (#5032)
**What changed?**

gogo/protobuf has been replaced with Google's official go compiler. 

**Why?**

gogo/protobuf has been deprecated for some time and the community is
moving on, building new tools (like vtproto) atop google's v2 compiler.

**How did you test it?**

`make test`

**Potential risks**

1. The change from embedded gogo-generated-structs to
google-generated-pointers-to-structs created a risk of nil pointer
exceptions. I've fixed all the ones our tests found but it's possible
there are more lurking in the new code.
2. This change may cause our performance to decrease. Certainly
encoding/deconding of proto objects will become slower, but the overuse
of pointers by the google compiler may negatively affect our overall
performance. We'll need to keep an eye on the GC stats
3. This breaks the HTTP API. We will not support [shortand payload
encoding](https://github.com/temporalio/proposals/blob/master/api/http-api.md#payload-formatting)
in this first pass; that will come once this initial work is in testing.

**Breaking changes for developers**

- `*time.Time` in proto structs will now be
[timestamppb.Timestamp](https://pkg.go.dev/google.golang.org/[email protected]/types/known/timestamppb#section-documentation)
- `*time.Duration` will now be
[durationpb.Duration](https://pkg.go.dev/google.golang.org/protobuf/types/known/durationpb)
- V2-generated structs embed locks, so you cannot dereference them. `go
vet` will scream at you about this. If you need a copy, use
`proto.Clone`.
- If the performance of this sucks then I will either update our code
generator to add shallow-clone methods or hand-roll the ones we need
- Proto enums will, when formatted to JSON, now be in
`SCREAMING_SNAKE_CASE` rather than `PascalCase`. We decided (in
discussion with the SDK team) that now was as good a time as any to rip
the bandage off.
- Proto objects, or objects embedding protos, cannot be compared using
`reflect.DeepEqual` or _anything_ that uses it. This includes `testify`
and `mock` equality testers!
- You will need to use the `common/testing/protorequire`,
`common/testing/protoassert`, or `common/testing/protomock` packages
instead. I've implemented proto-compatible matchers and assertions there
for all cases I've encountered
- If you need `reflect.DeepEqual` for any reason you can use
`go.temporal.io/api/temporalproto.DeepEqual` instead

Note that history loading will not be impacted by the JSON changes: I
rewrote history loading to dynamically fix incoming history JSON data
(like all our other sdks); you can find this code in [my fork of our go
API](https://github.com/tdeebswihart/temporal-api-go/blob/master/internal/temporalhistoryv1/load.go)
alongside its tests.

**🚨Sharp Edges Introduced🚨**

Beware `*timestamppb.Timestamp.AsTime()`. If you need to extract a time
value from a proto time (timestamppb) **always** make sure to check
whether it's nil first. When the proto object is `nil` `AsTime()` will
return a non-zero time at the proto epoch: UTC midnight on January 1,
1970.

I've made this mistake multiple times during this transition and each
time it's been a pain to debug

**Is hotfix candidate?**

No.
2023-11-21 15:19:37 -08:00

131 lines
6.0 KiB
Go

// The MIT License
//
// Copyright (c) 2020 Temporal Technologies Inc. All rights reserved.
//
// Copyright (c) 2020 Uber Technologies, Inc.
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in
// all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
// THE SOFTWARE.
package tdbg
import (
"fmt"
"io"
commonpb "go.temporal.io/api/common/v1"
"go.temporal.io/server/common/persistence/serialization"
"go.temporal.io/server/service/history/tasks"
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/proto"
)
type (
// TaskBlobEncoder takes a blob for a given task category and encodes it to a human-readable format.
// Here's a breakdown of the relationship between all the related types needed to implement a custom encoder:
// - NewCliApp accepts a list of Option objects.
// - Option objects modify Params.
// - Params contain a TaskBlobEncoder.
// - TaskBlobEncoder is implemented by ProtoTaskBlobEncoder.
// - ProtoTaskBlobEncoder uses protojson to marshal [proto.Message] objects from a TaskBlobProtoDeserializer.
// - TaskBlobProtoDeserializer is implemented by the stock PredefinedTaskBlobDeserializer.
// - PredefinedTaskBlobDeserializer deserializes [commonpb.DataBlob] objects into [proto.Message] objects.
TaskBlobEncoder interface {
Encode(writer io.Writer, taskCategoryID int, blob *commonpb.DataBlob) error
}
// TaskBlobEncoderFn implements TaskBlobEncoder by calling a function.
TaskBlobEncoderFn func(writer io.Writer, taskCategoryID int, blob *commonpb.DataBlob) error
// TaskBlobProtoDeserializer is used to deserialize task blobs into proto messages. This makes it easier to create
// an encoder if your tasks are all backed by protos. We separate this from the encoder because we don't want the
// encoder to be tied to protos as the wire format.
TaskBlobProtoDeserializer interface {
Deserialize(taskCategoryID int, blob *commonpb.DataBlob) (proto.Message, error)
}
// ProtoTaskBlobEncoder is a TaskBlobEncoder that uses a TaskBlobProtoDeserializer to deserialize the blob into a
// proto message, and then uses protojson to marshal the proto message into a human-readable format.
ProtoTaskBlobEncoder struct {
deserializer TaskBlobProtoDeserializer
}
// PredefinedTaskBlobDeserializer is a TaskBlobProtoDeserializer that deserializes task blobs into the predefined
// task categories that are used by Temporal. If your server has custom categories, you'll want to build something
// on top of this.
PredefinedTaskBlobDeserializer struct{}
)
var (
jsonpbMarshaler = protojson.MarshalOptions{
UseEnumNumbers: false, // It's ok to use strings because this is for human consumption.
Indent: " ", // Indent with two spaces to pretty-print.
UseProtoNames: true, // The proto field names are more JSON-esque than the Go field names.
}
)
// NewProtoTaskBlobEncoder returns a TaskBlobEncoder that uses a TaskBlobProtoDeserializer to deserialize the blob.
func NewProtoTaskBlobEncoder(deserializer TaskBlobProtoDeserializer) *ProtoTaskBlobEncoder {
return &ProtoTaskBlobEncoder{
deserializer: deserializer,
}
}
// NewPredefinedTaskBlobDeserializer returns a TaskBlobProtoDeserializer that works for the stock task categories of the
// server. You need to extend this if you have custom task categories.
func NewPredefinedTaskBlobDeserializer() PredefinedTaskBlobDeserializer {
return PredefinedTaskBlobDeserializer{}
}
// Deserialize a task blob from one of the server's predefined task categories into a proto message.
func (d PredefinedTaskBlobDeserializer) Deserialize(categoryID int, blob *commonpb.DataBlob) (proto.Message, error) {
switch categoryID {
case tasks.CategoryIDTransfer:
return serialization.TransferTaskInfoFromBlob(blob.Data, blob.EncodingType.String())
case tasks.CategoryIDTimer:
return serialization.TimerTaskInfoFromBlob(blob.Data, blob.EncodingType.String())
case tasks.CategoryIDVisibility:
return serialization.VisibilityTaskInfoFromBlob(blob.Data, blob.EncodingType.String())
case tasks.CategoryIDReplication:
return serialization.ReplicationTaskInfoFromBlob(blob.Data, blob.EncodingType.String())
case tasks.CategoryIDArchival:
return serialization.ArchivalTaskInfoFromBlob(blob.Data, blob.EncodingType.String())
default:
return nil, fmt.Errorf("unsupported task category %v", categoryID)
}
}
// Encode a blob for a given task category to a human-readable format by deserializing the blob into a proto message and
// then pretty-printing it using protojson.
func (e *ProtoTaskBlobEncoder) Encode(writer io.Writer, categoryID int, blob *commonpb.DataBlob) error {
message, err := e.deserializer.Deserialize(categoryID, blob)
if err != nil {
return fmt.Errorf("failed to deserialize task blob: %w", err)
}
bs, err := jsonpbMarshaler.Marshal(message)
if err != nil {
return fmt.Errorf("failed to marshal task blob: %w", err)
}
_, err = writer.Write(bs)
if err != nil {
return fmt.Errorf("failed to write marshalled task blob: %w", err)
}
return nil
}
// Encode the task by calling the function.
func (e TaskBlobEncoderFn) Encode(writer io.Writer, taskCategoryID int, blob *commonpb.DataBlob) error {
return e(writer, taskCategoryID, blob)
}