## What changed?
Add test to timer executor for invalid chasm pure task
## Why?
Verify that processing invalid chasm pure task still calls
`CloseTransaction`.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Set tree as dirty as executing pure task
## Why?
After executing an invalid pure task, the transaction needs to clean up
the tasks and possibly generate a new physical task for the next valid
pure task.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Replace calls to `payload.Encode` with
`sadefs.EncodeValue`/`sadefs.MustEncodeValue`.
## Why?
`payload.Encode` doesn't set the metadata type, which might be useful
when decoding. Besides, encoding search attributes value should've used
`sadefs.EncodeValue` anyway.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Remove keys with `nil` value from the memo when instantiating CHASM
Visibility.
## Why?
`nil` value is used to remove keys from the memo, and it needs to match
the behavior in workflows.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Add flag to skip setting up ES cluster settings. Eg:
`temporal-elasticsearch-tool setup-schema --skip-cluster-settings`
## Why?
Allow users to not use our provided cluster settings (eg: avoid
overwriting their cluster settings).
https://github.com/temporalio/temporal/issues/9857
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Create dynamic config for max service error message length
## Why?
Replacing constant max length with dynamic config for added flexibility.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Truncate service error message to prevent grpc-go to return `RST_STREAM`
instead of the service error.
## Why?
grpc-go returns `RST_STREAM` error when the error message is too long,
masking the actual service error returned by Temporal.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Add metric to track ListWorkflowExecutions query length
## Why?
Tracking query length
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
- Update Go 1.26.0 to 1.26.2
- Run `make update-dependencies` (skipped
`github.com/olekukonko/tablewriter`)
- Updated Docker Alpine version to 3.23.4
## Why?
Address CVEs
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Upgrade dependency of `aws-sdk-go` from v1 to v2.
Changes mostly made by agent
## Why?
Remove dependency of `aws-sdk-go` v1 which is already deprecated.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Introduce proto types for Visibility:
- `ChasmExecutionInfo`
- `ListChasmExecutionsRequest`
- `ListChasmExecutionsResponse`
Changed VisibilityManager API to use these protos.
## Why?
Make VisibilityManager API more accessible for potential external usage.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Add `WithSystemSearchAttributes` and `WithPredefinedSearchAttributes` to
`NameTypeMap` to overwrite the set of system/predefined search
attributes.
## Why?
Add more flexibility to be able to overwrite it, and together with
`SearchAttributesProvider`, the user can customize it.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Add `SearchAttributesMap.ToProto` function.
Refactored `VisibilityValue` to account for the supported types in
Visibility.
## Why?
Added a way to encode the CHASM search attribute to proto.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Move `encode_value.go` to `sadefs`.
## Why?
Use in CHASM without circular dependency.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Add metrics to track invalid search attributes comparisons in ES
## Why?
Be able to track how often and which namespaces are using invalid
comparisons.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Explicitly set the `"missing"` key in ES sorter. This is to address
pagination bug in OpenSearch 2.8.0+ (see
https://github.com/opensearch-project/OpenSearch/issues/8212#issuecomment-1615325095).
Run CI tests with latest version of OpenSearch 2 and OpenSearch 3.
## Why?
Full support of OpenSearch as visibility store.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Remove duplicate buildQueryParams function
## Why?
Probably introduced by accident during a merge.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Add chasm registry to `VisibilityFactory.NewVisibilityStore` interface
params
## Why?
`VisibilityStore` is now required to implement Chasm specific functions,
so the factory must include the chasm registry.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Fix test code
## Why?
PR https://github.com/temporalio/temporal/pull/8642 merged after PR
https://github.com/temporalio/temporal/pull/8590 refactoring Visibility
tests without rebasing introduced a bug.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Move the namespace filter from the inside the unified query converter to
the VisibilityStore code.
## Why?
That's a filter that needs to be added no matter the query converter
implementation.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Added functional test for eager workflow start with custom search
attributes and retry
## Why?
Verify that subsequent retry workflows have the correct custom search
attributes set and are visible from Visibility.
This is a follow up from #8585 (the new test fails before the PR change
and passes after)
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
## What changed?
Split `searchattribute.TestNameTypeMap` in two: one for ES setup, and
one for SQL DB setup.
## Why?
`searchattribute.TestNameTypeMap` is used in tests as the predefined
list of custom search attributes. However, it mixes how custom search
attributes work differently in Elasticsearch vs SQL databases. This PR
splits the variable in two (one for each) so the tests will be a better
representation of the actual system.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Unify Visibility query converters.
The new query converter is the struct `query.QueryConverter[ExprT]`. The
generic parameter specify the output type of the converter (in ES is
`elastic.Query`, in SQL is `sqlparser.Expr`).
The query converter has a `query.StoreQueryConverter[ExprT]`. This is
the Visibility store specific implementation for building the output
query.
For example, the store needs to implement `BuildAndExpr(exprs)` to tell
the query converter how to build the `AND` expression. In SQL, it's
`*sqlparser.AndExpr{exprs}`. In Elasticsearch, it's
`elastic.NewBoolQuery().Filter(exprs)`.
The store query converter for SQL needs a plugin query converter called
`sqlplugin.VisibilityQueryConverter` which is implemented by each plugin
(MySQL, PostgreSQL, SQLite). This interface is needed to build DB
specific syntax for `KeywordList` and `Text` searches as well as to
build the final `SELECT` statements.
Added dynamic config `system.VisibilityEnableUnifiedQueryConverter` that
acts as switch between the legacy and the unified query converters.
## Why?
There are two query converters: one for Elasticsearch, and another for
SQL.
They are somewhat similar, and have quite a few code duplications for
validation, modifying the query, etc.
Unifying the query converter ensures that we provide the exact same
behavior across different Visibility stores.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
I've added a lot of unit tests for the new code with coverage close to
100%.
Note that there has been basically no changes to functional tests, which
suggests that the change works well.
## Potential risks
There might be some edge cases that either query converters that this
unified query converter might deal differently.
## What changed?
Rename current Visibility query converter with 'legacy' suffix
## Why?
Preparation for the unified query converter that will replace the legacy
one.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Fix the search attributes `Decode` function to not swallow errors when
the type map is passed to the function.
## Why?
Recent PR https://github.com/temporalio/temporal/pull/8491 changed the
behavior allowing search attributes with metadata to passthrough when
the type map is not nil and the custom search attribute is not defined.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Alias search attributes of WorkflowStartedEvent in the eager workflow
task in the StartWorkflowResponse in frontend instead of history.
## Why?
Modifying in history modifies the object in history cache, which breaks
the retry logic.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Add a top-level Visibility config in the server config file.
Option to overwrite the default number of preallocated custom search
attributes in the server config
(`visibility.persistenceCustomSearchAttributes`).
Register the preallocated custom search attributes when using custom
Visibility store.
The config can be changed at any point: increasing the number of custom
search attributes will register additional custom search attributes in
the cluster metadata; decreasing the number of custom search attributes
is no-op.
Replace struct validator library with
`github.com/go-playground/validator`.
## Why?
Be able to add more custom search attributes when using SQL Visibility
store.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Move converting `time.Time` values for `Datetime` type search attributes
to the DB plugin code.
## Why?
This gives more flexibility for the plugin implementation to deal with
it as needed.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
If the visibility manager's search attribute cache is cold (ie., never
loaded yet), and reading cluster metadata failed (`Unavailable` error),
then retry randomly within 5 seconds instead of the default 20 seconds.
## Why?
If the cache is cold and there are custom search attributes, then
reading or writing workflows with custom search attributes would fail
for 20 seconds. Retrying within 5 seconds to minimize the impact.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Bump Temporal API to v1.52.0
## Why?
Bump Temporal API to v1.52.0
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Enable functional tests with OpenSearch 2 in CICD
## Why?
Run compatibility with OpenSearch 2
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Perform writes to dual visibility mode concurrently instead of
sequentially.
Fixed shadow read context: shadow read requests were using the same
context as the main request, so if the main request finishes earlier,
the context would close, and fail the shadow request immediately. This
PR copies the deadline so it has the same timeout, but it's a new
context so it won't be using the main context.
General refactor of dual visibility code.
## Why?
Each write buffers a Elasticsearch request, so executing sequentially
makes the overall visibility task latency much higher.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
