Commit Graph
94 Commits
Author SHA1 Message Date
Nikolay TretyakandGitHub 87ea6512d4 pgx support (#4913)
<!-- Describe what has changed in this PR -->
**What changed?**
`Driver` interface (with two implementations) was added to the
postgresql persistence package in order to support other drivers than
`lib/pq`. Fixes https://github.com/temporalio/temporal/issues/1775

Essentially, this PR does
https://github.com/temporalio/temporal/pull/4914 in a
backward-compatible and extendable way

<!-- Tell your future self why have you made these changes -->
**Why?**
Since `lib/pq` does not support multi-node clusters


<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
**How did you test it?**
Existing tests


<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
**Potential risks**
No, since you should explicitly change the driver in your config file.
However, `connectAttributes` depend on the driver (e.g.,
`binary_parameters` is supported only by `lib/pq` and is not a part of
postresql standard), so one should be careful changing drivers in
production.


<!-- Is this PR a hotfix candidate or require that a notification be
sent to the broader community? (Yes/No) -->
**Is hotfix candidate?**
No
2023-10-09 17:38:51 -05:00
Alexandre VilainandGitHub 517443da3b Fix temporal-sql-tool database creation on postgresql when database name contains hyphen (#4735)
<!-- Describe what has changed in this PR -->
**What changed?**

Added double quotes around database name in the create database query
for PostgreSQL.

<!-- Tell your future self why have you made these changes -->
**Why?**

The issue has been created on temporal-operator project (see:
https://github.com/alexandrevilain/temporal-operator/issues/402).
When we try to use the `temporal-sql-tool` `create` command with a
database name containing hyphen(s), even if escaped in command, it
throws the following error: `pq: syntax error at or near \"-\"`.

<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
**How did you test it?**

Before the fix:
```bash
$ go run ./cmd/tools/sql --endpoint="localhost" --port="5432" --user="temporal-aws-dev" --password="$TEMPORAL_DEFAULT_DATASTORE_PASSWORD" --database="temporal-aws-dev" --plugin="postgres" create
2023-08-05T20:44:58.957+0200    ERROR   Unable to create SQL database.  {"error": "pq: syntax error at or near \"-\"", "logging-call-at": "handler.go:94"}
exit status 1
```

After the fix:
```bash
$ go run ./cmd/tools/sql --endpoint="localhost" --port="5432" --user="temporal-aws-dev" --password="$TEMPORAL_DEFAULT_DATASTORE_PASSWORD" --database="temporal-aws-dev" --plugin="postgres" create
$ echo $?
0
```

<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
**Potential risks**
Nothing.

<!-- Is this PR a hotfix candidate or require that a notification be
sent to the broader community? (Yes/No) -->
**Is hotfix candidate?**
No.
2023-09-25 11:34:41 -05:00
Rodrigo ZhouandGitHub f144062d7f Support count group by in visibility with SQL (#4666) 2023-07-27 16:03:09 -05:00
Roey BermanandGitHub a30e8a2e7d Implement build id scavenger (#4430) 2023-06-09 10:36:38 -07:00
Matt McShaneandGitHub 932003d936 Inline UpdateInfo Storage (#4420)
* Revert "Capture UpdateInfos in MutableState mutation (#4372)"

This reverts commit c78ee02eee.

* Revert "Adds update state indexing to MutableState (#4224)"

This reverts commit f8303d8e7e.

* Add UpdateInfo map to WorkflowExecutionInfo

This inlines the UpdateInfo index into the pre-existing execution blob
so that downstream persistence implementations won't see any change.
2023-05-31 13:46:55 -04:00
Roey BermanandDavid Reiss fea92f6358 Address comments from #4346 review (#4405)
Note: This commit came from a feature branch and is not expected to build.
2023-05-26 16:03:31 -04:00
Roey BermanandDavid Reiss d98b6668b0 Worker versioning leftovers (#4400)
Note: This commit came from a feature branch and is not expected to build.
2023-05-26 16:03:31 -04:00
Roey BermanandDavid Reiss cbfa37e0af Implement GetWorkerTaskReachability API (#4346)
Note: This commit came from a feature branch and is not expected to build.
2023-05-26 16:03:31 -04:00
Roey BermanandDavid Reiss 8a2a3e7a6a Implement task queue user data seed (forced replication) (#4259)
Note: This commit came from a feature branch and is not expected to build.
2023-05-26 16:03:31 -04:00
Roey BermanandDavid Reiss e3b52e558d Refactor to use new protos and persistence (#4194)
Note: This commit came from a feature branch and is not expected to build.
2023-05-26 16:03:31 -04:00
Yu XiaandGitHub 2218dea8fb Add replication dlq metrics (#3876) 2023-05-11 17:33:24 -07:00
Matt McShaneandGitHub f8303d8e7e Adds update state indexing to MutableState (#4224)
Add UpdateInfo records to MutableState

A mechanism for storing pointers into the workflow history representing
update accepted or completed events. Storage added for Cassandra,
PosgtreSQL, MySQL, and SQLite.
2023-05-03 12:24:32 -04:00
Yichao YangandGitHub 0befbfc700 Remove unused persistence GetHistoryTask API (#4003) 2023-03-06 14:32:18 -08:00
Rodrigo ZhouandGitHub b2a733823e Support CountWorkflowExecutions for SQL DB (#3955) 2023-02-14 16:59:12 -08:00
Alex ShtinandGitHub 743be2c562 Fix ExecutionTime rounding when using PostgreSQL (#3952) 2023-02-13 18:50:13 -08:00
Rodrigo ZhouandGitHub d7f4606485 Advanced visibility for PostgreSQL (#3896) 2023-02-02 22:16:37 -08:00
Rodrigo ZhouandGitHub d49786debf [Advanced Visibility with SQL] Adding PostgreSQL 12 schema and interface (#3844)
* Create base schema for PostgreSQL 12

* Changes to PostgreSQL 12 visibility schame to support advanced visibility

* Create PostgreSQL 12 db interface and configs
2023-01-31 14:35:51 -08:00
Rodrigo ZhouandGitHub 7e1ddb70dc Add GetIndexName method to visibility manager (#3820) 2023-01-27 00:44:39 -08:00
Rodrigo ZhouandGitHub 77f0af8625 Fix time fields of rows from GetClusterMembers (#3840) 2023-01-25 17:18:50 -08:00
Rodrigo ZhouandGitHub f5c2446dc3 Fix GetWorkflowExecution in PostgreSQL (#3816) 2023-01-24 12:20:45 -08:00
Yichao YangandGitHub b5ff033db0 Support arbitrary history task category in SQL persistence (#3489) 2022-10-19 16:08:01 -07:00
Rodrigo ZhouandGitHub 4213f95d0f Create visibility GetWorkflowExecution API (#3488) 2022-10-17 19:26:16 -07:00
Michael SnowdenandGitHub 73f94d9300 Implement GetAllHistoryTreeBranches for SQL persistence backends (#3438) 2022-09-29 17:00:06 -07:00
Alexander MaysandGitHub 74c74e0a45 Remove rds iam auth plugin (#3079)
* Revert "Fix auth plugin configuration to match the new SQLAuthPlugin config block (#2844)"

This reverts commit 37ab2b5664.

* Revert rds-iam-auth - current drivers dont support authentication callbacks

Signed-off-by: Alexander Mays <[email protected]>
2022-07-10 14:31:02 -07:00
Alexander MaysandGitHub b8f3f89555 Add RDS IAM auth plugin for SQL drivers (#2830)
Signed-off-by: Alexander Mays <[email protected]>
Added:
A new environment variable to the CLI: SQL_AUTH_PLUGIN=rds-iam-auth
A new flag arg to the CLI: --sql-auth-plugin rds-iam-auth
2 new docker template variables: authPlugin: {{ default .Env.SQL_AUTH_PLUGIN "" }} and authPlugin: {{ default .Env.SQL_VIS_AUTH_PLUGIN "" }}
A new SQL configuration attribute authPlugin
2022-05-12 16:28:26 -07:00
Jeremy BreidingandGitHub 5d0296a421 Add sqlite persistence tests (#2681)
* Add sqlite persistence tests

* fix up a test

* Remove limit on prune operation for sql plugin
2022-03-30 13:05:33 -07:00
Yichao YangandGitHub ae23339fca Make matching persistence range queries [inclusive, exclusive) (#2599)
- Range for matching GetTasks is now [inclusive, exclusive), instead of (exclusive, inclusive]
- Range for matching CompleteTasksLessThan is now < max TaskID, instead of <=
- Range for admin ListTaskQueueTasks is now [inclusive, exclusive)
2022-03-15 10:42:12 -07:00
Yichao YangandGitHub be2e4a2d56 Support pagination for SQL GetTransfer/VisibilityTasks (#2564)
- support pagination for sql GetTransfer/VisibilityTasks
- improve pagination for sql GetTimer/ReplicationTasks
- new ExecutionMutableStateTaskSuite suite
2022-03-04 12:29:31 -08:00
Yichao YangandGitHub 3896610a3d Unify history task persistence range query behavior (#2547)
- Make range based on taskID and task fire time both [inclusive, exclusive)
2022-02-28 12:08:18 -08:00
Jeremy BreidingandGitHub a5ddc3ca73 Sql task queue visibility (#2537)
* Adding task_queue to read/write paths
for postgresql executions_visibility

* Adding task_queue for mysql and sqlite
2022-02-24 18:22:46 -08:00
Mikhail GryzykhinandGitHub ef2063c9e5 Add support for GetWorkflowHistory in reverse order (#2442)
Add API to fetch workflow history in reverse order
2022-02-14 12:49:02 -08:00
Yichao YangandGitHub 13b102065e Remove unused queue processor (#2473) 2022-02-08 10:39:11 -08:00
wxing1292andGitHub 7258e62519 Separate SQL session management into a dedicated file (#2391)
* Create reusable MySQL session util
* Create reusable PostgreSQL session util
2022-01-19 13:33:07 -08:00
Yu XiaandGitHub c3a5c35916 Remove cluster metadata dual write (#2358)
* Remove cluster metadata dual write

* Update sample cluster config
2022-01-11 14:09:09 -08:00
wxing1292andGitHub fa2db04ca3 Split lease task queue API into 2 (#2269)
* Split matching persistence LeaseTaskQueue API into
  * CreateTaskQueue
  * GetTaskQueue
* Add new task queue UT
* Add new task queue task UT
* Deprecate legacy matching task tests
* Modify logic accordingly
2021-12-10 21:39:05 -08:00
wxing1292andGitHub 0aa486930d Remove unnecessary start version from current execution table (#2236) 2021-11-29 14:46:56 -08:00
dbbbb27dd6 Add List Cluster Metadata API (#2206)
* Add List Cluster Metdata API

Co-authored-by: yux0 <[email protected]>
2021-11-19 14:31:25 -08:00
VitalyandGitHub a12c16d612 Add tiered storage queue processor (#2189) 2021-11-18 21:34:13 -08:00
5abf350905 Make cluster metadata schema backward compatible (#2177)
* Make cluster metadata schema backward compatible

Co-authored-by: yux0 <[email protected]>
2021-11-15 13:48:07 -08:00
c38d135021 Migrate cluster metadata store to use cluster metadata info table (#2155)
* Migrate cluster metadata store to use cluster metadata info table

Co-authored-by: yux0 <[email protected]>
2021-11-11 15:51:43 -08:00
Yimin ChenandGitHub 1ddc3c397f Promote local namespace to global namespace (#2105) 2021-11-01 15:05:26 -07:00
Michael NussbaumandGitHub 923d9ca555 Make SQL CLI tool more idempotent (#2104)
* Make PostgreSQL CLI tool DB create and schema create idempotent
* Don't error if a target schema version is specified and schema is up-to-date
2021-10-28 23:02:56 -07:00
Harshita PoojaryandGitHub 997d0211e1 Escape password in connection url (#2015)
* Escape password in connection url

* Run make goimports in root
2021-10-03 11:18:15 -07:00
wxing1292andGitHub 0ecaee5672 Use internal error to indicate code bug / contract breach (#1958)
* Update sql persistence layer
* Update cassandra persistence layer
* Update visibility persistence layer
2021-09-21 17:59:56 -07:00
wxing1292andGitHub b7a94537c0 Use serviceerror.Unavailable error type instead of serviceerror.Internal (#1916)
* Use serviceerror.Unavailable error type instead of serviceerror.Internal
  * serviceerror.Unavailable indicates errors like service's dependency is down, retryable
  * serviceerror.Internal indicates errors like NPE in service logic, non-retryable
2021-09-13 18:31:14 -07:00
Yimin ChenandGitHub 5b336f4883 Merge AppendHistory with CreateWorkflow (#1853) 2021-08-26 17:12:34 -07:00
Yimin ChenandGitHub c0f5486fd6 Refactor Queue for namespace replication (#1625)
* Refactor Queue for namespace replication

* fix test
2021-06-15 14:00:44 -07:00
wxing1292andGitHub a865a21973 Show connection error details for PostgreSQL layer (#1581)
* Return errors encountered when trying to connect to PostgreSQL compatible DBs
2021-05-27 17:42:44 -07:00
wxing1292andGitHub a8efffb0e7 Implement history node deletion logic (#1443)
* SQL: Rename SelectFromHistoryNode to RangeSelectFromHistoryNode
* SQL: Rename DeleteFromHistoryNode to RangeDeleteFromHistoryNode
* SQL: New DeleteFromHistoryNode aiming to delete one record from history node table
2021-04-13 16:52:53 -07:00
wxing1292andGitHub 65cadaed5c Unify ReadHistoryBranch logic (#1441)
* New InternalHistoryNode struct containing history node info
* Modify SQL persistence layer to use key set pagination, instead of existing hacky solution
* ReadHistoryBranch in addition will allow returning history node metadata only
* Move history node filtering logic from persistence layer to upper history store layer
2021-04-13 16:14:40 -07:00