Commit Graph
78 Commits
Author SHA1 Message Date
Rodrigo ZhouandGitHub d49786debf [Advanced Visibility with SQL] Adding PostgreSQL 12 schema and interface (#3844)
* Create base schema for PostgreSQL 12

* Changes to PostgreSQL 12 visibility schame to support advanced visibility

* Create PostgreSQL 12 db interface and configs
2023-01-31 14:35:51 -08:00
Rodrigo ZhouandGitHub 7e1ddb70dc Add GetIndexName method to visibility manager (#3820) 2023-01-27 00:44:39 -08:00
Rodrigo ZhouandGitHub 77f0af8625 Fix time fields of rows from GetClusterMembers (#3840) 2023-01-25 17:18:50 -08:00
Rodrigo ZhouandGitHub f5c2446dc3 Fix GetWorkflowExecution in PostgreSQL (#3816) 2023-01-24 12:20:45 -08:00
Yichao YangandGitHub b5ff033db0 Support arbitrary history task category in SQL persistence (#3489) 2022-10-19 16:08:01 -07:00
Rodrigo ZhouandGitHub 4213f95d0f Create visibility GetWorkflowExecution API (#3488) 2022-10-17 19:26:16 -07:00
Michael SnowdenandGitHub 73f94d9300 Implement GetAllHistoryTreeBranches for SQL persistence backends (#3438) 2022-09-29 17:00:06 -07:00
Alexander MaysandGitHub 74c74e0a45 Remove rds iam auth plugin (#3079)
* Revert "Fix auth plugin configuration to match the new SQLAuthPlugin config block (#2844)"

This reverts commit 37ab2b5664.

* Revert rds-iam-auth - current drivers dont support authentication callbacks

Signed-off-by: Alexander Mays <[email protected]>
2022-07-10 14:31:02 -07:00
Alexander MaysandGitHub b8f3f89555 Add RDS IAM auth plugin for SQL drivers (#2830)
Signed-off-by: Alexander Mays <[email protected]>
Added:
A new environment variable to the CLI: SQL_AUTH_PLUGIN=rds-iam-auth
A new flag arg to the CLI: --sql-auth-plugin rds-iam-auth
2 new docker template variables: authPlugin: {{ default .Env.SQL_AUTH_PLUGIN "" }} and authPlugin: {{ default .Env.SQL_VIS_AUTH_PLUGIN "" }}
A new SQL configuration attribute authPlugin
2022-05-12 16:28:26 -07:00
Jeremy BreidingandGitHub 5d0296a421 Add sqlite persistence tests (#2681)
* Add sqlite persistence tests

* fix up a test

* Remove limit on prune operation for sql plugin
2022-03-30 13:05:33 -07:00
Yichao YangandGitHub ae23339fca Make matching persistence range queries [inclusive, exclusive) (#2599)
- Range for matching GetTasks is now [inclusive, exclusive), instead of (exclusive, inclusive]
- Range for matching CompleteTasksLessThan is now < max TaskID, instead of <=
- Range for admin ListTaskQueueTasks is now [inclusive, exclusive)
2022-03-15 10:42:12 -07:00
Yichao YangandGitHub be2e4a2d56 Support pagination for SQL GetTransfer/VisibilityTasks (#2564)
- support pagination for sql GetTransfer/VisibilityTasks
- improve pagination for sql GetTimer/ReplicationTasks
- new ExecutionMutableStateTaskSuite suite
2022-03-04 12:29:31 -08:00
Yichao YangandGitHub 3896610a3d Unify history task persistence range query behavior (#2547)
- Make range based on taskID and task fire time both [inclusive, exclusive)
2022-02-28 12:08:18 -08:00
Jeremy BreidingandGitHub a5ddc3ca73 Sql task queue visibility (#2537)
* Adding task_queue to read/write paths
for postgresql executions_visibility

* Adding task_queue for mysql and sqlite
2022-02-24 18:22:46 -08:00
Mikhail GryzykhinandGitHub ef2063c9e5 Add support for GetWorkflowHistory in reverse order (#2442)
Add API to fetch workflow history in reverse order
2022-02-14 12:49:02 -08:00
Yichao YangandGitHub 13b102065e Remove unused queue processor (#2473) 2022-02-08 10:39:11 -08:00
wxing1292andGitHub 7258e62519 Separate SQL session management into a dedicated file (#2391)
* Create reusable MySQL session util
* Create reusable PostgreSQL session util
2022-01-19 13:33:07 -08:00
Yu XiaandGitHub c3a5c35916 Remove cluster metadata dual write (#2358)
* Remove cluster metadata dual write

* Update sample cluster config
2022-01-11 14:09:09 -08:00
wxing1292andGitHub fa2db04ca3 Split lease task queue API into 2 (#2269)
* Split matching persistence LeaseTaskQueue API into
  * CreateTaskQueue
  * GetTaskQueue
* Add new task queue UT
* Add new task queue task UT
* Deprecate legacy matching task tests
* Modify logic accordingly
2021-12-10 21:39:05 -08:00
wxing1292andGitHub 0aa486930d Remove unnecessary start version from current execution table (#2236) 2021-11-29 14:46:56 -08:00
dbbbb27dd6 Add List Cluster Metadata API (#2206)
* Add List Cluster Metdata API

Co-authored-by: yux0 <[email protected]>
2021-11-19 14:31:25 -08:00
VitalyandGitHub a12c16d612 Add tiered storage queue processor (#2189) 2021-11-18 21:34:13 -08:00
5abf350905 Make cluster metadata schema backward compatible (#2177)
* Make cluster metadata schema backward compatible

Co-authored-by: yux0 <[email protected]>
2021-11-15 13:48:07 -08:00
c38d135021 Migrate cluster metadata store to use cluster metadata info table (#2155)
* Migrate cluster metadata store to use cluster metadata info table

Co-authored-by: yux0 <[email protected]>
2021-11-11 15:51:43 -08:00
Yimin ChenandGitHub 1ddc3c397f Promote local namespace to global namespace (#2105) 2021-11-01 15:05:26 -07:00
Michael NussbaumandGitHub 923d9ca555 Make SQL CLI tool more idempotent (#2104)
* Make PostgreSQL CLI tool DB create and schema create idempotent
* Don't error if a target schema version is specified and schema is up-to-date
2021-10-28 23:02:56 -07:00
Harshita PoojaryandGitHub 997d0211e1 Escape password in connection url (#2015)
* Escape password in connection url

* Run make goimports in root
2021-10-03 11:18:15 -07:00
wxing1292andGitHub 0ecaee5672 Use internal error to indicate code bug / contract breach (#1958)
* Update sql persistence layer
* Update cassandra persistence layer
* Update visibility persistence layer
2021-09-21 17:59:56 -07:00
wxing1292andGitHub b7a94537c0 Use serviceerror.Unavailable error type instead of serviceerror.Internal (#1916)
* Use serviceerror.Unavailable error type instead of serviceerror.Internal
  * serviceerror.Unavailable indicates errors like service's dependency is down, retryable
  * serviceerror.Internal indicates errors like NPE in service logic, non-retryable
2021-09-13 18:31:14 -07:00
Yimin ChenandGitHub 5b336f4883 Merge AppendHistory with CreateWorkflow (#1853) 2021-08-26 17:12:34 -07:00
Yimin ChenandGitHub c0f5486fd6 Refactor Queue for namespace replication (#1625)
* Refactor Queue for namespace replication

* fix test
2021-06-15 14:00:44 -07:00
wxing1292andGitHub a865a21973 Show connection error details for PostgreSQL layer (#1581)
* Return errors encountered when trying to connect to PostgreSQL compatible DBs
2021-05-27 17:42:44 -07:00
wxing1292andGitHub a8efffb0e7 Implement history node deletion logic (#1443)
* SQL: Rename SelectFromHistoryNode to RangeSelectFromHistoryNode
* SQL: Rename DeleteFromHistoryNode to RangeDeleteFromHistoryNode
* SQL: New DeleteFromHistoryNode aiming to delete one record from history node table
2021-04-13 16:52:53 -07:00
wxing1292andGitHub 65cadaed5c Unify ReadHistoryBranch logic (#1441)
* New InternalHistoryNode struct containing history node info
* Modify SQL persistence layer to use key set pagination, instead of existing hacky solution
* ReadHistoryBranch in addition will allow returning history node metadata only
* Move history node filtering logic from persistence layer to upper history store layer
2021-04-13 16:14:40 -07:00
wxing1292andGitHub 1a4d21bc14 Add prev transaction ID attribute to history node table for events chaining (#1435)
* New last_history_node_txn_id within execution info tracking the last valid history node
* New prev_txn_id within history node table tracking previous valid history node record
2021-04-06 15:38:08 -07:00
wxing1292andGitHub cc0c337790 Begin the migration of executions table operation using atomic counter (#1420)
* Add db version attribute to executions table for better CAS guarantee
* 1.9.x will by default use existing next event ID for CAS, but will also support the new DB version CAS
* 1.10.x will by default use DB version CAS

Last event ID is not enough for CAS operation protection: e.g. activity heartbeat / buffer events.
The newly introduced DB version will be increased by 1 each time execution table is updated (update to mutable state record).
2021-04-05 15:03:57 -07:00
wxing1292andGitHub 048bbe8832 Improve compatibility with Aiven DB (#1413)
* Try using a set of default database names when initializing admin connection

Aiven DB is a PostgreSQL compatible DB, but comes with a different default DB name
2021-03-26 16:59:52 -07:00
Alex ShtinandGitHub 996824e53c Move config and dynamicconfig packages to the common (#1372) 2021-03-15 10:45:11 -07:00
wxing1292andGitHub a4e4b37136 Unify db version check (#1299)
* Add version check method to SQL interface
* Remove tools version check method in favor of functions defined in persistence package
* Refactor tests to not depend on tools package
2021-02-18 10:41:44 -08:00
Alex ShtinandGitHub c4d40f4db9 Bugfix: NoopResolver was always used for version compatibility check (#1233) 2021-01-28 07:47:41 -08:00
wxing1292andGitHub d3d96148fb Cleanup DB schema (#1130) 2021-01-27 16:07:17 -08:00
wxing1292andGitHub 066448b5cf Improve PostgreSQL config (#1214)
* Include connectAttributes as part of connection string
2021-01-22 11:27:44 -08:00
Alex ShtinandGitHub b530bb0f5f Add ServiceResolver interface and corresponding ServerOption (#1188) 2021-01-14 21:29:04 -08:00
wxing1292andGitHub a8430b6561 Cleanup debug logs (#1184) 2021-01-13 12:22:55 -08:00
Alex ShtinandGitHub 46fe71cf04 Drop database/keyspace if exists (#1181) 2021-01-12 22:07:05 -08:00
wxing1292andGitHub 3eeeb8d967 Bugfix schema migration (#1129)
* Since 1.5.x is still using the to be deprecated column names, postpone the deprecation
2020-12-23 12:58:13 -08:00
wxing1292andGitHub d38faebead Cleanup cluster metadata table (#1102) 2020-12-22 22:54:42 -08:00
wxing1292andGitHub eb65ab4b26 Add visibility tasks table for SQL (#1055)
* Add SQL visibility_tasks table for Kafka deprecation
2020-12-09 11:17:23 -08:00
wxing1292andGitHub f330e30bdd Fix PostgreSQL EnableHostVerification behavior (#1048) 2020-12-02 18:11:47 -08:00
wxing1292andGitHub d26f78a13d Remove forced namespace cache refresh (#1014) (#1016)
* Remove forced namespace cache refresh
* Make SQL insert into history tree idempotent

NOTE: Previously, if a namespace is newly created and not loaded
in cache, the namespace cache can do forced cache refresh reducing
the wait time for caller. This forced cache refreshment may able
to trigger a new callback for failover. Now, if a task with deleted
namespace is processed, namespace cache will trigger the above
cache refreshment and causing a deadlock. The only solution now is
to remove the namespace cache forced refreshment logic.
2020-11-20 14:09:12 -08:00