Commit Graph
64 Commits
Author SHA1 Message Date
Mikhail GryzykhinandGitHub ef2063c9e5 Add support for GetWorkflowHistory in reverse order (#2442)
Add API to fetch workflow history in reverse order
2022-02-14 12:49:02 -08:00
Yichao YangandGitHub 13b102065e Remove unused queue processor (#2473) 2022-02-08 10:39:11 -08:00
wxing1292andGitHub 7258e62519 Separate SQL session management into a dedicated file (#2391)
* Create reusable MySQL session util
* Create reusable PostgreSQL session util
2022-01-19 13:33:07 -08:00
Yu XiaandGitHub c3a5c35916 Remove cluster metadata dual write (#2358)
* Remove cluster metadata dual write

* Update sample cluster config
2022-01-11 14:09:09 -08:00
wxing1292andGitHub fa2db04ca3 Split lease task queue API into 2 (#2269)
* Split matching persistence LeaseTaskQueue API into
  * CreateTaskQueue
  * GetTaskQueue
* Add new task queue UT
* Add new task queue task UT
* Deprecate legacy matching task tests
* Modify logic accordingly
2021-12-10 21:39:05 -08:00
wxing1292andGitHub 0aa486930d Remove unnecessary start version from current execution table (#2236) 2021-11-29 14:46:56 -08:00
dbbbb27dd6 Add List Cluster Metadata API (#2206)
* Add List Cluster Metdata API

Co-authored-by: yux0 <[email protected]>
2021-11-19 14:31:25 -08:00
VitalyandGitHub a12c16d612 Add tiered storage queue processor (#2189) 2021-11-18 21:34:13 -08:00
5abf350905 Make cluster metadata schema backward compatible (#2177)
* Make cluster metadata schema backward compatible

Co-authored-by: yux0 <[email protected]>
2021-11-15 13:48:07 -08:00
c38d135021 Migrate cluster metadata store to use cluster metadata info table (#2155)
* Migrate cluster metadata store to use cluster metadata info table

Co-authored-by: yux0 <[email protected]>
2021-11-11 15:51:43 -08:00
Yimin ChenandGitHub 1ddc3c397f Promote local namespace to global namespace (#2105) 2021-11-01 15:05:26 -07:00
Michael NussbaumandGitHub 923d9ca555 Make SQL CLI tool more idempotent (#2104)
* Make PostgreSQL CLI tool DB create and schema create idempotent
* Don't error if a target schema version is specified and schema is up-to-date
2021-10-28 23:02:56 -07:00
Harshita PoojaryandGitHub 997d0211e1 Escape password in connection url (#2015)
* Escape password in connection url

* Run make goimports in root
2021-10-03 11:18:15 -07:00
wxing1292andGitHub 0ecaee5672 Use internal error to indicate code bug / contract breach (#1958)
* Update sql persistence layer
* Update cassandra persistence layer
* Update visibility persistence layer
2021-09-21 17:59:56 -07:00
wxing1292andGitHub b7a94537c0 Use serviceerror.Unavailable error type instead of serviceerror.Internal (#1916)
* Use serviceerror.Unavailable error type instead of serviceerror.Internal
  * serviceerror.Unavailable indicates errors like service's dependency is down, retryable
  * serviceerror.Internal indicates errors like NPE in service logic, non-retryable
2021-09-13 18:31:14 -07:00
Yimin ChenandGitHub 5b336f4883 Merge AppendHistory with CreateWorkflow (#1853) 2021-08-26 17:12:34 -07:00
Yimin ChenandGitHub c0f5486fd6 Refactor Queue for namespace replication (#1625)
* Refactor Queue for namespace replication

* fix test
2021-06-15 14:00:44 -07:00
wxing1292andGitHub a865a21973 Show connection error details for PostgreSQL layer (#1581)
* Return errors encountered when trying to connect to PostgreSQL compatible DBs
2021-05-27 17:42:44 -07:00
wxing1292andGitHub a8efffb0e7 Implement history node deletion logic (#1443)
* SQL: Rename SelectFromHistoryNode to RangeSelectFromHistoryNode
* SQL: Rename DeleteFromHistoryNode to RangeDeleteFromHistoryNode
* SQL: New DeleteFromHistoryNode aiming to delete one record from history node table
2021-04-13 16:52:53 -07:00
wxing1292andGitHub 65cadaed5c Unify ReadHistoryBranch logic (#1441)
* New InternalHistoryNode struct containing history node info
* Modify SQL persistence layer to use key set pagination, instead of existing hacky solution
* ReadHistoryBranch in addition will allow returning history node metadata only
* Move history node filtering logic from persistence layer to upper history store layer
2021-04-13 16:14:40 -07:00
wxing1292andGitHub 1a4d21bc14 Add prev transaction ID attribute to history node table for events chaining (#1435)
* New last_history_node_txn_id within execution info tracking the last valid history node
* New prev_txn_id within history node table tracking previous valid history node record
2021-04-06 15:38:08 -07:00
wxing1292andGitHub cc0c337790 Begin the migration of executions table operation using atomic counter (#1420)
* Add db version attribute to executions table for better CAS guarantee
* 1.9.x will by default use existing next event ID for CAS, but will also support the new DB version CAS
* 1.10.x will by default use DB version CAS

Last event ID is not enough for CAS operation protection: e.g. activity heartbeat / buffer events.
The newly introduced DB version will be increased by 1 each time execution table is updated (update to mutable state record).
2021-04-05 15:03:57 -07:00
wxing1292andGitHub 048bbe8832 Improve compatibility with Aiven DB (#1413)
* Try using a set of default database names when initializing admin connection

Aiven DB is a PostgreSQL compatible DB, but comes with a different default DB name
2021-03-26 16:59:52 -07:00
Alex ShtinandGitHub 996824e53c Move config and dynamicconfig packages to the common (#1372) 2021-03-15 10:45:11 -07:00
wxing1292andGitHub a4e4b37136 Unify db version check (#1299)
* Add version check method to SQL interface
* Remove tools version check method in favor of functions defined in persistence package
* Refactor tests to not depend on tools package
2021-02-18 10:41:44 -08:00
Alex ShtinandGitHub c4d40f4db9 Bugfix: NoopResolver was always used for version compatibility check (#1233) 2021-01-28 07:47:41 -08:00
wxing1292andGitHub d3d96148fb Cleanup DB schema (#1130) 2021-01-27 16:07:17 -08:00
wxing1292andGitHub 066448b5cf Improve PostgreSQL config (#1214)
* Include connectAttributes as part of connection string
2021-01-22 11:27:44 -08:00
Alex ShtinandGitHub b530bb0f5f Add ServiceResolver interface and corresponding ServerOption (#1188) 2021-01-14 21:29:04 -08:00
wxing1292andGitHub a8430b6561 Cleanup debug logs (#1184) 2021-01-13 12:22:55 -08:00
Alex ShtinandGitHub 46fe71cf04 Drop database/keyspace if exists (#1181) 2021-01-12 22:07:05 -08:00
wxing1292andGitHub 3eeeb8d967 Bugfix schema migration (#1129)
* Since 1.5.x is still using the to be deprecated column names, postpone the deprecation
2020-12-23 12:58:13 -08:00
wxing1292andGitHub d38faebead Cleanup cluster metadata table (#1102) 2020-12-22 22:54:42 -08:00
wxing1292andGitHub eb65ab4b26 Add visibility tasks table for SQL (#1055)
* Add SQL visibility_tasks table for Kafka deprecation
2020-12-09 11:17:23 -08:00
wxing1292andGitHub f330e30bdd Fix PostgreSQL EnableHostVerification behavior (#1048) 2020-12-02 18:11:47 -08:00
wxing1292andGitHub d26f78a13d Remove forced namespace cache refresh (#1014) (#1016)
* Remove forced namespace cache refresh
* Make SQL insert into history tree idempotent

NOTE: Previously, if a namespace is newly created and not loaded
in cache, the namespace cache can do forced cache refresh reducing
the wait time for caller. This forced cache refreshment may able
to trigger a new callback for failover. Now, if a task with deleted
namespace is processed, namespace cache will trigger the above
cache refreshment and causing a deadlock. The only solution now is
to remove the namespace cache forced refreshment logic.
2020-11-20 14:09:12 -08:00
wxing1292andGitHub 6b04aa3997 Optimize SQL layer supporting batch delete (#982)
* Optimize SQL layer supporting batch delete
* Split DeleteFromActivityInfoMaps into DeleteFromActivityInfoMaps and DeleteAllFromActivityInfoMaps
* Split DeleteFromTimerInfoMaps into DeleteFromTimerInfoMaps and DeleteAllFromTimerInfoMaps
* Split DeleteFromChildExecutionInfoMaps into DeleteFromChildExecutionInfoMaps and DeleteAllFromChildExecutionInfoMaps
* Split DeleteFromRequestCancelInfoMaps into DeleteFromRequestCancelInfoMaps and DeleteAllFromRequestCancelInfoMaps
* Split DeleteFromSignalInfoMaps into DeleteFromSignalInfoMaps and DeleteAllFromSignalInfoMaps
* Split DeleteFromSignalsRequestedSets into DeleteFromSignalsRequestedSets and DeleteAllFromSignalsRequestedSets
2020-11-16 16:50:30 -08:00
wxing1292andGitHub 24b80b2f91 Add context to all SQL interfaces (#980)
* Add context to all SQL interfaces
* Wire context with SQLx
* Set default execution DB timeout to 5s
* Set default visibility DB timeout to 10s
2020-11-11 16:30:36 -08:00
wxing1292andGitHub ef7860ce5d Add SQL specific tests for messages, part 2 (#914)
* Rewrite queue metadata APIs similar to transfer tasks, timer tasks
* Add queue metadata tests
* Add missing encoding type
2020-11-09 11:57:38 -08:00
wxing1292andGitHub a8e5c2fa03 Bugfix: matching persistence layer update sticky task queue (#935)
* Sticky task queue update should be done within transaction lock or conditional update
2020-11-02 11:54:52 -08:00
wxing1292andGitHub 5942de0661 Use filter struct instead of pointer for matching SQL layer (#934) 2020-10-31 11:10:14 -07:00
wxing1292andGitHub a13e753cd7 Add SQL specific tests for messages, part 1 (#892)
* Rewrite APIs similar to transfer tasks, timer tasks
* Add queue message tests
2020-10-26 19:13:45 -07:00
wxing1292andGitHub 0dc3f9d611 Split remaining SQL persistence interfaces into dedicated files (#872)
* Move ClusterMetadata into dedicated file
* Move HistoryExecutionBuffer into dedicated file
* Move HistoryShard into dedicated file
* Move Namespace into dedicated file
* Move MatchingTask into dedicated file
* Move MatchingTaskQueue into dedicated file
* Use filter struct directly instead of pointer to struct
2020-10-21 10:01:11 -07:00
wxing1292andGitHub 3f4c9d0742 Add SQL specific tests for replication tasks (#859)
* Some minor code structure change
* Add replication tasks tests
* Add replication DLQ tasks tests
2020-10-15 15:55:28 -07:00
wxing1292andGitHub 75a0122292 Update PostgreSQL TLS config (#853)
* Allow CA or key / cert to be separately set for PostgreSQL
* Add support of host name verification & fix existing issue
2020-10-14 09:28:39 -07:00
wxing1292andGitHub 4d8d473fd2 Add TLS configuration for PostgreSQL (#849) 2020-10-13 19:00:26 -07:00
VitalyandGitHub f6a48e2b30 Adding dual read/write for immutable metadata (#840) 2020-10-12 21:22:25 -07:00
wxing1292andGitHub 08eefd0741 Split overloaded SQL filters (#834)
* Split ActivityInfoMapsFilter into ActivityInfoMapsSelectFilter and ActivityInfoMapsDeleteFilter
* Split TimerInfoMapsFilter into TimerInfoMapsSelectFilter and TimerInfoMapsDeleteFilter
* Split ChildExecutionInfoMapsFilter into ChildExecutionInfoMapsSelectFilter and ChildExecutionInfoMapsDeleteFilter
* Split RequestCancelInfoMapsFilter into RequestCancelInfoMapsSelectFilter and RequestCancelInfoMapsDeleteFilter
* Split SignalInfoMapsFilter into SignalInfoMapsSelectFilter and SignalInfoMapsDeleteFilter
* Split SignalsRequestedSetsFilter into SignalsRequestedSetsSelectFilter and SignalsRequestedSetsDeleteFilter
2020-10-12 18:05:17 -07:00
wxing1292andGitHub 9ff4ba4d3c Reduce redundant history tree creation for workflow reset (#825)
* When performing workflow reset & terminating current workflow, one unnecessary create history tree API is called, this PR remove the redundant call to improve performance
2020-10-12 17:24:43 -07:00
wxing1292andGitHub 7b086939f1 Enable PostgreSQL tests on github (#821)
* Enable PostgreSQL tests on Github
* Move some tests around to avoid import cycle
2020-10-09 17:16:40 -07:00