Commit Graph
155 Commits
Author SHA1 Message Date
Alex ShtinandGitHub 685f071ffa Rename scheduleID to scheduledEventID and startedID to startedEventID (#2992) 2022-06-15 11:48:11 -07:00
Alexander MaysandGitHub b8f3f89555 Add RDS IAM auth plugin for SQL drivers (#2830)
Signed-off-by: Alexander Mays <[email protected]>
Added:
A new environment variable to the CLI: SQL_AUTH_PLUGIN=rds-iam-auth
A new flag arg to the CLI: --sql-auth-plugin rds-iam-auth
2 new docker template variables: authPlugin: {{ default .Env.SQL_AUTH_PLUGIN "" }} and authPlugin: {{ default .Env.SQL_VIS_AUTH_PLUGIN "" }}
A new SQL configuration attribute authPlugin
2022-05-12 16:28:26 -07:00
nagl-temporalandGitHub 99cbfe6380 Fix sporadic duplicate key errors in mysql queue implementation (#2802)
* More informative failures in queuePersistenceTest

* Fix sporadic duplicate key errors in mysql queue

* Fix lint
2022-05-09 09:24:07 -07:00
Jeremy BreidingandGitHub efcfd1c538 Revert back to using file URI for sqlite dsn (#2731) 2022-04-19 19:47:35 -07:00
Jeremy BreidingandGitHub d3b480cab9 replace sqlite3 with modernc (#2704)
* replace sqlite3 with modernc

* file mode tests

* Expand tests to include file persistence for sqlite
2022-04-13 15:29:38 -07:00
Jeremy BreidingandGitHub 5d0296a421 Add sqlite persistence tests (#2681)
* Add sqlite persistence tests

* fix up a test

* Remove limit on prune operation for sql plugin
2022-03-30 13:05:33 -07:00
Yichao YangandGitHub ae23339fca Make matching persistence range queries [inclusive, exclusive) (#2599)
- Range for matching GetTasks is now [inclusive, exclusive), instead of (exclusive, inclusive]
- Range for matching CompleteTasksLessThan is now < max TaskID, instead of <=
- Range for admin ListTaskQueueTasks is now [inclusive, exclusive)
2022-03-15 10:42:12 -07:00
Yichao YangandGitHub be2e4a2d56 Support pagination for SQL GetTransfer/VisibilityTasks (#2564)
- support pagination for sql GetTransfer/VisibilityTasks
- improve pagination for sql GetTimer/ReplicationTasks
- new ExecutionMutableStateTaskSuite suite
2022-03-04 12:29:31 -08:00
Yichao YangandGitHub 3896610a3d Unify history task persistence range query behavior (#2547)
- Make range based on taskID and task fire time both [inclusive, exclusive)
2022-02-28 12:08:18 -08:00
Jeremy BreidingandGitHub a5ddc3ca73 Sql task queue visibility (#2537)
* Adding task_queue to read/write paths
for postgresql executions_visibility

* Adding task_queue for mysql and sqlite
2022-02-24 18:22:46 -08:00
Mikhail GryzykhinandGitHub ef2063c9e5 Add support for GetWorkflowHistory in reverse order (#2442)
Add API to fetch workflow history in reverse order
2022-02-14 12:49:02 -08:00
Yichao YangandGitHub 13b102065e Remove unused queue processor (#2473) 2022-02-08 10:39:11 -08:00
Anton TitovandGitHub 1eb5f58597 Single binary server (SQLite) and concurrency bugfixes for in memory storages (#2114) 2022-01-31 22:34:33 -08:00
Alex ShtinandGitHub 87c5e9873a Enable cgo by default and disable it explicitly for docker image build (#2396) 2022-01-20 17:06:23 -08:00
wxing1292andGitHub 7258e62519 Separate SQL session management into a dedicated file (#2391)
* Create reusable MySQL session util
* Create reusable PostgreSQL session util
2022-01-19 13:33:07 -08:00
Yu XiaandGitHub c3a5c35916 Remove cluster metadata dual write (#2358)
* Remove cluster metadata dual write

* Update sample cluster config
2022-01-11 14:09:09 -08:00
wxing1292andGitHub fa2db04ca3 Split lease task queue API into 2 (#2269)
* Split matching persistence LeaseTaskQueue API into
  * CreateTaskQueue
  * GetTaskQueue
* Add new task queue UT
* Add new task queue task UT
* Deprecate legacy matching task tests
* Modify logic accordingly
2021-12-10 21:39:05 -08:00
wxing1292andGitHub 0aa486930d Remove unnecessary start version from current execution table (#2236) 2021-11-29 14:46:56 -08:00
Alex ShtinandGitHub 881e87aa13 Pin MySQL version to 8.0.19 (#2210) 2021-11-22 15:21:28 -08:00
dbbbb27dd6 Add List Cluster Metadata API (#2206)
* Add List Cluster Metdata API

Co-authored-by: yux0 <[email protected]>
2021-11-19 14:31:25 -08:00
VitalyandGitHub a12c16d612 Add tiered storage queue processor (#2189) 2021-11-18 21:34:13 -08:00
5abf350905 Make cluster metadata schema backward compatible (#2177)
* Make cluster metadata schema backward compatible

Co-authored-by: yux0 <[email protected]>
2021-11-15 13:48:07 -08:00
c38d135021 Migrate cluster metadata store to use cluster metadata info table (#2155)
* Migrate cluster metadata store to use cluster metadata info table

Co-authored-by: yux0 <[email protected]>
2021-11-11 15:51:43 -08:00
Yimin ChenandGitHub 1ddc3c397f Promote local namespace to global namespace (#2105) 2021-11-01 15:05:26 -07:00
Michael NussbaumandGitHub 923d9ca555 Make SQL CLI tool more idempotent (#2104)
* Make PostgreSQL CLI tool DB create and schema create idempotent
* Don't error if a target schema version is specified and schema is up-to-date
2021-10-28 23:02:56 -07:00
Eng Zer JunandGitHub fd62d2afbc Move from io/ioutil to io and os package (#2062)
* This PR replaces the existing io/ioutil functions with their new definitions in io and os packages.
2021-10-25 17:20:08 -07:00
Jacob LeGroneandGitHub 30913221b1 Add schema setup and namespace registration utilities for SQLite (#2029)
* Add new public functions to the `schema/sqlite` package which help with initial schema setup and pre-registering namespaces to enable fast startup time (workaround for https://github.com/temporalio/temporal/issues/1336).

NOTE: SQLite support is still in development stage
2021-10-08 10:52:27 -07:00
Harshita PoojaryandGitHub 997d0211e1 Escape password in connection url (#2015)
* Escape password in connection url

* Run make goimports in root
2021-10-03 11:18:15 -07:00
Jacob LeGroneandGitHub c6eee98792 Add SQLite storage driver (#1968)
* Import SQLite driver from Temporalite: these files originated from https://github.com/DataDog/temporalite
* Add SQLite driver layer & UT
* Use mattn SQLite driver

NOTE: SQLite support is still in development stage
2021-09-29 14:20:40 -07:00
wxing1292andGitHub 0ecaee5672 Use internal error to indicate code bug / contract breach (#1958)
* Update sql persistence layer
* Update cassandra persistence layer
* Update visibility persistence layer
2021-09-21 17:59:56 -07:00
wxing1292andGitHub b7a94537c0 Use serviceerror.Unavailable error type instead of serviceerror.Internal (#1916)
* Use serviceerror.Unavailable error type instead of serviceerror.Internal
  * serviceerror.Unavailable indicates errors like service's dependency is down, retryable
  * serviceerror.Internal indicates errors like NPE in service logic, non-retryable
2021-09-13 18:31:14 -07:00
Yimin ChenandGitHub 5b336f4883 Merge AppendHistory with CreateWorkflow (#1853) 2021-08-26 17:12:34 -07:00
wxing1292andGitHub 827a5afa11 Use mysql.Config for mysql connection string management (#1705)
* Use mysql.Config for mysql connection string management
* Set mysql.Config.RejectReadOnly = true in case mysql encounters automatic failover event
2021-07-02 18:53:30 -07:00
Yimin ChenandGitHub c0f5486fd6 Refactor Queue for namespace replication (#1625)
* Refactor Queue for namespace replication

* fix test
2021-06-15 14:00:44 -07:00
Alex ShtinandGitHub f9a8a73309 Use time.Time in internal visibility APIs (#1576) 2021-06-02 15:29:04 -07:00
wxing1292andGitHub a865a21973 Show connection error details for PostgreSQL layer (#1581)
* Return errors encountered when trying to connect to PostgreSQL compatible DBs
2021-05-27 17:42:44 -07:00
wxing1292andGitHub 134dda2e35 Disable EnableDBRecordVersion for 1.10 (#1539)
* Disable EnableDBRecordVersion for 1.10
* Add new integration test framework
2021-05-11 11:59:58 -07:00
wxing1292andGitHub 9c9ad65417 Implement history node validation & trimming logic (#1449) 2021-04-14 00:07:40 -07:00
wxing1292andGitHub a8efffb0e7 Implement history node deletion logic (#1443)
* SQL: Rename SelectFromHistoryNode to RangeSelectFromHistoryNode
* SQL: Rename DeleteFromHistoryNode to RangeDeleteFromHistoryNode
* SQL: New DeleteFromHistoryNode aiming to delete one record from history node table
2021-04-13 16:52:53 -07:00
wxing1292andGitHub 65cadaed5c Unify ReadHistoryBranch logic (#1441)
* New InternalHistoryNode struct containing history node info
* Modify SQL persistence layer to use key set pagination, instead of existing hacky solution
* ReadHistoryBranch in addition will allow returning history node metadata only
* Move history node filtering logic from persistence layer to upper history store layer
2021-04-13 16:14:40 -07:00
wxing1292andGitHub 1a4d21bc14 Add prev transaction ID attribute to history node table for events chaining (#1435)
* New last_history_node_txn_id within execution info tracking the last valid history node
* New prev_txn_id within history node table tracking previous valid history node record
2021-04-06 15:38:08 -07:00
wxing1292andGitHub cc0c337790 Begin the migration of executions table operation using atomic counter (#1420)
* Add db version attribute to executions table for better CAS guarantee
* 1.9.x will by default use existing next event ID for CAS, but will also support the new DB version CAS
* 1.10.x will by default use DB version CAS

Last event ID is not enough for CAS operation protection: e.g. activity heartbeat / buffer events.
The newly introduced DB version will be increased by 1 each time execution table is updated (update to mutable state record).
2021-04-05 15:03:57 -07:00
wxing1292andGitHub 048bbe8832 Improve compatibility with Aiven DB (#1413)
* Try using a set of default database names when initializing admin connection

Aiven DB is a PostgreSQL compatible DB, but comes with a different default DB name
2021-03-26 16:59:52 -07:00
Alex ShtinandGitHub 996824e53c Move config and dynamicconfig packages to the common (#1372) 2021-03-15 10:45:11 -07:00
wxing1292andGitHub 4d3241c47a Make tests execute one SQL / CQL statement at a time (#1365)
* Change test code to execute one SQL / CQL statement at a time.

Some database does not handle well if seeing a batch of sql statements.
This PR breaks the SQL / CQL statements used by test setup and execute
one statement at a time.
2021-03-10 11:01:07 -08:00
Mikhail GryzykhinandGitHub a5d5b9fe5d Remove unnecessary Times(1) calls (#1321)
* Remove unnecessary Times(1) calls
2021-02-23 19:46:24 -08:00
wxing1292andGitHub a4e4b37136 Unify db version check (#1299)
* Add version check method to SQL interface
* Remove tools version check method in favor of functions defined in persistence package
* Refactor tests to not depend on tools package
2021-02-18 10:41:44 -08:00
Alex ShtinandGitHub c4d40f4db9 Bugfix: NoopResolver was always used for version compatibility check (#1233) 2021-01-28 07:47:41 -08:00
wxing1292andGitHub d3d96148fb Cleanup DB schema (#1130) 2021-01-27 16:07:17 -08:00
wxing1292andGitHub 066448b5cf Improve PostgreSQL config (#1214)
* Include connectAttributes as part of connection string
2021-01-22 11:27:44 -08:00