Commit Graph
41 Commits
Author SHA1 Message Date
VitalyandGitHub a12c16d612 Add tiered storage queue processor (#2189) 2021-11-18 21:34:13 -08:00
Jacob LeGroneandGitHub 30913221b1 Add schema setup and namespace registration utilities for SQLite (#2029)
* Add new public functions to the `schema/sqlite` package which help with initial schema setup and pre-registering namespaces to enable fast startup time (workaround for https://github.com/temporalio/temporal/issues/1336).

NOTE: SQLite support is still in development stage
2021-10-08 10:52:27 -07:00
Jacob LeGroneandGitHub c6eee98792 Add SQLite storage driver (#1968)
* Import SQLite driver from Temporalite: these files originated from https://github.com/DataDog/temporalite
* Add SQLite driver layer & UT
* Use mattn SQLite driver

NOTE: SQLite support is still in development stage
2021-09-29 14:20:40 -07:00
Yimin ChenandGitHub 5b336f4883 Merge AppendHistory with CreateWorkflow (#1853) 2021-08-26 17:12:34 -07:00
Yimin ChenandGitHub c0f5486fd6 Refactor Queue for namespace replication (#1625)
* Refactor Queue for namespace replication

* fix test
2021-06-15 14:00:44 -07:00
wxing1292andGitHub 9c9ad65417 Implement history node validation & trimming logic (#1449) 2021-04-14 00:07:40 -07:00
wxing1292andGitHub a8efffb0e7 Implement history node deletion logic (#1443)
* SQL: Rename SelectFromHistoryNode to RangeSelectFromHistoryNode
* SQL: Rename DeleteFromHistoryNode to RangeDeleteFromHistoryNode
* SQL: New DeleteFromHistoryNode aiming to delete one record from history node table
2021-04-13 16:52:53 -07:00
wxing1292andGitHub 65cadaed5c Unify ReadHistoryBranch logic (#1441)
* New InternalHistoryNode struct containing history node info
* Modify SQL persistence layer to use key set pagination, instead of existing hacky solution
* ReadHistoryBranch in addition will allow returning history node metadata only
* Move history node filtering logic from persistence layer to upper history store layer
2021-04-13 16:14:40 -07:00
wxing1292andGitHub 1a4d21bc14 Add prev transaction ID attribute to history node table for events chaining (#1435)
* New last_history_node_txn_id within execution info tracking the last valid history node
* New prev_txn_id within history node table tracking previous valid history node record
2021-04-06 15:38:08 -07:00
wxing1292andGitHub cc0c337790 Begin the migration of executions table operation using atomic counter (#1420)
* Add db version attribute to executions table for better CAS guarantee
* 1.9.x will by default use existing next event ID for CAS, but will also support the new DB version CAS
* 1.10.x will by default use DB version CAS

Last event ID is not enough for CAS operation protection: e.g. activity heartbeat / buffer events.
The newly introduced DB version will be increased by 1 each time execution table is updated (update to mutable state record).
2021-04-05 15:03:57 -07:00
Alex ShtinandGitHub 996824e53c Move config and dynamicconfig packages to the common (#1372) 2021-03-15 10:45:11 -07:00
wxing1292andGitHub 4d3241c47a Make tests execute one SQL / CQL statement at a time (#1365)
* Change test code to execute one SQL / CQL statement at a time.

Some database does not handle well if seeing a batch of sql statements.
This PR breaks the SQL / CQL statements used by test setup and execute
one statement at a time.
2021-03-10 11:01:07 -08:00
wxing1292andGitHub a4e4b37136 Unify db version check (#1299)
* Add version check method to SQL interface
* Remove tools version check method in favor of functions defined in persistence package
* Refactor tests to not depend on tools package
2021-02-18 10:41:44 -08:00
Alex ShtinandGitHub b530bb0f5f Add ServiceResolver interface and corresponding ServerOption (#1188) 2021-01-14 21:29:04 -08:00
wxing1292andGitHub eb65ab4b26 Add visibility tasks table for SQL (#1055)
* Add SQL visibility_tasks table for Kafka deprecation
2020-12-09 11:17:23 -08:00
wxing1292andGitHub d26f78a13d Remove forced namespace cache refresh (#1014) (#1016)
* Remove forced namespace cache refresh
* Make SQL insert into history tree idempotent

NOTE: Previously, if a namespace is newly created and not loaded
in cache, the namespace cache can do forced cache refresh reducing
the wait time for caller. This forced cache refreshment may able
to trigger a new callback for failover. Now, if a task with deleted
namespace is processed, namespace cache will trigger the above
cache refreshment and causing a deadlock. The only solution now is
to remove the namespace cache forced refreshment logic.
2020-11-20 14:09:12 -08:00
wxing1292andGitHub 6b04aa3997 Optimize SQL layer supporting batch delete (#982)
* Optimize SQL layer supporting batch delete
* Split DeleteFromActivityInfoMaps into DeleteFromActivityInfoMaps and DeleteAllFromActivityInfoMaps
* Split DeleteFromTimerInfoMaps into DeleteFromTimerInfoMaps and DeleteAllFromTimerInfoMaps
* Split DeleteFromChildExecutionInfoMaps into DeleteFromChildExecutionInfoMaps and DeleteAllFromChildExecutionInfoMaps
* Split DeleteFromRequestCancelInfoMaps into DeleteFromRequestCancelInfoMaps and DeleteAllFromRequestCancelInfoMaps
* Split DeleteFromSignalInfoMaps into DeleteFromSignalInfoMaps and DeleteAllFromSignalInfoMaps
* Split DeleteFromSignalsRequestedSets into DeleteFromSignalsRequestedSets and DeleteAllFromSignalsRequestedSets
2020-11-16 16:50:30 -08:00
wxing1292andGitHub 24b80b2f91 Add context to all SQL interfaces (#980)
* Add context to all SQL interfaces
* Wire context with SQLx
* Set default execution DB timeout to 5s
* Set default visibility DB timeout to 10s
2020-11-11 16:30:36 -08:00
wxing1292andGitHub ef7860ce5d Add SQL specific tests for messages, part 2 (#914)
* Rewrite queue metadata APIs similar to transfer tasks, timer tasks
* Add queue metadata tests
* Add missing encoding type
2020-11-09 11:57:38 -08:00
wxing1292andGitHub a8e5c2fa03 Bugfix: matching persistence layer update sticky task queue (#935)
* Sticky task queue update should be done within transaction lock or conditional update
2020-11-02 11:54:52 -08:00
wxing1292andGitHub 5942de0661 Use filter struct instead of pointer for matching SQL layer (#934) 2020-10-31 11:10:14 -07:00
wxing1292andGitHub a13e753cd7 Add SQL specific tests for messages, part 1 (#892)
* Rewrite APIs similar to transfer tasks, timer tasks
* Add queue message tests
2020-10-26 19:13:45 -07:00
wxing1292andGitHub 0dc3f9d611 Split remaining SQL persistence interfaces into dedicated files (#872)
* Move ClusterMetadata into dedicated file
* Move HistoryExecutionBuffer into dedicated file
* Move HistoryShard into dedicated file
* Move Namespace into dedicated file
* Move MatchingTask into dedicated file
* Move MatchingTaskQueue into dedicated file
* Use filter struct directly instead of pointer to struct
2020-10-21 10:01:11 -07:00
wxing1292andGitHub 3f4c9d0742 Add SQL specific tests for replication tasks (#859)
* Some minor code structure change
* Add replication tasks tests
* Add replication DLQ tasks tests
2020-10-15 15:55:28 -07:00
Alex ShtinandGitHub 3f02bf677e Update license header in all go files (#837) 2020-10-13 15:23:23 -07:00
wxing1292andGitHub 08eefd0741 Split overloaded SQL filters (#834)
* Split ActivityInfoMapsFilter into ActivityInfoMapsSelectFilter and ActivityInfoMapsDeleteFilter
* Split TimerInfoMapsFilter into TimerInfoMapsSelectFilter and TimerInfoMapsDeleteFilter
* Split ChildExecutionInfoMapsFilter into ChildExecutionInfoMapsSelectFilter and ChildExecutionInfoMapsDeleteFilter
* Split RequestCancelInfoMapsFilter into RequestCancelInfoMapsSelectFilter and RequestCancelInfoMapsDeleteFilter
* Split SignalInfoMapsFilter into SignalInfoMapsSelectFilter and SignalInfoMapsDeleteFilter
* Split SignalsRequestedSetsFilter into SignalsRequestedSetsSelectFilter and SignalsRequestedSetsDeleteFilter
2020-10-12 18:05:17 -07:00
wxing1292andGitHub 9ff4ba4d3c Reduce redundant history tree creation for workflow reset (#825)
* When performing workflow reset & terminating current workflow, one unnecessary create history tree API is called, this PR remove the redundant call to improve performance
2020-10-12 17:24:43 -07:00
wxing1292andGitHub 33e374bbea Add SQL specific tests for history events (#820)
* Add history node tests
* Add history tree tests
2020-10-09 16:55:22 -07:00
wxing1292andGitHub a1369eeca6 Use timestamp instead of convert for time.Time pointer conversion (#819) 2020-10-08 15:40:59 -07:00
wxing1292andGitHub 9aee69ab46 Add SQL specific tests for executions (#814)
* Add history current execution tests
* Add history execution tests
2020-10-08 15:23:04 -07:00
wxing1292andGitHub e844e65f4b Change shard ID type from int to int32 (#811)
* Unify the shard ID type to int32
* Remove unused admin Kafka resend command
2020-10-07 23:46:48 -07:00
wxing1292andGitHub b802e5e67a Add SQL specific tests for visibility, part 2 (#813)
* Add visibility tests
* Add visibility pagination tests
2020-10-07 21:17:56 -07:00
wxing1292andGitHub 47ddc73ab1 Add SQL specific tests for visibility, part 1 (#809)
* Add visibility tests
* Unify the behavior on MySQL & PostgreSQL
2020-10-07 19:53:44 -07:00
wxing1292andGitHub bb46fa2649 Split overloaded transfer / timer task APIs (#806)
* Split TransferTasksFilter into TransferTasksFilter and TransferTasksRangeFilter
* Split SelectFromTransferTasks into SelectFromTransferTasks and RangeSelectFromTransferTasks
* Split DeleteFromTransferTasks into DeleteFromTransferTasks and RangeDeleteFromTransferTasks
* Split TimerTasksFilter into TimerTasksFilter and TimerTasksRangeFilter
* Split SelectFromTimerTasks into SelectFromTimerTasks and RangeSelectFromTimerTasks
* Split DeleteFromTimerTasks into DeleteFromTimerTasks and RangeDeleteFromTimerTasks
2020-10-07 15:47:31 -07:00
wxing1292andGitHub 8f7e868b56 Add SQL specific tests for transfer / timer tasks (#802)
* Add history transfer task tests
* Add history timer task tests
* Add missing functionalities
2020-10-06 12:51:55 -07:00
wxing1292andGitHub 0682f421f8 Add SQL specific tests for mutable state, part 2 (#800)
* Add history execution request cancel tests
* Add history execution signal tests
* Add history execution signal request tests
2020-10-06 12:08:05 -07:00
wxing1292andGitHub b28183f9b0 Add SQL specific tests for mutable state, part 1 (#799)
* Add history execution buffer tests
* Add history execution activity tests
* Add history execution child workflow tests
* Add history execution timer tests
2020-10-05 15:37:10 -07:00
wxing1292andGitHub c3f861e075 Add SQL specific tests (#796)
* Add history shard tests
2020-10-05 14:59:24 -07:00
wxing1292andGitHub 1f56c0492a Add SQL specific tests for namespace (#793)
* Add namespace tests
* Remove redundant select during creation of namespace
2020-10-05 14:34:04 -07:00
wxing1292andGitHub 845940ab38 Add SQL persistence specific tests (#790)
* Add matching task queue tests
2020-10-02 21:40:50 -07:00
wxing1292andGitHub c3d126b59a Add SQL persistence specific tests (#789)
* Add matching task tests
2020-10-02 17:58:59 -07:00