Commit Graph
33 Commits
Author SHA1 Message Date
Yimin ChenandGitHub 6503401bc7 Fix buffered events ordering (#7922)
## Summary
- ensure buffered events from SQL-based DBs are read in insert order by
adding `ORDER BY id`

## Testing
- `go test ./...` *(fails: downloading toolchain forbidden)*

------
https://chatgpt.com/codex/tasks/task_b_684efbf31978832cbebc08b6c9b70ba1
2025-06-16 14:30:20 -07:00
Alex ShtinandGitHub 8e8f1b0f7c Remove license header from every file (#7689)
## What changed?
<!-- Describe what has changed in this PR -->
Remove license header from every file. Because it is really hard to
follow in this PR here is the summary:
1. License header is removed from all `.go` and `.proto` files
:fireworks::fireworks:🎆.
2. `LICENSE` file in the root directory has only Temporal and Uber
copyrights.
3. 5 other `LICENSE` files added to the packages which have copyrights
different from Temporal and Uber: Datadog, Xargin, "Mat Ryer, Tyler
Bunnell and contributors".
4. `license_file` flag is removed from all code generation tools.
5. `copyright_file` flag is removed from `go:generate mockgen`
directive.
6. All copyright related targets are removed from `Makefile`.
7. Updated Temporal copyright year to 2025 everywhere.

## Why?
<!-- Tell your future self why have you made these changes -->
I double checked with legal department that it is not needed to have
license header in every file. One file per repo is enough. I put all
copyrights to the root `LICENSE` file and removed header from all other
files. Also updated tools and `Makefile`.
2025-05-01 18:50:21 -07:00
Rodrigo ZhouandGitHub 0de527be90 Add OnConflictOptions to StartWorkflowExecution (#7080)
## What changed?
<!-- Describe what has changed in this PR -->
Add `OnConflictOptions` to `StartWorkflowExecution`.
- if the workflow id conflict policy is not `USE_EXISTING`, then no-op;
- if the field is nil, then no-op;
- otherwise, then it will add a `WorkflowExecutionOptionsUpdatedEvent`
to the existing running workflow.

## Why?
<!-- Tell your future self why have you made these changes -->
Ability to attach completion callbacks to existing running workflows.

## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
WIP: writing tests

## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->

## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->

## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
2025-02-03 15:53:42 -06:00
YuriandGitHub 3eb4154958 Add start_time column to the current execution table. Some refactoring (#6310)
## What changed?
Add "start_time" nullable column to the current_executions table. 
Some refactoring of the existing code - send row as a parameter rather
then every single field of that row.

## Why?
Adding column - to allow fast checking workflow creation time to avoid
noisy workflow restarts. Without it we need to get workflow start time
from a mutable state, which can affect latency.

## Is hotfix candidate?
No
2024-07-24 14:17:46 -07:00
Tim Deeb-SwihartandGitHub be7b5514a5 Reconnect to SQL databases when connections fail (#5926)
## What changed?

Both our PostgreSQL and MySQL database backends will now automatically
reconnect to the database when certain errors occur: all errors chosen
have been experienced when testing this behavior through an AWS Aurora
RDS failover of either MySQL or PostgreSQL.

For both backends we will reconnect when we see:
- `ECONNRESET`
- `ECONNABORTED`
- `ECONNREFUSED`
- `io.EOF`
- `io.ErrUnexpectedEOF`
- `database/sql/driver.ErrBadConn`

for postgres we will also reconnect on the following SQLStates:
- `25006` read-only transaction
- `57P03` cannot connect now
- `0A000` feature not supported, but ONLY when the message is `cannot
set transaction read-write mode during recovery`

for mysql we will also reconnect when we see the following error codes:
- `1040` too many connections 
- `1792` read-only transaction (SQLstate `25006`)
- `1836` running in read-only mode

This logic is easily extensible should we discover more failure modes
over time

## Why?

We've had multiple community reports of Temporal problems during RDS
failover. One part of this is the fact that we wouldn't necessarily
reconnect; we were at the whims of our chosen SQL abstraction's
connection pooling logic.

## How did you test it?

I manually tested this functionality in the presence of repeated RDS
failovers: 

- [x] postgres12 plugin with pq driver
- [x] postgres12 plugin with pgx driver
- [x] mysql plugin

Automated testing will be added to our regular testing pipelines once
our infrastructure friends have added the support I need (it's in
progress)

## Potential risks

We're concerned there's a correctness issue in our PostgreSQL backend
that's related to our behavior during an RDS failover. If we merge this
before I figure out what's going on, we could hide the issue and make it
harder to reproduce.

## Documentation
N/A

## Is hotfix candidate?
Yes?
2024-05-17 16:59:15 -07:00
Yu XiaandGitHub 2218dea8fb Add replication dlq metrics (#3876) 2023-05-11 17:33:24 -07:00
Yichao YangandGitHub 0befbfc700 Remove unused persistence GetHistoryTask API (#4003) 2023-03-06 14:32:18 -08:00
Yichao YangandGitHub b5ff033db0 Support arbitrary history task category in SQL persistence (#3489) 2022-10-19 16:08:01 -07:00
Yichao YangandGitHub be2e4a2d56 Support pagination for SQL GetTransfer/VisibilityTasks (#2564)
- support pagination for sql GetTransfer/VisibilityTasks
- improve pagination for sql GetTimer/ReplicationTasks
- new ExecutionMutableStateTaskSuite suite
2022-03-04 12:29:31 -08:00
Yichao YangandGitHub 3896610a3d Unify history task persistence range query behavior (#2547)
- Make range based on taskID and task fire time both [inclusive, exclusive)
2022-02-28 12:08:18 -08:00
Yichao YangandGitHub 13b102065e Remove unused queue processor (#2473) 2022-02-08 10:39:11 -08:00
wxing1292andGitHub 0aa486930d Remove unnecessary start version from current execution table (#2236) 2021-11-29 14:46:56 -08:00
VitalyandGitHub a12c16d612 Add tiered storage queue processor (#2189) 2021-11-18 21:34:13 -08:00
wxing1292andGitHub cc0c337790 Begin the migration of executions table operation using atomic counter (#1420)
* Add db version attribute to executions table for better CAS guarantee
* 1.9.x will by default use existing next event ID for CAS, but will also support the new DB version CAS
* 1.10.x will by default use DB version CAS

Last event ID is not enough for CAS operation protection: e.g. activity heartbeat / buffer events.
The newly introduced DB version will be increased by 1 each time execution table is updated (update to mutable state record).
2021-04-05 15:03:57 -07:00
wxing1292andGitHub eb65ab4b26 Add visibility tasks table for SQL (#1055)
* Add SQL visibility_tasks table for Kafka deprecation
2020-12-09 11:17:23 -08:00
wxing1292andGitHub 24b80b2f91 Add context to all SQL interfaces (#980)
* Add context to all SQL interfaces
* Wire context with SQLx
* Set default execution DB timeout to 5s
* Set default visibility DB timeout to 10s
2020-11-11 16:30:36 -08:00
wxing1292andGitHub 0dc3f9d611 Split remaining SQL persistence interfaces into dedicated files (#872)
* Move ClusterMetadata into dedicated file
* Move HistoryExecutionBuffer into dedicated file
* Move HistoryShard into dedicated file
* Move Namespace into dedicated file
* Move MatchingTask into dedicated file
* Move MatchingTaskQueue into dedicated file
* Use filter struct directly instead of pointer to struct
2020-10-21 10:01:11 -07:00
wxing1292andGitHub 3f4c9d0742 Add SQL specific tests for replication tasks (#859)
* Some minor code structure change
* Add replication tasks tests
* Add replication DLQ tasks tests
2020-10-15 15:55:28 -07:00
wxing1292andGitHub 9aee69ab46 Add SQL specific tests for executions (#814)
* Add history current execution tests
* Add history execution tests
2020-10-08 15:23:04 -07:00
wxing1292andGitHub bb46fa2649 Split overloaded transfer / timer task APIs (#806)
* Split TransferTasksFilter into TransferTasksFilter and TransferTasksRangeFilter
* Split SelectFromTransferTasks into SelectFromTransferTasks and RangeSelectFromTransferTasks
* Split DeleteFromTransferTasks into DeleteFromTransferTasks and RangeDeleteFromTransferTasks
* Split TimerTasksFilter into TimerTasksFilter and TimerTasksRangeFilter
* Split SelectFromTimerTasks into SelectFromTimerTasks and RangeSelectFromTimerTasks
* Split DeleteFromTimerTasks into DeleteFromTimerTasks and RangeDeleteFromTimerTasks
2020-10-07 15:47:31 -07:00
wxing1292andGitHub 8f7e868b56 Add SQL specific tests for transfer / timer tasks (#802)
* Add history transfer task tests
* Add history timer task tests
* Add missing functionalities
2020-10-06 12:51:55 -07:00
Alex ShtinandGitHub 6a6909e92a Use vanity import package name go.temporal.io/server (#511) 2020-07-08 16:09:12 -07:00
Alex ShtinandGitHub 4a25002ccc Update LICENSE file and all go files with new license header (#285) 2020-04-08 20:39:01 -07:00
Alex ShtinandGitHub 0ba363e940 Rename WorkflowExecutionCloseStatus to WorkflowExecutionStatus (#271) 2020-04-01 13:16:09 -07:00
Alex ShtinandGitHub 88548fcb42 Rename domain to namespace (#260) 2020-03-28 12:35:46 -07:00
Shawn HathawayandGitHub 4b5c0fe07b WorkflowExecutionInfo/State Proto Persistence (#191)
- ProtoWorkflowExecutionInfo definition - Moves SQL from Thrift and Cassandra from frozen
- WorkflowExecutionInfo split into two objects (added WorkflowExecutionState) to support UpdateCurrentExecution workflows where the store layer wont want to fetch and manipulate an existing monolith proto. 
- Checksum proto introduced, Cassandra moves from frozen Checksum to proto
- This does not move any protos into the request layer and leaves InternalWorkflowMutableState, so common field mapping functions exist
- The go defined ReplicationInfo struct is consolidated to the proto replication package ReplicationInfo
2020-03-06 15:25:35 -08:00
Samar Abbas 8e55282c63 merge from cadence 02-18-2020 2020-02-23 01:29:24 -08:00
Yu XiaandGitHub 221ea8d0f3 Add operation APIs for history replication DLQ (#3019)
* Integrate CLI with history replication DLQ

* Add history replication DLQ
2020-02-12 17:53:28 -08:00
Alex Shtin b8169a84bf Merge remote-tracking branch 'cadence/master' into merge/cadence-12-19-19
Conflicts:
	.gen/go/history/history.go
	.gen/go/temporal/temporal.go
	cmd/server/cadence/cadence.go
	cmd/server/cadence/server.go
	cmd/tools/sql/main.go
	common/service/service.go
	docs/persistence.md
	go.mod
	go.sum
	service/frontend/dcRedirectionHandler.go
	service/frontend/service.go
	service/history/conflictResolver_test.go
	service/history/historyBuilder_test.go
	service/history/historyCache_test.go
	service/history/historyEngine.go
	service/history/historyEngine2_test.go
	service/history/historyEngine3_eventsv2_test.go
	service/history/historyEngine_test.go
	service/history/historyReplicator_test.go
	service/history/mutableStateBuilder_test.go
	service/history/nDCActivityReplicator_test.go
	service/history/nDCBranchMgr_test.go
	service/history/nDCConflictResolver_test.go
	service/history/nDCStateRebuilder_test.go
	service/history/nDCTransactionMgr_test.go
	service/history/nDCWorkflowResetter_test.go
	service/history/queueAckMgr_test.go
	service/history/replicatorQueueProcessor_test.go
	service/history/shardController_test.go
	service/history/stateBuilder_test.go
	service/history/taskProcessor_test.go
	service/history/timerQueueAckMgr_test.go
	service/history/timerQueueActiveProcessor_test.go
	service/history/timerQueueProcessorBase_test.go
	service/history/timerQueueStandbyProcessor_test.go
	service/history/transferQueueActiveProcessor_test.go
	service/history/transferQueueStandbyProcessor_test.go
	service/history/workflowResetor_test.go
	service/history/workflowResetter_test.go
	service/matching/handler.go
2019-12-19 16:07:20 -08:00
Quanzheng LongandGitHub d22e8104cf Enforce ordering in reading transfer tasks (#2916) 2019-12-17 17:11:22 -08:00
Yu XiaandGitHub 385dd85f24 [Kafka deprecation] Move ack level to source cluster (#2896)
* Add ack level to source cluster and source cluster can clean up replication task based on min ack level from all other clusters
* Add sync shard info from get replication messages API
2019-12-17 15:18:25 -08:00
Samar Abbas 34bd0a03c2 resolve all merge conflicts and unit tests to work again 2019-12-10 12:20:56 -08:00
Quanzheng LongandGitHub fdee70fac9 Persistence refactoring to be pluggable for any database (#2836) 2019-12-05 15:59:09 -08:00