mirror of
https://github.com/hanzoai/tasks.git
synced 2026-10-04 02:04:03 +00:00
Shahab TajikandGitHub
faba3e2eb8
<!-- Describe what has changed in this PR -->
**What changed?**
- Added a config to define a threshold for *negligible* backlog based on
the age of the oldest task in the backlog. The value is configurable per
TQ, but planning to set it to 5 secs by default in prod. A backlog older
than this threshold is considered non-negligible.
- Stop forwarding tasks when there is a non-negligible backlog
- This in turn stops pollers to be forwarded in presence of a
non-negligible backlog.
- This ensures that all the partition accumulate and process backlog at
the same rate. Effectively, the backlog age diff between partitions
should not grow significantly larger than the negligibility threshold
(e.g. 5 secs).
- There is an exception to this: forwarding tasks is allowed when the
partition has not received a poll recently (configurable, default
200ms). This is to ensure that in a TQ with few pollers, tasks and
pollers are do not stuck in different partitions for a long time.
- Prevent sync match when there is a non-negligible backlog. This is
already mostly happening (i.e. history tasks are all spooled and only
synced matched in race conditions, when there is a backlog). However,
forwarded tasks are also “sync match”ed in the root partition. In effect
this change makes sure that the root partition does not dispatch
forwarded tasks when itself has a backlog. This helps further with
keeping the partitions balanced.
- Also added the following metrics to be able to monitor poll and task
dispatch behavior at TQ partition level:
- `poll_latency` with `forwarded=false/true`
- `task_dispatch_latency` with `forwarded=false/true` and
`source=History/DbBacklog`
<!-- Tell your future self why have you made these changes -->
**Why?**
With previous forwarding logic, when a backlog started to form, the root
partition was effectively receiving more pollers and was able to process
its backlog faster. This mixed up the order of dispatching tasks: a new
task could be dispatched faster by going to the root partitions while
old tasks are backlogged in other partitions. This change prevents that
situation.
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
**How did you test it?**
- Unit tests
- Manually verified against 9 workload+worker configurations as defined
here:
https://github.com/ShahabT/temporal-tests/blob/main/balanced-partitions/experiments.go
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
**Potential risks**
The change is config protected and the default value is disabled (by
setting a very high threshold). Will be enabled by default in a future
release after successful verification in prod workloads.
<!-- Is this PR a hotfix candidate or require that a notification be
sent to the broader community? (Yes/No) -->
**Is hotfix candidate?**
No.