Prathyush PVandGitHub
35594955a6
<!-- Describe what has changed in this PR --> **What changed?** Adding documentation on how to deal with messages that are added to DLQs. <!-- Tell your future self why have you made these changes --> **Why?** This will help operators to find messages in DLQ, the reason why it got enqueued into DLQ etc. It also has useful tdbg commands for deleting or re-enqueuing the messages from DLQ <!-- How have you verified this change? Tested locally? Added a unit test? Checked in staging env? --> **How did you test it?** <!-- Assuming the worst case, what can be broken when deploying this change to production? --> **Potential risks** <!-- Is this PR a hotfix candidate or require that a notification be sent to the broader community? (Yes/No) --> **Is hotfix candidate?**
3.1 KiB
Managing DLQ Using tdbg
When a task in the Temporal server encounters a terminal error, it may be enqueued in the Dead Letter Queue (DLQ). A terminal error is a non-retryable error, such as failing to deserialize data due to data corruption. To effectively manage these DLQ messages, follow these steps:
- There is a metric
dlq_writes, which is incremented each time a message is enqueued to the DLQ. You can use this to determine when a task encountered a terminal error and needs manual resolution. - Search
Task enqueued to DLQin logs and see task details. From the log trace, you can identify the task type, source cluster, target cluster, and DLQ message ID. Source cluster and target cluster are different only for replication tasks. - From the task type, get the corresponding DLQ type from this map:
- transfer: 1
- timer: 2
- replication: 3
- visibility: 4
- You can list the DLQ messages using the command
tdbg dlq --dlq-version v2 read --dlq-type {type}. Substitute{type}with the integer value from step 3. You can specify the maximum message ID to read using --last-message-id flag. In case of replication tasks, you can specify the source cluster using the flag --cluster. - Search
Marking task as terminally failed, will send to DLQin logs and find the terminal error that caused the task to be enqueued to the DLQ.
If you can't find the logs, you can list all DLQs using the command tdbg dlq --dlq-version v2 list and find non-empty queues.
This command will list all queues in the decreasing order of message count.
Now these DLQ tasks can either be purged(removed from DLQ), or merged(Re-enqueued to the original queue and try to execute again).
To purge a message, execute the command tdbg dlq --dlq-version v2 purge --dlq-type {type} --last_message_id {message_id}.
Note that this command will purge all messages with an ID less than or equal to the specified message_id.
The output of this command will have a job token which can be used to manage the purge job.
Before executing this command, you can list the messages in the queue using the command mentioned in step 6 above to make sure more messages are not purged by mistake.
To merge a message, execute the command tdbg dlq --dlq-version v2 merge --dlq-type {type} --last_message_id {message_id}.
This command will merge all messages with an ID less than or equal to message_id back into the original queue for reprocessing.
The output of this command will have a job token that can be used to manage the merge job.
Once merge or purge command is executed, it will create a DLQ job which will process the DLQ messages. You can get the status of this job using the command
tdbg dlq --dlq-version v2 job describe --job-token {job-token}. The value of job-token will be printed in the output of merge and purge commands.
The output of the describe command will have details like the last processed message ID, number of messages processed, etc.
If you want to cancel a specific DLQ job, you can execute the command tdbg dlq --dlq-version v2 job cancel --job-token {job-token} --reason {reason}.