All Writing
Automation7 min read

AI automation that survives retries.

Turn a model-assisted support workflow into a system that can recover from duplicate events, interrupted workers, and uncertain deliveries.

By Muhammad Faizan Rauf.

A support message arrives. An LLM classifies it, drafts a reply, and an API sends the reply. The demo takes three boxes on a whiteboard. The production problem appears when the webhook arrives twice, the model takes too long, or the worker loses its connection just after the email provider accepts the send.

Reliability comes from making these situations explicit. The model can help interpret a message, while ordinary application code controls state transitions and external effects. The following design is a practical starting point for a reply-drafting workflow; it is not a claim of exactly-once delivery.

Design around the business operation
“Send the approved reply for ticket 483, revision 2” is an operation you can identify and recover. “Run this prompt again” is only another computation.

The full path from event to effect

Use the diagram to follow the durable record through the workflow. The important question at each step is: what would a replacement worker know after this process stops?

1
Receive
Step 1 of 5Verify the event

Authenticate the incoming webhook, validate its shape, and derive a stable event key. Persist the event before acknowledging it so a process crash cannot erase accepted work.

1. Accept the event once

Verify the webhook using the sender’s documented authentication method. Save its event ID under a unique database constraint. A repeated event should find the existing record, not create a second reply. Keep the event ID separate from the operation ID: two distinct events can still refer to the same business action.

If you write an event to a database and separately add a queue job, there is a crash window between the writes. An outbox design stores the pending job record in the same database transaction as the event. A dispatcher later publishes it; publishing twice must still be safe. This moves the recovery problem into a small, inspectable component.

2. Save progress as explicit states

workflow-states.txt
1received -> drafting -> needs_review -> approved -> sending -> sent
2 | | |
3 v v v
4 draft_failed retry_wait reconcile
5
6Terminal outcomes: sent | rejected | failed
7Operation key: ticket-483-reply-revision-2

Store timestamps, attempt counts, a lease or concurrency guard, the approved draft, and the provider receipt. State transitions should be conditional: one worker can claim an approved operation, and a stale worker cannot overwrite a newer result. A lease alone does not stop an external send after the lease expires; the delivery operation also needs protection.

Keep the model step small

Ask for a bounded output such as a category and a draft, then validate it. Limit input length and remove secrets before sending content to a model. Treat the incoming support message as untrusted data; it cannot authorize refunds, change recipients, or expand the workflow’s permissions.

Validate categories against an enum. Derive the recipient from the trusted ticket record, not from a model-generated address. If a refund amount appears in a draft, compare it with the authoritative order data. A response can satisfy a JSON schema while still contradicting the business facts.

Save the accepted draft and its prompt/model version before delivery. Otherwise, retrying a send might also regenerate the draft, creating two different messages for what was supposed to be the same action. Rejecting a draft and requesting a new revision should create an explicit new revision.

Put review at the consequential boundary
A workflow may automatically classify tickets and prepare replies. Sending, refunding, or changing account state should follow the user’s granted scope. If review is required, show the exact action and save what was approved.

Retry by failure type

BullMQ supports bounded attempts and backoff strategies, including exponential backoff and jitter. Its retry documentation describes the queue settings. This illustrative job waits longer between attempts; the processor must still classify errors and avoid retrying a permanent failure.

enqueue.ts
1await queue.add('draft-reply', { operationId }, {
2 jobId: operationId, // Use an ID without colons for BullMQ.
3 attempts: 4,
4 backoff: { type: 'exponential', delay: 1000, jitter: 0.5 },
5});
  • Transient failure: a temporary dependency outage may recover. Retry within a time and attempt budget, and respect the provider’s rate-limit instructions.
  • Permanent failure: missing permission, an invalid recipient, or an unsupported category needs a correction or terminal result. More attempts will not repair it.
  • Ambiguous effect: a timeout during sending leaves the outcome unknown. Query the provider using an operation key or receipt when possible; otherwise hold it for reconciliation.

Put the retry policy in a single responsible layer. If the HTTP client, queue worker, and outer workflow all retry independently, the actual number of external requests can multiply. Track a deadline for the business operation as well as a timeout for each attempt.

A job ID is not enough

BullMQ’s idempotent-job guidance recommends designing jobs so repeating them does not change the intended final outcome. Queue deduplication alone cannot protect an external email send, particularly after jobs are removed or delivery succeeds before the worker records completion.

Use a stable operation key at the provider if it offers durable idempotency with suitable retention. Store that key with the exact approved payload. Reusing a key for different content is a design error. When the provider lacks idempotency or lookup support, document the duplicate risk and make uncertain deliveries a recoverable review state.

Test recovery, not just the happy path

Replay the same event, stop the worker after drafting, and simulate a timeout immediately after the provider accepts a send. Each scenario should leave a record that explains what happened and a safe next action. Also test malformed model output and denied authorization: neither should reach delivery.

Monitor queue age, retry counts, validation failures, review backlog, and time spent in reconciliation. These reveal whether the automation is actually finishing work. A dashboard that only counts successful model responses misses the part the user cares about: whether the approved task completed correctly.