All Writing
Solution Architect7 min read

Reliable event-driven integrations.

Connect databases, message brokers, and partner APIs without losing accepted work or treating every retry as safe.

By Muhammad Faizan Rauf.

Event-driven integration lets a product keep accepting work while downstream systems run at a different pace. It also creates handoffs where processes can fail. A broker cannot automatically make a database transaction and a partner request succeed as one operation.

Consider a hypothetical platform connecting merchants to personalization and fulfillment partners. Every accepted order should stay recoverable, tenant boundaries should remain intact, and duplicate messages should not silently create duplicate paid operations.

Follow ownership of accepted work

A useful event contract identifies its producer, tenant, business entity, and schema version. Follow the diagram to see what persists at each boundary.

1
Accept
Step 1 of 5Verify the source

Authenticate the webhook, resolve its tenant from a trusted integration, and validate the event before accepting work.

order-event.json
1{
2 "eventId": "evt-48f2",
3 "type": "order.accepted",
4 "schemaVersion": 1,
5 "tenantId": "merchant-17",
6 "orderId": "order-483",
7 "orderVersion": 3,
8 "occurredAt": "2026-10-14T09:30:00Z",
9 "correlationId": "request-a92e"
10}

Derive tenant ownership from the authenticated integration and authoritative records. A tenant ID in an incoming body is not sufficient authorization. Keep credentials and sensitive customer details out of messages when a consumer can fetch the minimum required fields through an authorized service.

Close the database-to-broker crash window

Updating an order and separately publishing an event creates a crash window. Reversing the actions can publish a change that later rolls back. AWS’s transactional outbox guidance describes storing business changes and pending events in one database transaction, then publishing committed events separately.

accept-order.sql
1-- Schematic PostgreSQL transaction; values are bound parameters.
2BEGIN;
3INSERT INTO accepted_webhooks (tenant_id, provider_event_id)
4VALUES ($1, $2); -- UNIQUE (tenant_id, provider_event_id)
5
6UPDATE orders SET status = 'accepted', version = version + 1
7WHERE tenant_id = $1 AND id = $3 AND status = 'pending';
8-- Verify exactly one row changed; otherwise roll back.
9
10INSERT INTO outbox (event_id, tenant_id, order_id, event_type)
11VALUES ($4, $1, $3, 'order.accepted');
12COMMIT;

This is a transaction sketch, not a complete dispatcher or migration. Receiving code must handle a duplicate webhook constraint by finding the prior operation. An unrelated database error must not be mistaken for an already processed event.

Publication can succeed before the dispatcher records success. As the same outbox guidance notes, duplicate messages remain possible. The outbox closes a lost-event gap; consumers still need to handle repeated delivery.

Separate event identity from operation identity

An event ID identifies a delivery fact. A business operation key identifies an intended effect, such as merchant-17:order-483:fulfill:revision-2. Different events may refer to the same effect. Replaying an event and requesting a new approved revision need different identities.

Protect local state in one transaction

A consumer can insert an event ID into an inbox table and update local business state in the same transaction. A unique constraint handles concurrent duplicates. A rollback removes both changes, allowing recovery. Marking an event complete before updating state would instead lose work.

Protect a remote operation at its own boundary

A local inbox does not atomically cover a fulfillment request. AWS’s idempotent API guidance explains the value of caller-provided request identity. Verify the partner’s actual contract: key scope, retention, and behavior when a repeated key has changed parameters.

A timeout leaves the outcome uncertain
The partner may have accepted fulfillment even when the response is lost. Preserve the operation key and payload. Use lookup or idempotency support to reconcile the outcome; if neither exists, make the uncertainty visible for review.

Define ordering and replay deliberately

Keep an entity version when an older event must not overwrite newer state. A status projection may ignore stale versions; an action processor might need a missing earlier transition before continuing. One “latest event wins” rule is unsafe for every workflow.

  • Schema evolution: version contracts and plan handling for unknown versions.
  • Replay scope: rebuilding a read model must not automatically resend paid fulfillment. Separate those consumers or make replay mode explicit.
  • Tenant fairness: cap concurrency and quotas per integration so one merchant cannot consume all capacity.

Give failures a bounded recovery path

Separate transient failures from invalid requests and missing permissions. AWS’s timeout and retry guidance discusses load amplification and backoff with jitter. Bound attempts and assign a clear retry owner instead of multiplying retries across layers.

An SQS dead-letter queue can isolate messages that exceed a configured receive threshold. Give it an operational owner: inspect failures, correct the cause, check prior external effects, and redrive deliberately.

Monitor oldest pending outbox age, queue age, and unreconciled operations. Test by stopping a dispatcher after publication and a worker after submission. Those interrupted operations should remain explainable and recoverable without inventing a fresh business action.