An EC2 instance can be running while users cannot submit an order. A container can restart while its worker repeats an external action. Useful reliability work connects infrastructure behavior to the business operation the application must complete.
Consider a hypothetical Next.js frontend, NestJS API, Python processing service, database, and workers on AWS. This is a design starting point. Redundancy, recovery objectives, and operating cost depend on the workload and the organization running it.
Treat recovery as an observable loop
The AWS Well-Architected Reliability Pillar considers foundations, workload architecture, change management, and failure recovery together. A process restart is one mechanism within that larger design.
Observe
Collect outcomes, latency, queue age, structured logs, and traces. A running process does not prove that an operation completed.
Observe the operation from end to end
Choose signals that answer concrete questions. Can users reach the API? Is accepted work completing? Is a partner slowing down? Is an error concentrated in a new release? CPU usage alone cannot answer these questions.
- Request metrics: rate, latency percentiles, and failures by bounded route and operation labels.
- Workflow metrics: queue age, accepted-to-completed time, failures, and pending reconciliation.
- Structured logs: service, environment, release, request ID, operation ID, and an explicit outcome.
- Traces: connect API requests to downstream calls and propagate context through background work.
Amazon CloudWatch provides metrics, alarms, dashboards, and log tools for applications and infrastructure. Configure collection deliberately; not every application-level signal appears simply because the workload runs on AWS.
1{2 "service": "fulfillment-worker",3 "environment": "production",4 "release": "sha-a94c2f",5 "requestId": "req-92f1",6 "operationId": "order-483-revision-2",7 "stage": "partner-submit",8 "outcome": "reconcile-required",9 "durationMs": 300010}
Redact credentials and personal information before collection. Keep high-cardinality request IDs in logs or traces instead of creating a metric series per request. Use retention and sampling policies to keep telemetry useful and affordable.
Give health checks a clear purpose
Liveness and readiness answer different questions
Liveness asks whether the process can continue. Readiness asks whether an instance should receive traffic. If a global dependency outage fails liveness on every replica, restarts may add noise without fixing the dependency. Design each check around the recovery action it triggers.
A synthetic check should exercise a representative user path with controlled test data. Ensure it cannot create paid external effects. A response from /health does not prove the complete order or inference workflow works.
Bound the resources a failure can consume
Set downstream deadlines, cap worker concurrency, and apply per-partner limits. Unlimited waiting in a shared pool can let one slow integration exhaust database connections and block unrelated requests.
AWS’s discussion of timeouts, retries, and jitter explains how retries can increase load during outages. Retry safe operations with a bounded budget. Reconcile an ambiguous external write instead of assuming failure.
Adding workers does not increase a partner’s quota or create unlimited database connections. Identify the constrained dependency before treating more replicas as the solution.
Make deployments reversible
Build a versioned artifact, run relevant checks, and deploy it consistently. Attach the release ID to telemetry. Use a limited rollout when supported, compare user-facing indicators, and define when to stop or roll back.
- Configuration: validate required values before accepting traffic; keep secrets outside images.
- Schema changes: expand and contract so old and new versions can coexist during rollout.
- Rollback: verify the previous application still works with the current schema. Reverting an image does not undo every data change.
- Permissions: scope application and deployment identities to their responsibilities.
Agree recovery objectives, then test them
Recovery time objective is the targeted restoration time. Recovery point objective is the targeted data loss, commonly expressed as time. These business choices affect backup frequency, redundancy, procedures, and cost. A successful backup job does not prove either target.
1Scenario: primary database unavailable2Before: agree restoration-time and data-loss objectives3Exercise: restore a backup into an isolated environment4Verify: schema, access, records, and representative application workflows5Reconcile: accepted work and uncertain partner submissions6Record: elapsed time, gaps, owner, and follow-up actions
A useful exercise ends with application evidence: the representative operation works, recovered records make sense, and interrupted work has an owner. Test procedures in controlled conditions before depending on them during an incident.
Assess the design through three questions: what is failing, how is the impact bounded, and how will successful recovery be verified?