1. Establish the actual failure boundary

  • Write down the last known-good deployment version and timestamp.
  • Separate model-quality failure from serving, data, dependency, networking, configuration, and capacity failure.
  • Identify whether the incident affects every request, one route, one tenant, one model version, or one region.
  • Capture one failing request end to end before changing the system.

2. Restore useful observability before broad changes

  • Confirm request IDs propagate through gateway, serving layer, model runtime, and downstream dependencies.
  • Compare latency, error rate, saturation, queue depth, memory pressure, and dependency failures against the last known-good window.
  • Check for silent fallback behavior that can hide the original failure while producing degraded output.

3. Test the smallest reversible hypotheses first

  • Configuration or secret drift.
  • Missing or incompatible model artifact.
  • Schema or feature contract change.
  • Runtime or dependency version mismatch.
  • Capacity exhaustion, cold-start amplification, or queue backpressure.
  • Network, DNS, storage, registry, or permissions failure.

4. Decide whether rollback is actually safer

  • Verify the previous binary, model artifact, configuration, feature contract, and database assumptions are still compatible.
  • Prefer a bounded canary or traffic shift when the failure is uncertain.
  • Do not call rollback successful until requests, outputs, latency, and downstream effects are all re-observed.

5. Verify recovery as a user-visible outcome

  • Replay representative requests, including the request class that originally failed.
  • Check response correctness as well as HTTP success.
  • Confirm queues drain, saturation falls, retries normalize, and error budgets stop deteriorating.
  • Record the causal chain: trigger → failure → detection → recovery action → observed recovery.

6. Convert the incident into a reusable runbook

  • Preserve only the evidence that changed the diagnosis or recovery action.
  • Automate repeatable collection and remediation steps rather than relying on a future operator to remember them.
  • Add a fast detection path for the specific recurrence pattern.

Need the deeper version?

Production ML Deployment Triage is the longer field guide by Evan Brown, published on Leanpub. It is designed for engineers who need a compact incident companion rather than a broad MLOps textbook.

View Production ML Deployment Triage on Leanpub →

The Leanpub page displays the current provider-native price and purchase terms.