Engineering AI that works. Building cloud platforms that scale.

Observed symptoms

p95 latency on one API endpoint has roughly tripled. Error rates are flat. It started shortly after a deployment.

Evidence to collect

• Latency by endpoint and release version
• Traces for the slowest requests
• Database query timings and plans
• The diff of the recent deployment

Diagnostic reasoning

Only one endpoint is slow and errors haven't moved, which points to a change in that code path rather than the infrastructure. Traces show the time is spent waiting on one database query that the new release added — and that query isn't using an index.

Remediation

• Roll back or switch off the change behind its feature flag
• Add the missing index through a reviewed migration
• Redeploy the fix

Validation

• p95 back within its objective for a sustained period
• The query plan now uses the index
• A latency regression test and alert are added

Observed symptoms

Pods for one service restart every few minutes. Their last state shows OOMKilled with exit code 137.

Evidence to collect

• kubectl describe pod (last state and exit code)
• Container memory usage against its limit
• Recent changes to limits, traffic or runtime settings
• A heap profile from a running pod

Diagnostic reasoning

OOMKilled means the container exceeded its memory limit. Memory climbs steadily under normal traffic instead of spiking with load, which suggests the runtime heap is sized above the container limit or there is a leak — not a traffic surge.

Remediation

• Align the runtime heap settings with the container limit
• Set requests and limits from observed usage
• Fix the leak if the heap profile confirms one

Validation

• No restarts across a full traffic cycle
• Memory plateaus below the limit
• An alert fires when memory approaches the limit

Observed symptoms

The pipeline fails at the deploy step because the rollout times out. The previous version is still serving traffic.

Evidence to collect

• Pipeline logs for the failed job
• kubectl rollout status and events for the new pods
• Readiness probe results
• The manifest diff for this release

Diagnostic reasoning

The new pods never become ready, and their events show the readiness probe failing. The release moved the health endpoint to a new path, but the probe in the manifest still checks the old one.

Remediation

• Correct the probe path in the manifest (or restore the old endpoint)
• Re-run the pipeline while the previous version keeps serving
• Add manifest validation to the pipeline

Validation

• The rollout completes and probes pass
• Smoke tests pass against the new version
• The validation step catches the same mistake in a test branch

Observed symptoms

An agent's “create ticket” step has started failing. Users see a generic error and the task stops.

Evidence to collect

• Agent traces: prompt, chosen tool and arguments, tool response
• The ticketing API's error logs
• Recent changes to tool schemas, prompts or model version

Diagnostic reasoning

The traces show the model sending a priority value the ticketing API now rejects. The API's allowed values changed, but the tool description given to the model wasn't updated, so the agent keeps generating the old value.

Remediation

• Update the tool definition with the allowed values
• Validate tool arguments before calling the API
• Return a structured error the agent can recover from

Validation

• Replaying the failed traces now succeeds
• Tool-call success rate returns to normal
• These cases are added to the agent's regression suite