How to find which change caused the outage: a change-aware RCA method
Working from the change timeline through the topology to the symptom
Most production outages follow a change, so the fastest route to root cause is to collect every change in the window before the first symptom, link each candidate through the dependency topology to the failing service, and test the candidates against the evidence. This guide sets out that method, and how to record the result as a statement of what changed rather than an assignment of blame.
Ask what changed before you ask what is anomalous. Anomaly detection tells you what looks unusual. The change timeline tells you what someone did. In an estate of thousands of resources, the second list is shorter, and it is where the cause usually is.
Why is change the first hypothesis?
A production system that was stable an hour ago and is failing now has, in the overwhelming majority of cases, been changed: a deployment, a configuration edit, an infrastructure-as-code apply, a feature flag flip, a certificate rotation, a scaling event or a cloud-provider change underneath it. Load and hardware failures happen, but they are the minority. Starting from the change list bounds the search to a handful of candidates with known timestamps and known scope, which is far more tractable than searching all telemetry for anomalies. It also produces the most actionable output: a specific diff that can be rolled back, and a specific process that let it through.
What counts as a change?
The change timeline needs to be broader than the deployment pipeline. In most estates the causal change is in a system nobody thought of as change management.
Application deployments
Releases, rollbacks and canaries from CI/CD, with the commit range and the approver.
Configuration and feature flags
Config repo commits, runtime config edits, flag flips, secrets and certificate rotations. Frequently untracked, frequently causal.
Infrastructure-as-code applies
Terraform, CloudFormation, Helm and similar applies, with the plan diff. A one-line change to a security group or a pool size is a classic cause.
Cloud-provider and platform events
Managed service maintenance, instance retirements, control-plane changes, quota changes and regional events under your workloads.
ITSM change records
Approved and emergency changes in ServiceNow or equivalent, including manual database changes and network changes that never touched a pipeline.
Scaling and traffic shifts
Autoscaling events, load balancer reconfigurations, DNS changes and traffic migrations. Not always initiated by a person, but changes nonetheless.
How do you find the causal change?
The method, in the order it is run during an investigation.
- 1
Fix the first-symptom time precisely
Not the first alert, which may lag, but the first customer-visible or telemetry-visible deviation. Use traces, error rates and synthetic checks. Everything else is measured relative to this timestamp.
- 2
Collect every change in the window before it
Pull all change sources for a window that is generous enough to include slow-burn causes: typically the preceding hours for most incidents, longer if a certificate, quota or scheduled job is suspected. Normalise into one timeline with source, scope, timestamp and approver.
- 3
Map each candidate change through the topology
For each change, identify the services it touched and, through the dependency graph, the services downstream of those. Discard candidates with no path to the failing service. What remains is a short list.
- 4
Test the remaining candidates against the evidence
For each candidate, ask what you would expect to see in telemetry if it were the cause, and check. Error signatures, latency shifts, resource curves and log content should line up with the change's scope and timing. Contradicting evidence eliminates the candidate; supporting evidence is recorded in the evidence index.
- 5
Confirm with reversal where safe
If a rollback or flag reversal was performed and the symptom cleared, that is strong evidence. If it was not, and it is safe, a controlled reversal in a non-production replica settles most remaining doubt.
- 6
Record what changed, with the diff and the approval record
The output is a change attribution record: the specific change, its diff, when it was applied, the approval or pipeline that admitted it, and the path from that change to the failure. Write it as what changed. The corrective action addresses the process, not the person.
- 7
State confidence and what would raise it
If two candidates remain and the evidence cannot separate them, say so, list both, and name the telemetry or test that would decide it. That is more useful than picking one.
What if the change was not in your pipeline?
Then the change timeline is incomplete, and that is itself a finding for the RCA. Manual database changes, network changes made from a console, and cloud-provider maintenance are the common gaps. Part of a mature change-aware practice is widening the sources the timeline draws from and, where a source cannot be instrumented, noting it as a coverage gap in the RCA readiness assessment so its absence is known before the next incident.
Onepane's investigation is change-aware by default.
Onepane assembles the change timeline from your deployment, IaC, configuration, ITSM and cloud-provider sources, links each candidate through your observed topology to the failing service and its owner, and delivers a Change Attribution Record with the diff and the approval record, framed as what changed. See /artifacts/change-attribution-record.
The bottom line: Fix the first-symptom time, collect every change before it from every source, map candidates through the topology, test them against evidence, confirm by reversal where safe, and record what changed with the diff. Say when the evidence cannot decide.
Onepane links deployment, IaC, config, ITSM and cloud-provider change through your topology to the failing service and delivers the Change Attribution Record.