# Pipeline recovery runbook template

By Varveta · Updated 2026-10-02

Copy this worksheet before recovering a failed load. Record the affected data and destination write state, agree a safe replay, and collect the evidence needed to release downstream work. Replace every placeholder with facts from your environment.

## 1. Record the incident and impact

Keep secrets and customer-level data out of the incident record. Link to restricted evidence in your own systems when needed.

| Field | Record |
| --- | --- |
| Incident / run / task identifiers | [Identifiers and links] |
| Affected source, destination and interval | [Systems, timezone and exact interval] |
| Last verified output | [Checkpoint, refresh time and evidence] |
| Business impact and owner | [Affected reports, users and decision owner] |
| Operator and escalation contact | [Names or team channels] |

## 2. Decide whether replay is safe

Stop and escalate if you cannot establish whether destination writes committed, cannot isolate the affected interval, or lack authorization for the proposed correction.

- [ ] Identify the observed failure and record the supporting evidence.
- [ ] Inspect partial files, committed records and any downstream publication.
- [ ] Agree whether to pause downstream jobs or flag stale reports.
- [ ] Document a fixed replay interval and the duplicate-safe write strategy.
- [ ] Check notifications, external processes and other side effects separately.
- [ ] Record the reviewed correction and a rollback or recovery path.

## 3. Collect validation evidence

Set criteria before replay. The table names checks to consider; it does not supply universal thresholds.

| Check | Expected result | Observed result / evidence |
| --- | --- | --- |
| Expected partitions and interval | [Agreed coverage] | [Result and evidence link] |
| Business-key uniqueness | [Key and duplicate rule] | [Result and evidence link] |
| Source / destination reconciliation | [Counts, totals and tolerances] | [Result and evidence link] |
| Freshness and downstream report | [Agreed refresh and report checks] | [Result and evidence link] |

## 4. Record publication and follow-up

Publication decision: [approved / held, decision owner and time]. Remaining exceptions: [issue, owner and next action]. Repeated-failure prevention: [change, owner and review date].

This is an unfilled operating template, not a completed incident report. Rehearse the adapted procedure in a safe environment before relying on it during an incident.

## References

- [Apache Airflow: task design and retries](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html)

Illustrative template. Adapt the checks and operating procedure to your environment. This is not a completed test or incident record.