Copy this worksheet before recovering a failed load. Record the affected data and destination write state, agree a safe replay, and collect the evidence needed to release downstream work. Replace every placeholder with facts from your environment.
1. Record the incident and impact
Keep secrets and customer-level data out of the incident record. Link to restricted evidence in your own systems when needed.
| Field | Record |
|---|---|
| Incident / run / task identifiers | [Identifiers and links] |
| Affected source, destination and interval | [Systems, timezone and exact interval] |
| Last verified output | [Checkpoint, refresh time and evidence] |
| Business impact and owner | [Affected reports, users and decision owner] |
| Operator and escalation contact | [Names or team channels] |
2. Decide whether replay is safe
Stop and escalate if you cannot establish whether destination writes committed, cannot isolate the affected interval, or lack authorization for the proposed correction.
- Identify the observed failure and record the supporting evidence.
- Inspect partial files, committed records and any downstream publication.
- Agree whether to pause downstream jobs or flag stale reports.
- Document a fixed replay interval and the duplicate-safe write strategy.
- Check notifications, external processes and other side effects separately.
- Record the reviewed correction and a rollback or recovery path.
3. Collect validation evidence
Set criteria before replay. The table names checks to consider; it does not supply universal thresholds.
| Check | Expected result | Observed result / evidence |
|---|---|---|
| Expected partitions and interval | [Agreed coverage] | [Result and evidence link] |
| Business-key uniqueness | [Key and duplicate rule] | [Result and evidence link] |
| Source / destination reconciliation | [Counts, totals and tolerances] | [Result and evidence link] |
| Freshness and downstream report | [Agreed refresh and report checks] | [Result and evidence link] |
4. Record publication and follow-up
Publication decision: [approved / held, decision owner and time]. Remaining exceptions: [issue, owner and next action]. Repeated-failure prevention: [change, owner and review date].
This is an unfilled operating template, not a completed incident report. Rehearse the adapted procedure in a safe environment before relying on it during an incident.
Technical references
Examples and templates are illustrative. Adapt acceptance criteria and operating procedures to your environment.