Before rerunning a failed pipeline, establish which data interval and outputs were affected. Check whether the failed task wrote partial results, make the rerun safe, and validate the destination data before releasing downstream work. A successful task status alone does not prove that the data is complete.
Establish what failed and who is affected
Start with the failed task, its logs and the last successful run. Record the affected data interval, source, destination and any dependent reports. Separate a delayed upstream delivery from a transformation error, permissions failure or unavailable destination; each calls for a different recovery action.
For example, a daily sales load may fail after writing some records. Rerunning the whole workflow immediately could duplicate those records or cause downstream reports to combine old and new data. Confirm the write state first, and tell the report owner which period is unreliable.
- Record the run and task identifiers and affected interval.
- Check destination records and any published files for partial output.
- Agree whether downstream jobs or report refreshes should pause.
Check that a retry will produce the right result
Airflow recommends designing tasks to produce the same outcome when rerun. Use a fixed input interval rather than whatever data happens to be newest. For database loads, consider an upsert or another write strategy that prevents duplicate records. The appropriate strategy depends on the destination and the business key.
Inspect external side effects too. A task that sends notifications or starts another process needs its own duplicate protection. Increasing retries will not repair incorrect source data or missing permissions; resolve the cause before repeating the operation.
Validate the data before releasing downstream work
Choose checks that represent the failed workload. Compare expected and actual partitions, row counts, key uniqueness and freshness. Reconcile a business total when one is available, and investigate differences rather than treating any nonempty table as a successful recovery.
For the sales-load example, check that the affected day contains the expected source orders, no order appears twice, and cancelled orders follow the agreed reporting rules. Then refresh the dependent report and ask its owner to confirm that the restored period is usable. These are example checks, not universal thresholds.
Turn the incident into an operating procedure
Write down the observed cause, safe recovery steps, validation evidence and the person responsible for follow-up. Include the conditions under which an operator should stop and escalate. A runbook should make the next recovery easier without encouraging someone to rerun destructive operations blindly.
If the same failure keeps returning, scope a stabilization review around the recurring pattern. Useful outputs include corrected dependencies, safe retries, data-quality checks, actionable alerts and a recovery walkthrough with the team that will operate the pipeline.
Technical references
Examples and templates are illustrative. Adapt acceptance criteria and operating procedures to your environment.