A successful task confirms that its code completed according to the task’s success conditions. It does not independently confirm that the correct records reached the report. Make the data acceptance checks part of the workflow before publishing a new reporting batch.
A load can finish with incomplete data
Imagine a daily order feed that should contain two source partitions. The extract delivers only one, and the load accepts it without an error. The code finishes, but the report describes only part of the agreed period.
This is an illustrative scenario. The missing control is an expectation about the data: which partitions should be present, when they should arrive, and what the operator should do when one is missing.
Define what ready means for this report
Start with the report owner’s decision. A daily order count needs a defined interval and deduplication rule. A sales total also needs agreement about cancellations, currency and late adjustments. A row count alone cannot establish that either result is correct.
| Check | Question to answer | Response to a failure |
|---|---|---|
| Completeness | Are all expected source partitions present? | Hold the new batch and investigate the missing interval. |
| Uniqueness | Does each business key have the intended number of records? | Inspect joins, retries and source duplicates. |
| Freshness | Does the data cover the period the report promises? | Show the last verified refresh and notify the owner. |
| Reconciliation | Do counts and agreed totals match the source? | Explain differences before publication. |
Put a decision between loading and publication
Separate staging from the data readers use. Validate the staged batch, retain the result of each check, and publish only when the agreed criteria pass. Where the architecture supports it, keep the last verified batch available and show that it is stale while recovery is in progress.
Retaining old data is not appropriate for every workflow. The owner must decide whether a stale view, an unavailable report or a partial result with explicit limits is acceptable. Do not silently present yesterday’s records as current.
Close the incident with evidence
Record the affected interval, correction, replay method and validation results. A safe retry needs a fixed input interval and a write strategy that does not create duplicate output. Check external side effects separately.
The useful handover is a repeatable procedure with stop conditions, a named operator and a clear escalation route. Start with the linked recovery guide and adapt its checks to your destination and business rules.