Data Extraction Errors: Rollback and Close
Errors in data extraction and transformation are unavoidable; what matters is being able to go back. Re-runnable loads, confirmed progress and month-end close.

Every data pipeline fails one day: a connection drops, a table arrives late, a date is read wrongly. What matters is not that errors never happen, but that you can go back safely when they do.
Types of errors
- Extraction error: a connection, timeout or permission problem.
- Missing or duplicate load: a document never arrives, or arrives twice.
- Transformation error: a type, date format or currency is misread.
- Later change at the source: a corrected or cancelled document, a changed cost.
Principles for going back
- Write loads so they can be re-run: if the same data arrives twice, the result does not change.
- Advance the progress marker only after the warehouse has confirmed the load; after a failed run the marker stays put and the next run retries the same window.
- Keep the raw data and do not overwrite it; make the transformation reproducible from the raw data.
- Do not pass over an error silently; show the deviation.
Month-end close
After the close, a correction can arrive at the source: a late-entered invoice, a cancellation. Quietly changing a closed number destroys trust. Keep the closing version, process the correction as a new version and show the difference between the two. Agree up front with accounting and your auditor in which cases retroactive corrections will be made.
Where Floemdata stands
According to the project documentation, the progress marker in Floemdata Analytics has two stages: when a batch is written to disk a 'pending' record is kept, and it is moved to the permanent position only after the center confirms the load; if sending fails the marker does not advance and the next run sends the same window again. The Bronze load can be re-run (the same record is not duplicated); Bronze is append-only and every version is kept with its extraction time, and Silver selects the latest version. If sending Gold data to the center fails, the pipeline warns and the next run retries. A feature such as a 'closing version' or an automatic difference report is not verified (not verified).
To get started, write down who is notified when your pipeline fails and how a load is re-run. Questions: info@floemdata.com.