Data Pipeline Monitoring: Freshness, Volume, Errors
Which alerts do you really need? A minimum monitoring set for data freshness, row-count drift and run errors, and where Floemdata's monitoring stands today.

On Tuesday morning the sales manager opens the report, and the numbers are from Monday. The pipeline quietly stopped on Sunday night and nobody was told. The most expensive failure of a data pipeline is that nobody knows it has stopped.
A minimum monitoring set: three questions
- Freshness: when was the last successful load? Alert if it exceeds the expected interval (26 hours, say, for a daily pipeline).
- Volume: how many rows arrived in this run? Far below (or above) normal usually means a problem at the source. Zero rows is especially suspicious.
- Errors: did the run fail, at which step, with which message?
Set thresholds from the pipeline's own history, not from a guess: look at the typical row counts of the last two weeks.
Avoid alert fatigue
- Do not raise an alert for every small deviation; an alert nobody reads is the same as no alert.
- Give every alert an owner and a sentence that says what to do.
- Do not send success notices; show the "last successful load" time on a dashboard instead of relying on silence.
Make it visible to users too
A line above the report such as "Data last updated: 09.11.2026 06:10" shows readers stale data even when the alerting misses it. It is the cheapest form of monitoring.
Test with a weekend scenario
- Block access to the source for one night and see who is alerted, and when.
- Run the pipeline so that it returns empty data (in a test environment); did the volume rule catch it?
- Can the person who receives the alert read what to do from a document?
Where Floemdata stands
What we verified in the code: for each source system the warehouse stores the last extraction time (UTC) and the extraction status (succeeded, failed, skipped); the record count, duration and error summary per extraction sit in the same record. The agent sends a regular heartbeat to the central server. An anomaly layer also flags deviations in measured values using rules.
Not found: we found no rule that sets a freshness threshold and raises an alarm, that raises an alert from the record count automatically, or that sends a notification by e-mail; the record-count summary is, for now, a record rather than an alarm. We did not verify for this post how the central heartbeat data is shown and interpreted.
To start, print your pipeline's last successful run time above your reports. Questions are welcome at info@floemdata.com or via Floemdata.