Floemdata
Data & ETL

Testing with Synthetic Data: What It Proves, What Not

Synthetic data checks logic and consistency but cannot prove how a real system's schema and dirt behave. The limits, a documented example and the right order of use.

Floemdata · · 5 min read
A round peg that fits a round hole beside an irregular shape too large to fit the same hole

To build a data pipeline without access to real customer data, most teams generate synthetic (fake) data. That is a sound start, but it turns into a wrong sentence somewhere along the way: the tests pass, so it will work on real data. That sentence overstates what the evidence can carry.

What synthetic data proves

  • That the logic is consistent in itself. Do totals match, are returns negative, is the discount spread across the lines?
  • Behaviour against traps you already know. Cancelled documents, service invoices, multi-currency lines: every case you deliberately plant can be tested.
  • Repeatability. Does the same input give the same output? On real data that is hard to measure cleanly.

What it does not prove

  • That the real schema is the same. You write the synthetic data, so it reflects how you believe the table looks, not how it really looks.
  • That unknown dirt is absent. A real system holds years of hand-typed, corrected and half-finished records. Synthetic data contains only the errors you thought of.
  • Scale and distribution. Volume, seasonal swings and outliers look different in reality.
  • That a rule is useful in real life. An alert that fires on fake data need not catch the event a real customer calls important.

An example: being wrong about the schema

Floemdata's development notes record this lesson plainly. The project uses a mock SAP database that carries the real SAP Business One table and column names. Even so, when the real schema was met, several columns turned out to differ from the mock: one movement table lacked the number column the mock assumed, the fiscal-period table signalled its lock status through a different field, and the real period table held monthly sub-periods while the mock held yearly ones. The note's summary: when adding a new extraction, do not trust the mock, check the real schema.

The right order

  • 1. Build and test the logic on synthetic data. Fast feedback, safe environment.
  • 2. Confirm the real schema separately. Read column names from the source system itself; do not guess.
  • 3. Validate on real data by reading only. Compare totals with the source system's own report.
  • 4. Word the result honestly. 'Verified on synthetic data' and 'verified on customer data' are two different sentences.

Where Floemdata stands

The only source that has run on real data is SAP Business One. The Logo and YouTube connectors are written and were tested against mock sources; they have not been tried against a real Logo or YouTube source. The anomaly engine is coded and was verified on synthetic data (it stays silent on healthy data and fires on deliberately corrupted data); it has not been verified on real customer data. The rule in this article applies to our own product too: working on synthetic data is a precondition for working, not proof that it works.

In your own project, start with a list: next to every test, write whether it ran on synthetic or real data. Questions are welcome at info@floemdata.com or via Floemdata.

← All guides