Floemdata
Data & ETL

Schema Drift: What Your Pipeline Should Do

A column is added, dropped or renamed in the source system. What should your data pipeline do that day? A data contract, three reactions and Floemdata's real status.

Floemdata · · 4 min read
Two combs meshing, with one extra hollow tooth on the right that has no partner on the left

On Monday morning the sales report comes up empty. The source system was updated and a column was renamed; the pipeline either stopped with an error or quietly read that column as blank. The second case is worse: the report opens, but the numbers are wrong.

Schema changes come in three forms

  • A column was added: usually harmless, but if the pipeline copies every column, the warehouse table grows in ways nobody planned.
  • A column was dropped or renamed: the field the pipeline expects is gone. This is where most breakage happens.
  • The type or meaning changed: the column keeps its name but now carries numbers instead of text, or the same code means something else. It raises no error, so it is the hardest to notice.

A data contract: a promise both sides can read

A data contract is a written list shared by the source and the pipeline: which tables, which columns, which types, which may be empty. It does not need an enterprise tool; a spreadsheet or a configuration file next to the pipeline will do. What matters is that the pipeline's expectation is written down.

What to do when the pipeline sees a difference

  • Stop and notify (strict): if an expected column is missing, skip that table and say who should look. Reports keep showing old but correct data.
  • Load and warn (lenient): ignore new columns, leave the missing one empty, log a warning. It keeps things moving, but an empty value can leak into a report.
  • Decide per column: strict for critical columns (amounts, dates, keys), lenient for informational ones.

Whichever you choose, the change must not stay silent. The price of strict behaviour is a report that is a day late; the price of silent behaviour is a wrong decision.

A short checklist

  • Does the pipeline owner hear about source-system updates in advance?
  • Does the pipeline compare the expected columns with the source on every run?
  • If a column is missing, does the report show the last good data or blanks?
  • Is it written down who decides what happens when a new column appears?

Where Floemdata stands

Floemdata's discovery step, run before setup, checks that the SAP Business One tables and the expected columns exist; if a table or column is missing, the profile is marked "not ready". The extraction manifest lists the columns to fetch one by one. If a user-defined field (UDF) allowed in the manifest does not exist in the source, extraction raises an error and stops.

What we did not find or did not try: we have not verified, in a real customer environment, how a running pipeline behaves when a source column is dropped; there is no check in the code that catches a changed column type or meaning. That a new column missing from the manifest is not fetched follows from the column list being written out explicitly; we did not test it separately.

To start, name the person who reads your source system's release notes and write the pipeline's table and column list on one page. Questions are welcome at info@floemdata.com or via Floemdata.

← All guides