Floemdata
AI

AI-Written Report Summaries: Accuracy Risks

Letting a language model summarise a report saves time but risks invented commentary and wrong figures. Ways to limit the risk and Floemdata's stated principle.

Floemdata · · 5 min read
A plus-shaped pixel pattern and its copy: in the copy one arm cell is missing and a stray amber cell sits in a corner

On Monday morning a language model writes the “weekly sales summary” that goes to the manager. The text is fluent, the tone is right. But the summary says “sales rose 12 percent” while the report shows 8. The model misread the figure or calculated its own, and nobody noticed because the text looked trustworthy. The figures are fictional; the risk is real.

Four typical errors

  • Wrong figure: it picks a number from another row or another period of the table.
  • Its own arithmetic: it calculates the percentage, difference or total itself; language models can make arithmetic mistakes.
  • Invented cause: it adds an explanation such as “rose because of the campaign” that is not in the data.
  • Missing context: it skips a warning or a data gap; it says “all branches” though two branches' data did not arrive.

Ways to limit the risk

  • Do not let the model do the arithmetic. Let a tool calculate totals, differences and percentages; the model only turns the finished figure into a sentence.
  • Tie each number to its source. Show next to every number which report and period it came from, so the reader can click and check.
  • Separate cause from fact. Let the model write “what happened”, and “why” only where the data supports it; otherwise a person adds the cause.
  • Spot-check before publishing. In the first weeks, verify every number in the summaries by hand; if the error rate is not close to zero, do not send automatically.
  • Human approval. Let the report owner review the summary once before it reaches the manager.

A small test

Have the model write a summary for five of your old reports and mark every number in the summary against its counterpart in the report. The share of numbers that do not match is the most honest sign of whether you are ready for automation.

Where Floemdata stands

Floemdata's design decisions are written in its documents as follows: the AI fills in definitions and does not do arithmetic; the figures come from the compiler, and the dashboard and the assistant go through the same compiler. The AI assistant is on the roadmap in the product definition as a later stage (Phase 2); a feature that writes report summaries does not ship with the product today. The methods in this post also apply when you produce summaries with another tool.

To start, have a language model write one report summary this week and compare every number in it against the report. Questions are welcome at info@floemdata.com or via Floemdata.

← All guides