Floemdata
AI

AI Explainability: Why Businesses Should Care

What to check before trusting AI output at work? A practical checklist on validation, explainability and human oversight, with a NIST reference.

Floemdata · · 3 min read
A decision tree with one highlighted path from the root to a single leaf

This article is not legal or financial advice. Before an AI output moves into a report, a quote or a decision, two questions are asked: can I trust it, and can I show why it gave this result? The first is reliability, the second explainability.

What does reliability depend on?

  • Measured accuracy: a tool's accuracy should rest on a test with your own examples, not on the vendor's description.
  • Consistency: does the same input give a similar output?
  • Knowing the limits: in which situations does the tool go wrong? Does it say so when it does not know?
  • Error types: which is more expensive, a false alarm or a missed event? Choose the threshold accordingly.

Explainability

Being able to explain an output means being able to show which data and which rule it rests on. At work this is often simple: which measure, which filter, which period was used? If these three pieces of information appear next to a number, a person can check it.

A framework: NIST AI RMF

The US NIST published the AI Risk Management Framework (AI RMF 1.0) on 26 January 2023. According to NIST's page, the framework is intended for voluntary use and aims to build trustworthiness considerations into the design, development, use and evaluation of AI products. We did not verify the framework's detailed list of characteristics in this article; those interested should consult the document itself.

A practical checklist

  • Define a step that compares critical output with the source.
  • Have a deterministic calculation, not the model, produce the number.
  • Show the data and rule used next to every output.
  • Make the person who decides aware that the output comes from AI.
  • Record faulty outputs and review them once a month.

Where Floemdata stands

The design principle of Floemdata Analytics is this: AI does not write SQL and does not do arithmetic; the report definition goes through a validator and the calculation happens in code. This is a design decision; an AI assistant is on the roadmap and is not yet a product feature (not verified).

To get started, pick an AI output you use today and recalculate its numbers by hand. Source: NIST, AI Risk Management Framework.

← All guides