The 3 scariest words in the English language: “Hey, quick question…”
It’s 4:55 PM on a Friday.
The CEO DMs you:
“Hey, quick question. How does our Q3 churn correlate with users who skipped the onboarding wizard in North America?”
Why this matters
When the “why” behind a good question is missing or fragmented, even small ideas trigger large firefights. Analysts are stuck on data cleanup, AI agents hallucinate answers, and trust erodes. Solving the question quickly is great—doing it right the first time is how you keep execs confident and your team sane.
The Old Way
- Query the database.
- Realize
onboarding_statusis full of NULLs with no agreement on what NULL means. - Spend hours backfilling data and reconciling “NA” vs “North America” vs “US/CA.”
- Build a dashboard that won’t be opened after Tuesday.
Time to Value: 4 days + a new patch of gray hair.
The “AI” Way
You let the CEO ask your chatbot connected to the raw database. The AI confidently returns a number while missing the most critical business logic:
– is_active = 0 might mean churn—or it might mean “paused.”
– The onboarding wizard changed in May, so “skipped” isn’t comparable across Q3.
– “North America” could refer to billing country, IP, ship-to address, or org HQ, depending on the dataset.
– NULL is not false; it may mean “unknown,” “not captured,” or “field added late.”
Time to Value: -100, because now you spend Friday explaining why the AI “lied.”
The Future = Data Products for AI
If you want AI to answer ad-hoc questions reliably, stop feeding it raw ingredients and start serving it a meal. Treat data as a product that is designed, governed, and documented for downstream consumers—human and machine.
Here’s what that looks like in practice:
1) Business definitions as code
“Churn” shouldn’t be a debate. It should be a versioned definition with automated tests. Example: churn = cancelled OR inactive for N days OR non-renewal within the grace period. The agent doesn’t infer churn; it executes churn.
2) A semantic + metric layer with governed building blocks
Don’t let the agent freestyle SQL across 400 tables. Give it certified dimensions (region, segment, cohort), certified metrics (churn_rate, retention, activation), and certified time windows (Q3 based on your fiscal calendar, not guesswork).
3) Context-rich metadata to stop dumb math
Tag every column with meaning, constraints, and caveats: “revenue excludes tax,” “this is a snapshot as-of date,” “NULL means not captured before 2025-06-01,” “wizard_skipped is derived from event telemetry so missing events ≠ skipped.” This is where correctness is won.
4) Data quality + observability built in
If onboarding_status jumps to 40% NULL this week, the agent should flag the quality issue, quantify the impact, fall back to a safer proxy, or route to the data owner with a clear alert instead of fabricating certainty.
5) AI-native governance + authorization
The system enforces what the agent can and can’t answer: row/column-level access, PII masking, allowed metrics per role, and audit trails for every query. Yes, it can calculate churn by segment. No, it cannot calculate the CEO’s salary just because you asked nicely.
The Result
Your AI assistant becomes a specialized analyst, not a storyteller. It grabs the right governed Data Product, applies known definitions, runs the logic, and returns the answer in seconds—with the exact filters, definition version, lineage, and quality checks documented.
Apply this
- Audit your data products—list the datasets the AI answers rely on and document the missing definitions, cohorts, and filters.
- Codify the logic—turn fuzzy business terms into executable definitions and automate regression tests.
- Layer metadata—tag columns with meaning, constraints, and freshness so the agent knows the boundaries.
- Embed observability and governance—surface quality issues before the agent answers, and enforce access controls for each data product.
Time to Value: 30 seconds.
Friday Status: Happy Hour.
Stop building dashboards for questions that haven’t been asked yet. Build Data Products that empower AI to answer questions instantly.
