What AI can and cannot do for analysis

Someone drops a screenshot into Slack: “ChatGPT says revenue was up 18% last quarter.” No grain. No filter list. No note about refunds, taxes, or which system the rows came from. Three people react with fire emojis. You feel the familiar drop in your stomach, because you know how easy it is to get a fluent wrong number.

That is the job this series is for. Not “become an ML engineer.” Not “ban the tools.” Practical habits for people who already care about SQL, Python, quality, and metrics, and who now have a model sitting next to the warehouse.

This is Part 1 of Practical AI for analytics people. We start with the line that keeps you out of trouble: what AI can assist with, what it cannot replace, and who is still liable when a number hits a deck. If you need the vocabulary baseline first, read our plain-English tour of LLMs, ChatGPT, and generative AI. If you already ship SQL with model help, keep How to check AI-written SQL open as the companion checklist.

What you will learn

A clean split between assist, replace, risk, and refuse
Why fluency is not evidence, and why that matters for analysis
Where liability for numbers lives at each step of a real workflow
A worked example of a “helpful” answer that would mislead a board slide
Common mistakes and a 20-minute practice you can run this week

Assist is real. Replace is a category error

Large language models are good at language-shaped work: drafting, rewriting, outlining steps, suggesting join ideas, turning a vague question into a clearer one, and proposing code you can test. They are bad at being the system of record for truth. They do not “know” your warehouse. They predict plausible text given what you pasted and what they were trained on.

That sounds pedantic until you watch a model invent a column name that almost exists, invent a filter that sounds policy-like, or invent a growth percentage with two decimal places of confidence. The prose is smooth. The liability is still yours.

Think of AI for analysis the way you already think of autocomplete for SQL. Autocomplete speeds typing. It does not sign the query. Your job is still grain, definitions, and “does this answer the question we asked?” The SQL series and the Python series teach craft for that reason. AI does not erase those series. It multiplies the speed at which you can make a confident-looking mistake.

Four boxes: assist, replace, risk, refuse

When someone asks “should we use AI for this analysis?”, answer with a box, not a vibe.

Three columns: assist replace risk and refuse for AI in analysis

The diagram is a decision map. In words:

Assist: draft SQL or pandas, rephrase a question, outline an analysis plan, suggest checks, write a first-pass narrative after you have numbers, turn a metric spec into a checklist.
Replace (almost never for numbers): produce the official KPI without a query or certified extract, decide policy exclusions, invent missing facts about your business, own the “board number.”
Risk: anything where a wrong fluent answer creates a decision, a public claim, or a customer-facing statement. Risk rises with audience size, money, and irreversibility.
Refuse: pasting secrets or regulated fields into tools that are not approved, asking for “the real revenue” when you have no data access, using model memory as if it were a warehouse, accepting invented columns or citations.

Assist is wide. Replace for numbers is narrow to empty. Risk is the middle lane where you still use the tool but you force verification. Refuse is not Luddism. It is how adults treat untrusted input in a pipeline.

Why models sound right when they are wrong

Analysis has a special failure mode. In casual writing, a wrong adjective is annoying. In analysis, a wrong filter is a different company. Models optimize for continuity and helpfulness. They will complete the story even when the story needs a missing join key.

Three patterns show up constantly:

Schema hallucination: columns and tables that “should” exist. Your warehouse has order_total_usd. The model writes revenue and keeps going.
Definition drift: “active customer” becomes whatever sounds reasonable in the prompt, not what Finance signed in the metric spec. Pair this with the discipline in our metrics series.
Coverage theater: the answer lists caveats in one sentence, then delivers a precise number as if the caveats were optional decoration.

If you have already fought AI SQL, you know the fix is not “prompt harder until it feels true.” The fix is verify against schema, sample rows, and known totals. That is the spirit of the AI SQL check tutorial, applied to the whole analysis path.

Liability: you still own the number

Organizations love to ask who is “responsible for AI.” For analysis, the useful question is smaller: who is accountable for the claim that leaves the room? Vendor terms, model cards, and internal AI policies matter. They do not move the board number off your desk if you put it on a slide.

Borrow language from risk management without the theater. NIST’s AI Risk Management Framework talks about mapping, measuring, and managing risk across the AI lifecycle. For an analyst, the lifecycle of a number is: question, data path, transform, check, narrative, decision. AI can sit in several of those steps. Liability for the decision-facing claim still lands on a human role: often the analyst who shipped it, the owner who signed the definition, or both.

Stewardship habits help here. If nobody owns the definition, AI will invent one. If quality checks are optional, AI will skip them. If the pipeline path is a mystery, AI will narrate a confident path that never ran. See the data stewardship series, the data quality series, and the data pipelines series when you need the non-AI half of the trust stack.

Table of analysis steps showing where AI helps and humans must own

Use that map as a default operating model:

Step	AI can help	Human must own
Frame the question	Clarify wording, list missing inputs	Decision and success criteria
Choose data	Suggest tables from a pasted schema	Approved source, grain, access
Transform	Draft SQL or Python	Run, read plan, compare samples
Aggregate	Propose groupings	Metric definition and filters
Check	Suggest tests and edge cases	Pass or fail against knowns
Narrate	Draft prose from your table	Claims, caveats, audience fit
Decide	List options and tradeoffs	The call and residual risk

If your team only remembers one row, make it Check and Narrate. That is where fluent lies become institutional truth.

Worked example: the helpful 18%

Imagine a product lead pastes this prompt into a general chat tool:

We sold 12,400 units last quarter and 10,500 the quarter before.
Marketing spend was roughly flat. What was our growth, and what
should I tell the board about drivers?

A model might answer something like:

Growth = (12400 - 10500) / 10500 ≈ 18.1%.
Drivers: strong product-market fit and efficient marketing.
Recommend highlighting momentum and reinvesting in acquisition.

The arithmetic on the two integers is fine. The analysis is not. Missing pieces an analyst would force into the open:

Are units shippable product, trials, or seats? Same grain both quarters?
Refunds, cancellations, or restocks after the quarter closed?
Price changes? Mix shift toward a cheap SKU?
Seasonality or one large customer that will not recur?
“Marketing spend flat” is not a driver analysis. It is a vibe.

A liability-aware version starts with the table you actually trust, not with a board sentence. Toy sketch:

Quarter	Units shipped	Net revenue (USD)	Top customer share
Q3	10,500	2,100,000	9%
Q4	12,400	2,170,000	22%

Unit growth is about 18%. Revenue growth is about 3%. One customer drove a lot of the unit spike. The board story is not “momentum and reinvest.” The board story is “unit volume up, mix and concentration risk, revenue barely moved.” AI can help you draft both narratives after you have the table. AI should not invent the concentration column from thin air.

Here is a safer assist pattern: paste a small result set (or a certified metric extract), ask for alternate framings and risks, then edit. Example prompt shape:

Here is a verified quarterly table (toy numbers). Do not invent
extra metrics. List three honest narratives and two risks a board
might miss. Flag any question the table cannot answer.

quarter,units_shipped,net_revenue_usd,top_customer_share
Q3,10500,2100000,0.09
Q4,12400,2170000,0.22

You still own the slide. The model is a second pair of eyes on framing, not a second system of record.

What “good assist” looks like on a normal Tuesday

Concrete assist jobs that earn their keep for analytics people:

Question sharpening: “What is missing before I can answer this?” is often more valuable than “Write the SQL.”
Draft then test: generate a query, run it, compare row counts to a known dashboard tile, fix, repeat. Never present draft-as-truth.
Explain after prove: once you have a correct result, ask for a plain-language explanation for a non-technical audience. Edit the explanation.
Checklist generation: turn your metric spec into a pre-publish checklist. Keep the checklist in your team doc, not only in chat history.
Rubber duck for joins and nulls: describe a weird null pattern and ask what you would investigate next. Treat answers as hypotheses.

Jobs that look like assist but are replace in disguise:

“What was our true NRR last year?” with no data attached.
“Write the SQL and the board paragraph” as one step with no verification gate.
“Fill in the competitive market size” from model memory for a pricing decision.
Pasting production dumps of customer emails into a consumer chat tool because “it is faster.”

That last one is a refuse. Privacy and vendor terms are not optional footnotes. If your company has an approved enterprise workspace, use it. If it does not, keep PII and secrets out of the paste box. Later parts of this series go deeper on safe paste habits. For now: treat unapproved tools like a public corridor.

How this fits what you already know

AI does not replace analytics craft. It sits on top of it.

If you cannot explain grain, you cannot validate a generated group by.
If you do not know the metric definition, the model will invent one.
If quality dimensions are fuzzy, you will accept pretty nonsense.
If pipeline ownership is unclear, nobody will know which number was “the AI one.”

Vector databases and retrieval show up when you want models to answer from your docs instead of inventing them. That is a real architecture pattern, not magic. When you are ready for that path, use our intro to vector databases, and keep expectations honest: retrieval reduces some hallucinations, it does not remove liability.

Browse the full learning map anytime on Learn.

Common mistakes

Fluent equals verified. Smooth prose is a presentation skill, not a test result.
Skipping schema. Asking for SQL without pasting or linking real table definitions is how you get invented columns.
One-shot publish. Generate, screenshot, ship. No run, no sample, no known-total check.
Definition shopping. Re-prompting until the growth rate matches the story you wanted.
Tool cosplay as process. Buying a “AI analytics” label without metric specs, owners, or quality checks.
Secret paste. Credentials, customer identifiers, or regulated fields in unapproved tools.
Blaming the model after the fact. Vendors did not put the number on your slide. Your process did.

Practice: 20 minutes this week

Pick one recurring analysis you touch (weekly revenue, funnel, inventory, support volume). Write three columns on a sticky note or doc: Assist / Risk / Refuse. Fill five bullets under Assist that you will allow yourself. Fill three under Risk that require a verification gate (query result, dashboard match, second person). Fill two under Refuse that are non-negotiable for your team.

Then take one old AI-assisted query or narrative. Re-check it against schema and a known total. Note what would have failed if you had shipped the first draft. Keep that failure as a teaching example, not a shame story.

Next, Part 2 covers tokens, context windows, and cost intuition: why giant pastes fail, how to think about what fits in the “suitcase,” and when chunking beats cramming. Part 3 turns that into prompt patterns for data work.

Quick recap

AI is strong at language-shaped assist work and weak as a system of record for numbers.
Use four boxes: assist, replace, risk, refuse. Replace for official numbers is almost never the right box.
Fluency is not evidence. Schema hallucination, definition drift, and coverage theater are normal failure modes.
Humans own decision-facing claims. Map each analysis step to what AI may draft and what a person must verify.
Prove with data first, narrate second. Attach real tables when you ask for stories.
Your existing craft (SQL, Python, quality, metrics, pipelines, stewardship) is the control system AI needs.

Sources

NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0): https://www.nist.gov/itl/ai-risk-management-framework
OpenAI, safety and usage policies (tool and data handling expectations): https://openai.com/policies/usage-policies
OpenAI, model documentation and prompting guides (capabilities and limits in product docs): https://platform.openai.com/docs
Anthropic, Claude documentation (prompting, limitations, and constitutional AI overview): https://platform.claude.com/docs
Bender, Gebru, McMillan-Major, and Shmitchell, “On the Dangers of Stochastic Parrots” (why fluent text is not grounded knowledge): https://dl.acm.org/doi/10.1145/3442188.3445922
Google, People + AI Guidebook (human-AI collaboration patterns): https://pair.withgoogle.com/guidebook