Tokens, context windows, and cost (intuition)

You paste the whole schema. Then last quarter’s CSV. Then three Slack threads “for context.” Then you ask for a careful answer. The model either truncates quietly, forgets the middle, invents a column you swear you included, or burns a surprising amount of money on a question that needed two tables and a one-line filter.

This is not you being bad at AI. This is you treating a limited working memory like a warehouse. Analysts already know that RAM and disk are different. Tokens and context windows are the same idea with friendlier branding.

This is Part 2 of Practical AI for analytics people. Part 1 covered assist versus replace and liability for numbers. Here we build intuition for tokens, context windows, and cost so your prompts fail less and your bills stay boring. If LLM vocabulary still feels fuzzy, skim What are LLMs, ChatGPT, generative AI, and more first. When you generate SQL, keep How to check AI-written SQL next to this post.

What you will learn

What a token is in plain English (and why “words” is the wrong unit)
How a context window works like a suitcase with a reply budget
Why long pastes fail even when the UI accepts them
Relative cost intuition for questions, schemas, CSVs, and chat history
Chunking patterns that match how analysts already break work
Mistakes and a practice drill you can run on a real prompt this week

Tokens: not words, not characters

Models do not “read” your prompt as a sequence of English words. They break text into tokens: common chunks of characters that the model was trained to handle. A short English word is often one token. A rare name, a long identifier, or dense code can split into several. Numbers and punctuation get their own pieces. Exact splits differ by model family. You do not need to memorize tokenizer tables. You need the operational idea: length in the UI is not length the model bills or fits.

Rough intuition many teams use for English prose: a token is often around three-quarters of a word, so 100 words is on the order of 130 tokens. Code, JSON, and CSV can be denser. A 50-column header line can cost more than a chatty paragraph about the same table. That is why pasting “just a little data” sometimes costs more than the question itself.

Vendor docs publish pricing per million tokens for input and output, and they publish context size limits per model. Those numbers change. Your habit should not depend on memorizing this month’s price list. Your habit should be: measure roughly, keep the suitcase intentional, and prefer small verified artifacts over dumping the lake into chat.

The context suitcase

A context window is the maximum amount of tokenized text the model can consider in one go for that conversation turn (prompt plus, in many products, prior messages, plus the room reserved for the reply). Think suitcase, not infinite cloud drive.

Context window as a suitcase filled with prompt schema history and reply budget

What competes for space:

System and product instructions you never see but still occupy room.
Your prompt (the question and constraints).
Schema and docs you pasted or the product retrieved.
Sample data or full extracts.
Chat history in multi-turn sessions.
Reply budget so the model can actually answer.

If you fill the suitcase with history and a giant CSV, the model has less room for a careful answer. Some products truncate older messages. Some summarize silently. Some error. Some keep accepting paste while quality falls off a cliff. From your side it feels like “the model got dumber.” Often the suitcase got messy.

Why long pastes fail

Analysts love completeness. Models punish unprioritized completeness. Long pastes fail in a few repeatable ways:

1. Hard limit hits

You exceed the window. The tool rejects the message, strips the end, or drops early history. The column definition you needed was in the part that fell off.

2. Soft attention failure

Everything fits, but important details in the middle get ignored. Research and practice both note that models can underuse the middle of long contexts compared with the beginning and end. Your critical grain note sits between two walls of CREATE TABLE noise. The model latches onto the last table it saw.

3. Noise becomes “fact”

You paste three conflicting metric definitions from Slack. The model averages them into a confident hybrid. Completeness without curation is not honesty. It is a blender.

4. Cost without lift

You pay for every token of the 40 unused tables in the schema dump. The answer would have been better with two tables and one sample query.

This is the same discipline you use when writing SQL: do not SELECT * from the whole warehouse into your head. Project the columns you need. The SQL series and the Python series already train that muscle. Context is just another place to project.

Relative cost intuition (not a price sheet)

Exact prices change by vendor and model. Relative shapes stay useful for planning. Treat the table as a teaching scale, not a quote:

Relative token cost table for questions schemas CSVs and chat history

Artifact	Relative size	What it buys you	When it is worth it
Clear question (1-3 sentences)	Tiny	Goal and success criteria	Always
Constraints and refuse rules	Small	Fewer invented columns	Almost always for data work
Two relevant tables + keys	Medium	Grounded SQL draft	Default for query help
Full warehouse schema dump	Large	Rarely better than a focused subset	Only if you will filter hard
20-row sample with headers	Medium	Grain and dirty values	High value for cleanup and joins
Full CSV extract (thousands of rows)	Huge	Often noise; privacy risk	Rarely; prefer aggregates or samples
Long chat history	Grows every turn	Continuity, also confusion	Reset when the thread drifts
Long generated answer	Output cost	Prose or multi-query dumps	Ask for structure; cap scope

Input tokens and output tokens are both billed in API settings. Chat products hide the meter, but the economics still shape product limits and quality. A habit that saves money also often saves quality: smaller, sharper context.

Worked example: one question, three suitcase packs

Business question: “Revenue by region for last month, excluding test accounts, matching the Finance definition.”

Pack A: the dump (usually bad)

Here is our entire schema export (200 tables)...
Here is orders_raw for 18 months (CSV)...
Here are Slack notes about revenue...
Write SQL for revenue by region last month.

Failure modes: wrong table chosen, test accounts defined from a random Slack message, region taken from shipping address instead of billing region, huge cost, and a confident answer you cannot audit.

Pack B: focused schema (better)

Task: draft SQL for net revenue by region for last calendar month.
Use ONLY these objects. If something is missing, say what you need.
Do not invent columns.

Table fct_orders(
  order_id, order_ts_utc, customer_id, region_code,
  gross_amount_usd, discount_usd, is_test
)
Table dim_region(region_code, region_name)

Finance rule: net = gross_amount_usd - discount_usd
Exclude is_test = true
Region from fct_orders.region_code
Return region_name, net_revenue_usd
Order by net_revenue_usd desc

This pack is mostly medium-small. It spends tokens on the contract, not on archaeology.

Pack C: focused + tiny sample (often best)

Keep Pack B, then add ten sample rows that show a test account and a discount:

order_id,order_ts_utc,customer_id,region_code,gross_amount_usd,discount_usd,is_test
1001,2026-01-05T12:00:00Z,c9,US-E,100.00,10.00,false
1002,2026-01-06T09:00:00Z,test_bot,US-W,50.00,0.00,true
1003,2025-12-28T18:00:00Z,c2,EU,80.00,0.00,false

The sample teaches grain and traps. You do not need 50,000 rows for that lesson. After the model drafts SQL, you run it in your warehouse or notebook and verify. That is still your liability from Part 1.

Illustrative output shape you would expect after a correct run (toy):

region_name	net_revenue_usd
US East	420150.25
US West	301992.10
EU	188440.00

If the draft SQL forgot is_test, your check against a known dashboard tile should catch it. Context design makes the draft better. Checks make the answer real.

Chunking: how analysts already think

Chunking means splitting work so each model call has a suitcase that matches one job. You already chunk pipelines and notebooks. Apply the same cuts.

Chunk by task

Call 1: clarify the question and list missing inputs.
Call 2: draft SQL for one metric with one schema slice.
Call 3: after you paste real results, draft narrative and risks.

Do not ask one prompt to discover schema, write five queries, plot strategy, and write the board memo.

Chunk by object

If you must document twenty tables, process them in groups of two or three with a stable template. Merge outputs yourself. Models are better at consistent micro-tasks than at “summarize the enterprise.”

Chunk by retrieval, not by dump

When the corpus is large (wiki, metric catalog, runbooks), the long-term pattern is retrieve relevant chunks, then generate. That is the family of ideas behind RAG and vector search. You do not need a production RAG stack to benefit from the mindset: search first, paste second. For architecture intuition, see Getting up to speed on vector databases. Later parts of this series return to retrieval. For now, manual retrieval (you pick the two tables) is already a win.

Reset the thread

When history is polluted with wrong assumptions, start a new chat with a clean pack. Continuity is not free. It costs tokens and sometimes steers the model toward yesterday’s mistake.

Practical packing rules

Put the task and constraints at the top and bottom. Important rules should not live only in the middle of a schema wall.
Prefer certified definitions over Slack archaeology. Metrics series habits still apply: one written definition beats three vibes.
Samples over extracts. Ten honest rows beat ten thousand opaque ones for drafting. Use SQL or Python for the heavy lift.
Name the grain in the prompt. “One row per order” prevents many bad joins before they start.
Budget the reply. “Return only SQL” or “return a 5-row plan” reduces rambling output cost and skimming pain.
Watch privacy. Token cost is not the only cost. PII in context is a risk cost. Prefer synthetic or redacted samples.

Quality and stewardship still sit under all of this. A tiny suitcase packed with the wrong definition is still wrong. Pair packing discipline with the data quality, metrics, and stewardship series when the definition itself is the hard part. Pipeline path questions belong with data pipelines. More learning paths live on Learn.

Common mistakes

Schema maximalism: pasting every table “just in case.”
CSV as context: uploading a full extract when a profile or sample would teach the model more.
Infinite threads: a week-old chat that still thinks refunds are included.
Hidden system bloat: stacking plugins, retrieval, and huge custom instructions until user content is squeezed.
Assuming the model “remembers the warehouse.” It only sees what is in this suitcase (plus training priors that are not your data).
Optimizing only for prompt length, never for structure. A short vague prompt is worse than a medium precise one.
Ignoring output scope. “Write a full report” when you needed a filter list.

Practice: repack one real prompt

Take a recent AI chat that went sideways. Copy the final prompt pack into a doc. Highlight four colors: question, constraints, schema or data, history leftovers. Delete everything that did not serve the question. Keep at most two tables (or one file profile). Add a one-line refuse rule: “If a column is not listed, do not invent it.”

Rerun the same question with the repacked suitcase. Compare: did the draft need fewer fixes? Did you notice lower latency or cost in API usage? Even in a chat UI, note subjective quality. Save the before and after packs as a team example.

Next, Part 3 turns packing into prompt patterns for data work: role, spec, output shape, and refuse rules that keep SQL and explanations honest.

Quick recap

Tokens are model chunks, not English words. Code and CSV can be surprisingly expensive.
Context is a suitcase: prompt, schema, samples, history, and reply budget compete for space.
Long pastes fail via hard limits, middle neglect, mixed “facts,” and cost without quality.
Spend tokens on task, constraints, relevant schema, and tiny samples. Not on the whole lake.
Chunk by task and object. Retrieve then paste. Reset dirty threads.
Smaller, sharper context is usually cheaper and more accurate. Verification still sits with you.

Sources

OpenAI, tokenizer and tokens guide: https://platform.openai.com/docs/concepts/tokens
OpenAI, pricing overview (input/output token billing patterns): https://openai.com/api/pricing/
Anthropic, context windows and prompting documentation: https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/overview
Liu et al., “Lost in the Middle: How Language Models Use Long Contexts” (long-context position effects): https://arxiv.org/abs/2307.03172
Google, Gemini API long-context and token documentation (product docs evolve; use current limits): https://ai.google.dev/gemini-api/docs/long-context
NIST AI RMF (risk thinking still applies when context includes sensitive data): https://www.nist.gov/itl/ai-risk-management-framework