Model size, quantization, and will my laptop run it?

Rafi sat on the kitchen stool with a 16 GB M2 Air and a coffee that went cold. A Slack thread said 70B was the size that “felt like GPT,” so he typed ollama run llama3.1:70b. Ollama’s library page, as of writing (August 2026), lists that tag at 43 GB, quantization Q4_K_M, 70.6B parameters. The SSD chewed through the pull. Activity Monitor later showed compressed swap past 39 GB. The fan was a leaf blower aimed at a notebook. The first token arrived after 4 minutes. Chrome, Slack, and a Notion tab were still open. He closed the lid with the cursor still blinking.

This is OS3, Part 3 of Open-source AI explained. OS2 split hosted open weights from a file on your disk. This page asks whether that file will run on the machine in front of you. Next: OS4, privacy reasons normal people care. Hands-on setup lives later in Run open models from scratch. A thank-you email still belongs in the chooser.

What a 7B actually needs

• Why 7B, 13B, and 70B are clothing sizes, not a scoreboard
• What GGUF quantization (Q4, Q5, Q8 as of writing) does to the file
• Why RAM, not disk, is the limiter, including swap and the fan
• A sticky-note pick: 8 to 16 GB stays near 7B to 8B Q4, 32 GB can try 13B to 14B, 70B wants a lot more RAM or a host
• How Rafi’s 43 GB pull and 4-minute first token happened on 16 GB

Billion is a clothing size

The B is billion parameters. A parameter is a weight in the network, a number the model learned. 7B means about seven billion of those numbers. 70B means about ten times as many. Treat the label like a shirt size. It tells you how much cloth you bought. Slack does not award points for XL.

Families reuse the same name at several sizes. Meta’s Llama 3.1, as of writing, ships 8B, 70B, and 405B. Qwen 2.5 ships 0.5B through 72B. The nearby pull today is often 14B (Ollama lists qwen2.5:14b at 9.0 GB, Q4_K_M). Hedge the tag. Open the library page the week you type ollama run.

Bigger can help with hard reasoning and long instructions. Smaller is faster, cooler, and the one that fits. A sharp 8B on your laptop beats a 70B that spends four minutes paging to the SSD. Pick a size your RAM can hold, then judge the prose on a prompt of yours. Quality without theater is OS5.

Mixture-of-experts cards complicate the label: a 30B or 120B name can ship a smaller file because only a slice of experts fire per token. Read the card. For a first local stack, stay dense and boring: 7B to 8B Q4.

Rule of thumb: The B is how much model you asked for. RAM is whether the machine can lift it. Disk is only whether the file landed.

Quantization is a smaller file that still talks

Full-precision weights are fat. llama.cpp’s quantize notes (as of writing) put Llama 3.1 8B original size near 32.1 GB and Q4_K_M near 4.9 GB. Same family, 70B: original near 280.9 GB, Q4_K_M near 43.1 GB. That is why Ollama lists llama3.1:8b at 4.9 GB and llama3.1:70b at 43 GB. You downloaded a packed version, not the textbook.

The pack format is GGUF. Hugging Face’s Hub docs describe it as tensors plus metadata, built for GGML-style engines. llama.cpp is the engine under a lot of desktop apps. Ollama and LM Studio speak that ecosystem. You do not have to compile llama.cpp. You do have to read the quant name on the file.

The names look like serial numbers. As of writing, the three you will meet first:

• Q4_K_M (and other Q4 flavors). Roughly 4 to 5 bits per weight. Ollama’s default Llama 3.1 tags use this. The “usually still talks” pull.
• Q5_K_M (and Q5 friends). A bit more memory, closer to the original numbers, when Q4 feels mushy and you have RAM.
• Q8_0. About 8 bits. Hugging Face still documents it. Nearer the original bits, roughly double a Q4 file. Fine on a 7B. Painful on a 70B.

There are more letters (Q4_0, _S vs _M, I-quants such as IQ4_XS). You do not need the whole menu on day one. If the Hub file and the Ollama tag both say Q4_K_M, you are in the main lane. Re-check the blob. Tags get retagged.

Quantization is a lossy zip for weights. The model usually still talks. It can get vaguer, worse at tiny facts, or sloppier at a fiddly format. It does not turn a 7B into a 70B. “Just Q8 it” spends RAM. On a 16 GB Air, that spend is how you meet the leaf blower. Pull a tagged Ollama file or a Hub GGUF that already lists the quant. llama-quantize is a later craft.

RAM is the limiter, not the download bar

Disk is cheap enough that a 43 GB file can finish. RAM is the room the weights need to sit in while they run. If the file is larger than the memory the OS will give you, the machine pages. On a Mac that looks like compressed memory, then swap on the SSD. The fan climbs. Tokens crawl. The first token is the worst because the runner is still hauling layers in.

Apple Silicon makes this easy to miss. An M2 Air has unified memory: CPU, GPU, and display share the same 16 GB. About This Mac is the number that matters, not “I have 256 GB of SSD.” A Windows laptop with 16 GB of RAM and a 4 GB GPU is the same shape. An 8 GB GPU can hold a 7B Q4. It will not hold a 43 GB 70B.

The file size is the floor, not the bill. You still owe the OS, the browser, and a KV cache (the conversation’s working memory). Long context spends more of that cache. A 4.9 GB 8B feels tight on 8 GB with 40 Chrome tabs and relaxed on 16 GB if you quit the junk. The pick below says “around” and “try,” not a chassis guarantee.

A model that fits in RAM (or VRAM) can stream. A model that spills to swap can still finish a sentence after you open mail in the gap. Rafi’s 4-minute first token was swap, not a Llama bug. The 70B file was honest. The 16 GB chassis was the wrong closet.

Write the pick on a sticky note

Look up RAM first. On a Mac: Apple menu, About This Mac. On Linux: free -h. Write the number, then pick a size that leaves leftover for the rest of the computer. As of writing, a starting pick.

If you have 8 to 16 GB, stay around 7B to 8B at Q4. If you have 32 GB, try 13B to 14B. 70B wants a lot more RAM or a host. Tape that to the bezel. The 405B tag (Ollama lists llama3.1:405b at 243 GB as of writing) is a joke on a laptop.

A 32 GB machine still cannot keep a 43 GB 70B Q4 resident. Ugly 2-bit 70B files on 32 GB are a hobby, not the Monday pick. If you need 70B-class answers on 16 GB, OS2 already named the door: a host. Groq is the inference company with a q, not xAI’s Grok. Together and Fireworks rent GPUs too. The prompt leaves. Trade that on purpose.

Worked example: 43 GB, 4 minutes, one Air

Rafi’s Air had 16 GB unified and maybe 180 GB free on disk. The pull succeeded. Disk was not the veto. macOS compressed, then swapped. First token at 4 minutes. Later tokens still took seconds. He asked for a rewrite of a public blog paragraph. llama3.1:8b would have finished while the 70B was still inhaling.

A check you can run before the 43 GB mistake. Flags and tag names move. Read the help text the week you try this.

# RAM pick, then a toy pull.
# As of writing (August 2026), Ollama library lists:
#   llama3.1:8b    4.9 GB   Q4_K_M
#   qwen2.5:7b     4.7 GB   Q4_K_M
#   qwen2.5:14b    9.0 GB   Q4_K_M
#   llama3.1:70b    43 GB   Q4_K_M

# macOS bytes of RAM:
sysctl hw.memsize
# Linux:
# free -h

# 8 to 16 GB RAM: stay here.
ollama run llama3.1:8b

# Toy prompt only. Time the first token with a watch.
# Reply in one sentence: what is 17 times 4?

# 32 GB RAM: you may try this after the 8B feels thin.
# ollama run qwen2.5:14b

# Do not run this on 16 GB:
# ollama run llama3.1:70b

What that check is for: the RAM number hits the screen before the 70B badge does. If the 8B’s first token is already slow, quit the 40 tabs and retry. Heat is not a request for 70B. If the 8B is fine, keep it for a week.

When a host is the honest 70B

If the job needs a large open model and the chassis is a 16 GB Air, pay a host and say so. You already met that location split in OS2. The size math: a Q4 70B wants tens of gigabytes of fast memory. Buy a Mac Studio class box, a used GPU pair, or someone else’s rack. You cannot will 43 GB into an Air with a bigger SSD.

Closed chat (ChatGPT, Claude, Gemini, Grok the product) may still be easier for writing. That is OS7, and the chooser. Do not build a local 70B so a thank-you sounds like a press release. Do not paste an NDA into a host because the local 8B felt “less smart.” Location is still the noun. Size is the fit.

Stay on the laptop when the work is offline. Then the sticky-note pick is the strategy. A 7B Q4 that answers in two seconds with Wi-Fi off beats a 70B that melts the case. Privacy that holds is OS4, plus the paste test in P4.

RAM is not the same as it ran

• Treating 70B as a badge. It is a clothing size. Slack does not award points.
• Reading disk free space as “it will run.” RAM runs it. Disk only stores it.
• Pulling llama3.1:70b on 16 GB because the command is one line.
• Q8 on a 70B “for quality” when you cannot hold Q4.
• Judging an 8B in swap and calling local “unusable.” Quit Chrome and retry.
• Skipping the model card’s quant line on Hugging Face. GGUF filenames are the spec.
• Mixing Groq the host with Grok the chatbot when you bounce off local 70B.

Pick one size you can load

Write your RAM on a sticky. Pull one 7B or 8B Q4 tag. Time a toy prompt. If you have 32 GB, try the 14B on the same prompt and keep the faster one. Do not pull 70B on 16 GB. Next: OS4, privacy reasons normal people care. Location from OS2 still applies. Closed-box setup: Learn.

Quantization in one page

• 7B, 13B, 70B are size labels. Write RAM first.
• Q4 / Q5 / Q8 (GGUF, llama.cpp, Ollama tags) shrink the file. The model usually still talks.
• 8 to 16 GB: stay near 7B to 8B Q4. 32 GB: try 13B to 14B. 70B: lots of RAM or a host.
• A 43 GB llama3.1:70b on a 16 GB Air is how you get a 4-minute first token and a leaf blower.

Sources

• Ollama and the Llama 3.1 library page (as of writing: llama3.1:8b 4.9 GB, llama3.1:70b 43 GB, both Q4_K_M)
• Qwen 2.5 on Ollama (qwen2.5:7b 4.7 GB, qwen2.5:14b 9.0 GB, Q4_K_M)
• llama.cpp and its quantize notes (original vs Q4_K_M sizes for Llama 3.1 8B / 70B / 405B)
• Hugging Face Hub: GGUF (format, quant type table, Q4_K / Q5_K / Q8_0 and newer IQ types)
• LM Studio (desktop runner that also loads GGUF)
• Groq (hosted inference; not xAI Grok) if 70B will not fit
• AMS OS2: hosted vs download and OS1: what open means

Merch on Etsy

Open store →

You're On Mute White Ceramic Coffee Mug, Data Science Mug, Data Mug, Analytics Mug, Statistics Mug, Programming Mug, Gift Mug, Coffee Mug View on Etsy → You're On Mute White Ceramic Coffee Mug, Data Science Mug, Data Mug, Analytics Mug, Statistics Mug, Programming Mug, Gift Mug, Coffee Mug View on Etsy → Y Hat Mouse Pad, Data Science Mouse Pad, Analytics Mouse Pad, Y Pred Mouse Pad View on Etsy → Boxplot Mouse Pad, Data Science Mouse Pad, Analytics Mouse Pad, Box Plot, Whisker Plot View on Etsy →