A laptop bought in the last few years can run a language model with no internet connection, no account and no charge per question. It cannot run the model you are probably picturing, and most disappointment with local models comes from not knowing the difference.
This guide covers why you might want one, what decides whether it fits, and how to choose a size for the job rather than the biggest file that will load.
Why run one locally at all
- Privacy. The text never leaves the machine. For client documents, ledgers or anything under a confidentiality agreement, that settles questions no hosted service's terms quite do. My years in audit left me with a firm view on where a client's ledger should travel: nowhere it does not need to.
- Cost. Once you own the hardware, a question costs a little electricity. Running a model over every row of a spreadsheet becomes an experiment rather than a budget decision.
- Offline. It works on a train, in an office with locked-down Wi-Fi, and when a provider is having a bad day.
Memory is the wall
One number decides what you can run: memory. Graphics memory if your laptop has a dedicated GPU, ordinary RAM if it does not, and the shared pool on machines with unified memory, such as Macs with Apple silicon.
The arithmetic is simple. A model is a very long list of numbers called parameters, and all of them sit in memory while it runs. At 16-bit precision each one takes two bytes, so a model with 7 billion parameters needs around 14 GB for its weights alone, before the operating system and your conversation get a share. Most laptops do not have that to spare.
Rough rule: billions of parameters times bytes per parameter gives gigabytes. Then leave generous headroom.
Quantisation, without the maths
Quantisation stores each of those numbers with fewer bits: 8 instead of 16, or 4. At 4 bits, the same 7 billion parameter model needs roughly 4 GB, which an ordinary 16 GB laptop can hold with room to spare.
The price is precision. Think of it as rounding a ledger. Round every entry to the nearest rupee and the totals barely move. Round to the nearest thousand and the totals turn into fiction. 8-bit stays close to the original. 4-bit is the usual compromise: slightly worse, mostly in careful reasoning and exact recall. Below 4 bits quality falls away faster, and a smaller model at 4 bits is often a better choice than a bigger one squeezed to 2 or 3.
Quantised models are commonly shared as GGUF files with the bit width in the name, such as Q8_0 or Q4_K_M. If memory is tight, start with a Q4 file.
The tools and the families
llama.cpp is the engine under much of this. It runs quantised models on an ordinary CPU and uses a GPU when there is one. Ollama builds on it with a simple installer, one command to download and run a model, and a local HTTP API for when another program needs to call it. LM Studio offers a desktop window with a model browser and chat, for anyone who would rather avoid a terminal.
The open model families worth knowing are Llama from Meta, Mistral, Gemma from Google, Phi from Microsoft and Qwen from Alibaba. Each comes in several sizes, and new versions arrive so often that any specific recommendation goes stale within months. Read the licence before using one in client work, because "open" covers a range of terms.
Choose the size for the job
Bigger models know more and reason better over several steps. Smaller ones are faster, and for narrow jobs often good enough. Roughly:
- 1 to 4 billion parameters. Sorting text into categories, pulling fields out of a short passage, rewriting into a fixed format. Quick, but weak on facts.
- 7 to 9 billion. The usual laptop choice at 4-bit. Summaries, first drafts, answers from a document you paste in.
- 13 billion and up. Needs more memory with every step, 32 GB or more for the larger ones. Slower, and noticeably better at complicated instructions. The 70 billion class belongs on a workstation or a server.
Start with the smallest model that might do the job, test it on real examples, and move up a size only when it fails. A small model fine-tuned for one narrow task (LoRA has made that cheap to try) can beat a much larger general one at that task.
Two more limits catch people out. Speed: if answers arrive more slowly than you can read, you will stop using the model, however clever it is. And context: the conversation is held in memory too, so a model that loads comfortably can still run out of room when you paste in a long contract.
Keep expectations honest
A 7 billion parameter model on a laptop is not a smaller copy of the best hosted assistants. It knows less, invents more readily and loses the thread in long instructions. It has a training cut-off and knows nothing after it. It will state a wrong figure in the same calm tone as a right one.
What it does well is work where you supply the facts and it supplies the shaping: summarise this, extract these fields, rewrite this in plain English. Give it the material, ask for one narrow thing, check the result. Used that way, a local model is useful, private and free per question. Asked to be an oracle, it will disappoint you.
An evening's experiment
To see for yourself, install one of the tools above, download a 7 billion parameter model at 4-bit, and give it a job you have already done by hand, such as a summary you wrote. Compare. An evening of that tells you more about your own laptop than any leaderboard, and costs nothing but the download.
