A Model That Fits in Three Gigabytes Is Not the Best One. It Is the One That Never Leaves.

Download a program, download a file, and a working AI assistant runs on a five-year-old laptop with no account, no subscription and no internet connection. That has been technically possible for a while. What changed this year is that the setup takes about ten minutes and the small models got genuinely usable.

The reason to care is not cost. It is that some documents should not be uploaded anywhere, and until now the only options were upload them or do the work by hand.

Look Behind the Curtain

Two things made this practical. The first is quantization, which stores a model's numbers at lower precision. A four-bit version of a model needs roughly six tenths of a gigabyte per billion parameters, so the smallest current models fit in about three gigabytes of memory and mid-sized ones in six or seven. Google's Gemma 4, released in April 2026, publishes its own figures: the smallest variant runs in 2.9 gigabytes at four-bit, which fits inside an ordinary eight-gigabyte laptop with room to spare.

The second is licensing. Gemma 4 moved to a standard Apache licence, the first genuinely permissive release in that family. OpenAI's gpt-oss models arrived in August 2025 under the same licence, with the twenty-billion version designed to run in sixteen gigabytes. Qwen, Mistral and Microsoft's Phi models are similarly open. Meta's Llama is downloadable but under a custom licence that is not, strictly speaking, open source, which matters if you are putting it in a business.

The software is free and mostly open. Ollama and llama.cpp are the plumbing, both MIT licensed. Jan and AnythingLLM are desktop applications that wrap them. LM Studio is the friendliest of the group and free for commercial use since mid-2025, though the application itself is closed source.

Meanwhile the operating systems already do a version of this. Apple's on-device foundation model is about three billion parameters and hands anything harder to its Private Cloud Compute servers. Android's on-device system routes through a service with no direct internet access that retains nothing after processing. In each case some of the work is genuinely local and some is not, and the split is not always visible to you.

The honest part: a small local model is not close to a frontier hosted one. On one widely used composite benchmark, the current top hosted models score in the high fifties out of a hundred while a twenty-billion model you can run at home scores around nine. That gap is real and it shows up on anything requiring multi-step reasoning or broad factual knowledge.

The Capability Multiplier

Which makes the question not "is it as good" but "what is it good enough for, and what is that worth."

The answer is a narrower list than the marketing suggests and a more valuable one than the benchmark implies. Summarizing a long document, drafting from notes, reformatting messy text, extracting fields from a stack of files, translating, answering questions about a folder of your own material: small models handle all of it acceptably. Those also happen to be the tasks most likely to involve something you would rather not upload.

The category that justifies the whole exercise is documents with other people's information in them. Client files, medical paperwork, employee records, a folder of contracts, anything under a confidentiality obligation. The calculation changes completely when nothing leaves the machine, because there is no vendor policy to read, no retention period to worry about and no breach to be notified about later.

There is a working pattern worth knowing. Use the local model for anything sensitive, and a hosted model for hard thinking on anything that is not. Most people default to one or the other and end up either uploading things they should not or struggling with a small model on a task it was never going to do.

The Sovereign Action

An evening, and then it is just there.

- Install LM Studio or Ollama, download one small model to start, and keep it around four gigabytes so it runs comfortably.

- Give it the jobs it is actually good at: summarizing, reformatting, extracting, drafting from material you supply.

- Reserve it specifically for documents you would not upload, and decide that boundary explicitly rather than case by case.

- Keep a hosted assistant for research and hard reasoning. This is a second tool, not a replacement.

- Check what your phone and laptop already do locally, and what they send onward. The answer differs by device and is not always what the marketing implies.