A small language model (SLM) and a large language model (LLM) work the same way. Both are trained to predict the next piece of text, and both answer prompts, summarize and write code. The difference is size. A small language model has far fewer parameters, so it needs less memory and compute and can run on a phone or a laptop. It also knows less, and that shows up in specific ways.
Same mechanism, fewer parameters
Parameters are the numbers a model learns during training. A large language model stores what it has learned about language and the world in those numbers, and more parameters give it more room.
There is no official cutoff between “small” and “large”. Microsoft’s documentation for Azure Kubernetes Service uses a rough rule: small language models generally have fewer than 10 billion parameters, and large language models have more than 10 billion. By that rule Microsoft’s own Phi-4-mini, at 3.8 billion parameters, is a small language model, while Phi-4, at 14 billion, sits on the large side. Microsoft’s Foundry Local documentation describes the small models it supports as ranging from under 1 billion to around 14 billion parameters, small enough to run on a single GPU or even a CPU. So the labels overlap, and a model one team calls small may be “large” in someone else’s chart.
The useful question is not where the line sits. It is what a model gives up when it has fewer parameters, and what it gets back.
What a small language model gives up
The biggest loss is stored knowledge. When Microsoft introduced Phi-3, it reported that the small model did worse on TriviaQA, a test of factual recall, and explained why: “the smaller model size results in less capacity to retain facts.” A small language model has less room for names, dates and long-tail facts, so it is more likely to guess, and guessing is how AI hallucinations happen.
Apple says the same about its own on-device model. Its roughly 3-billion-parameter foundation model is tuned for tasks such as summarizing, extracting information and refining text, and Apple states plainly: “It is not designed to be a chatbot for general world knowledge.”
Microsoft’s Azure overview of small language models also lists limits on complex language and on accuracy for harder tasks. In practice, a small language model handles a narrow, well-defined job well and a broad, open-ended question badly.
What a small language model gains
The gains are mostly about where and how cheaply the model can run.
- It fits on a device. Google’s Gemini Nano runs inside Android’s AICore system service, with no calls to a server. Google says AICore does not keep the inputs or the outputs. Apple’s on-device model runs on the iPhone, iPad and Mac itself.
- It responds without a network. A model that runs locally works offline and doesn’t wait on a round trip to a data center.
- It costs less to run. Fewer parameters mean less memory and less computing per answer, which is why a small language model can run on a single GPU or a CPU.
- It is easier to customize. Microsoft notes that small models like Phi-3 are easier to fine-tune for a specific task than large ones.
Keeping the work on the device also changes what leaves it. If the model runs locally, your prompt doesn’t have to travel to a company’s servers. Our guide to on-device vs. cloud processing covers what that does and doesn’t protect.
The line is getting blurrier
Model makers keep finding ways to get large-model behavior out of small-model hardware. In June 2026 Apple announced the third generation of its foundation models. AFM 3 Core is a 3-billion-parameter dense model. AFM 3 Core Advanced is a 20-billion-parameter sparse model that activates only 1 to 4 billion parameters for any given piece of text, and Apple keeps the full model in flash storage instead of memory.
That design means total size and running cost are no longer the same number. A sparse model can hold more knowledge than a 3-billion-parameter model while doing about as much work per answer. Labels like “small” and “large” describe less than they did a few years ago.
Phones also mix the two. Google describes hybrid inference for Android apps, where a request runs on Gemini Nano when the device supports it and goes to a cloud model when it doesn’t. Apple’s system works on a similar split between its on-device model and larger models on its servers.
Small language model vs. LLM at a glance
| Small language model | Large language model | |
|---|---|---|
| Typical size | Under about 10 billion parameters | Over about 10 billion parameters |
| Where it runs | Phone, laptop, single GPU or CPU | Data-center servers |
| Stored knowledge | Limited; weaker on facts | Broad |
| Best at | Narrow tasks: summarizing, extracting, rewriting | Open-ended questions, complex reasoning |
| Works offline | Often | Usually not |
| Cost to customize | Lower | Higher |
The size figures follow Microsoft’s rough rule, not a standard.
When each one makes sense
A small language model makes sense when the task is narrow and repeats a lot: sorting messages, pulling dates out of emails, rewriting a paragraph, suggesting a reply. It also makes sense when the data shouldn’t leave the device or when the app has to work without a connection. If you want to try one yourself, see running AI locally on your laptop.
A large language model makes sense when the question is open-ended or depends on facts the model has to know: research, long reasoning chains, broad coding help, or a chat about almost anything. That is the job Apple says its small model was not built for.
Many products use both. The phone handles the quick, private tasks with a small language model and sends the hard ones to a large model in the cloud.
