Outmake Beta Download Beta
Guides

Speculative decoding on a Mac, measured

Speculative decoding lets a model check several guessed words at once instead of writing them one by one. Whether it helps on a Mac depends on what the model is writing. Here is one measurement of each case, and the flag that does it.

The numbers

Measured on 21 September 2026 on a MacBook Pro with an M1 Pro and 16 GB of memory, running Qwen3 4B in llama.cpp.

What it wroteOffOn
An edit to a code file30.5 words a second (about 41 tokens)69.1 words a second (about 92 tokens)
A story29.8 words a second (about 40 tokens)30.7 words a second (about 41 tokens)

From one Mac. The size of the model file wasn’t written down with the run. Tokens are worked out from words at three words to four tokens.

Why the edit doubled and the story didn’t

This kind of speculative decoding guesses from what is already in the conversation. When a model edits a file, most of what it writes back is the file it was just given, so the guesses are right for long runs and the model confirms many words in one pass. A story repeats nothing, so there is nothing to guess from and the speed stays the same.

So it helps most with code edits, rewrites, and anything that quotes the page back. It doesn’t help with fresh prose, and in this test it didn’t slow it down.

The words are the same

A guess is kept only where it is the word the model would have written. The output is the model’s own either way. A model with a picture attached drafted the same way in the same test.

No second model

The older way to do this loads a small draft model beside the big one, which takes memory a 16 GB Mac doesn’t have to spare. Guessing from the conversation needs no second model and almost no memory.

In Outmake

Nothing to turn on. Every chat model Outmake runs on a Mac has it on. Two models on Outmake’s list, Qwen3.8 27B among them, carry a predictor of their own in the file, and Outmake uses that first.

In llama.cpp yourself

The same setting is one flag on llama-server (checked against build b11718, the one Outmake 0.5 ships):

llama-server -m your-model.gguf --spec-type ngram-mod

Outmake also runs with flash attention and an 8-bit cache for the conversation, which halve the memory a long chat takes.

Models that run on your Mac · How Outmake measures speed