Mafdet

The problem

A chat interface waits for Enter before doing anything. While you type, the GPU sits idle. When you send, the model first has to read the whole message (the prefill), and a reasoning model such as Qwen3 in thinking mode then writes hundreds of tokens of reasoning before the first word of the answer. On a mid-range GPU that reasoning is the longest part of the wait: ten seconds and more on a word problem, minutes on a model that does not fit in VRAM.

Mafdet uses the time you spend typing for both steps.

Reading ahead

The prefill stores intermediate results for every token in the KV cache, and those results never change for a given prefix. Mafdet keeps the cache in sync with the text while you type:

1. On every pause it tokenizes the current text.
2. It holds back the last few tokens, because a word still being typed can change how the text is split.
3. It finds the longest common prefix with the tokens already in cache, trims the cache there and prefills only what is new.
4. On Enter only a handful of tokens are left to process.

Correctness does not depend on how you type. Edits in the middle, pasted blocks and backspaces always bring the cache back to exactly what a from-scratch computation would produce. The heuristics only change how much work is wasted, never the result.

Thinking ahead

The model also reasons while you type, in short chunks, on the message as it is at that moment. When the text changes, the reasoning is kept as notes after the new version of the message, which costs only a prefill, and the model goes on from there. On Enter it finishes from where it got to.

This part is not exact, because the model reasons on unfinished versions of the message, so it was measured on answers that can be checked. The framing of the notes mattered: with a generic frame the model spent most of its notes remarking that the message was unfinished. A frame that asks for material for the answer, plus a logit bias on words such as user and typing applied only while reasoning ahead, brought that share down from 86 % to 17–26 % of the sentences.

Built for the hardware people own

The reference machine is an RTX 2060 with 6 GB of VRAM. Mafdet also runs on Apple Silicon through Metal, on AMD Radeon cards through Vulkan and on the CPU alone. On first start it detects the hardware, installs the matching llama.cpp build and downloads the model that runs best there. The catalog covers Qwen3, Llama 3, Gemma 3, Mistral and Phi-4 from 0.6B to 32B parameters, and any GGUF from Hugging Face can be added.

Every model comes with an estimate of what it will do on your machine before the download: whether it fits in VRAM, how fast it reads and writes and how long the answer takes after Enter. The estimate is within 12 % of the measured speeds on the machines tested, and one click replaces it with a real benchmark.

Measured, not guessed

Machine	Model	Without Mafdet	With Mafdet
RTX 2060, 6 GB	Qwen3-4B, 24 problems	12.2 s mean	3.2 s
RTX 2060, 6 GB	Qwen3-14B, 6 problems	108 s mean	47 s
M2 Pro, 16 GB	Qwen3-4B, 24 problems	18.1 s mean	5.6 s
M2 Pro, 16 GB	Qwen3-14B, 6 problems	32.7 s mean	12.1 s

Accuracy stays the same: 24/24 on the 4B and 6/6 on the 14B, with and without thinking ahead. The measuring tools ship with the app: mafdet hunt replays realistic typing (clean, messy and pasted input), mafdet race benchmarks prefill and decode, and mafdet judge checks every result against a from-scratch computation and must end with zero errors.