Making Local LLM Go Brrr

How to run your local LLM well: fast, reliable and with good quality.

Key metrics:

• Prefill speed: prompt/input tokens per second
• Decode speed: generated tokens per second
• Time to first token (latency)
• VRAM usage at target context length
• Quality at chosen model/quant/context settings
• Concurrency, if serving multiple users

Software (Inference)

Choose the serving stack based on workload:

• llama.cpp: best general local path, especially GGUF, CPU, Apple Silicon, and mixed CPU/GPU.
  • beelama.cpp: DFlash & TurboQuant in llama.cpp with up to 3x faster generation and 7.5x more KV cache in same VRAM
  • ik_llama.cpp: llama.cpp fork with additional SOTA quants and improved performance
• vLLM: strong GPU server for batching, throughput, OpenAI-compatible APIs, and production-style serving for modern GPUs.
• SGLang: good for structured/agentic serving and high-throughput multi-call workloads for modern GPUs.
• ZML: Zig based model run time.
• LuceBox: Local LLM inference server built for speed. Custom kernels, speculative prefill & decoding. (very advanced optimizations like hot MoE VRAM cache)
• Uzu: A high-performance inference engine for AI models (optimized for Apple hardware)

Performance checklist:

• Use the fastest supported attention kernels: FlashAttention, FlashInfer, FlashMLA, etc.
• Try speculative decoding / MTP / EAGLE-style decoding when supported, but benchmark with your actual model and sampling settings.
• Preserve prefix/KV cacheability:
  • keep the system prompt byte-identical
  • append new messages rather than rebuilding/changing history
  • avoid dynamic timestamps or changing tool schemas in the prompt prefix
  • use server-side prefix caching when available
• Tune KV cache precision:
  • for long context, test q8 KV even with q4 weights
  • aggressively quantized KV can hurt long-context coherence
• Approximate KV-cache compression for long-context or high-concurrency workloads:
  • TurboQuant: developed by Google (blog article)
  • KVarN: claims to beat TurboQuant on all dimensions (beellama.cpp supports it)

Tool-calling reliability:

• Add middleware for schema repair, retries, validation, and constrained tool loops -> https://github.com/antoinezambelli/forge

TODO:

• eval dynamic model routing based on query complexity (fast vs smart model)

Open Models

Run LLM models locally for complete control and privacy. Open-source (reproducible training) vs open-weight (free model weights) models. Dense models are smart but big and slow, Mixture of Expert (MoE) models only compute a fraction of parameters per layer and thus are faster to execute than dense models.

Compare model capability: https://artificialanalysis.ai/models
Find compatible models for your hardware: https://www.canirun.ai/ or try https://github.com/AlexsJones/llmfit -> rule of thumb: plan 70% of VRAM for the model weights and 20% for the KV-Cache.
Community benchmarks for local LLM: https://localmaxxing.com

Most models are too big for consumer GPUs, so quantized versions (compressed parameters) are used. Mixture of Quants (MoQ) is a new very efficient quant variant that does not quant weights uniformly but based on importance.

Curated open model list:

• Qwen3.6-35B-A3B Q4_K_XL: Strong MoE (3B active) model with MTP can run even on 8GB VRAM GPU with CPU offloading
  • Uncensored version - Reddit thread
  • https://x.com/witcheer/status/2053809265538678789
  • https://www.reddit.com/r/LocalLLaMA/comments/1tc132c/llamacpp_docker_images_to_run_mtp_models/
  • Custom thinking grammar (limit overthinking): https://github.com/andthattoo/structured-cot
    • TODO: find optimal thinking grammar using GEPA
• Qwen3.6 27B Q3_K_M - dense model, very good (Sonnet 4.6 performance) can run on 16GB VRAM
• Qwen3.5 9B Distilled - small but capable agentic dense model good for <8GB VRAM
  • MoQ variant (very efficient quantization)
• LFM2.5-8B-A1B - very fast MoE model 1.5B active + 128k context (agentic usefulness is limited though...)
• MiniCPM5-1B - optimized for mobile CPU/NPU inference (32k context window)

Hardware (GPU)

VRAM matters more than raw TFLOPs for model & context (prompt) size, but memory bandwidth and tensor cores matter for speed. Used datacenter GPUs can be good value, but check form factor, cooling, power, driver support, and PCIe vs SXM.

Interesting used GPU options:

One or two used Tesla V100 16GB cards are the best bang for the buck.

TODO:

• Check current AMD ROCm support.
• Compare used datacenter GPUs against RTX 3090/4090/5090-class consumer cards.
• Benchmark watts/token, not just tokens/sec.

References

• https://vllm.ai/blog/2026-05-11-turboquant
• https://blog.tymscar.com/posts/v100localllm/

#AI #tutorial