Running a dense 33B agent model on one 24 GB GPU: the offload trap and the empty reply

Clover-35B-A3B ran at 208 tokens/s on one RTX 4090 with llama.cpp v0.9.0, as an ordinary model. The traps before it answers, and what to measure.

Running local LLMs via Ollama, LM Studio, or llama.cpp usually starts with the same guessing game:...

A local LLM chat app built on llama.cpp, Node and a hand-built web UI, plus what it took to make localhost secure and ship the same model on Android.

Every LLM you've used, ChatGPT, Claude, Gemini, Llama, does the same thing under the hood. It...

How I wrapped llama.cpp in a small, coroutine-first Kotlin API, and the three problems that took the most time: threads, cancellation and broken emoji.

I started working on Llama Legends, a browser-based card collector, in mid-December 2025. The goal...

A while back I benchmarked llama.cpp on a 96GB GPU setup and found its concurrency behavior...

A 74B vision agent model in 22 GB on one A40: 136 tokens/s with its drafter, 70 without, about 27 GB used. What the 24 GB leaves out.

Here is the exact exchange. The model is llama3.2:latest (3.2B, Q4_K_M), running locally through...

A week of benchmarking Muse Glimmer 30B on AMD Strix Halo with llama.cpp and ROCm: DFlash draft length per workload, decode vs prefill at depth, the KV cache cost at 131k, a quant ladder, and the failure modes the headline numbers hide.