vLLM on 4 GPUs / 64GB: A Multi-Model Benchmark — For a Home Multi-Host Setup, I'd Go with vLLM

A 74B vision agent model in 22 GB on one A40: 136 tokens/s with its drafter, 70 without, about 27 GB used. What the 24 GB leaves out.

Here is the exact exchange. The model is llama3.2:latest (3.2B, Q4_K_M), running locally through...

A week of benchmarking Muse Glimmer 30B on AMD Strix Halo with llama.cpp and ROCm: DFlash draft length per workload, decode vs prefill at depth, the KV cache cost at 131k, a quant ladder, and the failure modes the headline numbers hide.

Llama.cpp is a high-performance inference platform designed for Large Language Models (LLMs) like...

Stop dragging PDFs and screenshots out of ~/Downloads. SlothTidy uses small open-weight models via Ollama and llama.cpp to automatically classify, rename, and organize your files 100% offline.

A local-first, zero-telemetry PII and secrets redaction proxy with bidirectional rehydration for open-weight models (Gemma, Llama, Mistral).

ENTERPRISE AI ARCHITECTURE & INFERENCE SYSTEMS ENGINEERING Technical Audit: PagedAttention KV...

Meta AI training appears explicitly in upcoming commercial terms. Here is who is covered, what the website permission says, and where robots.txt fits.

Beyond chat with llamadart · Part 4 Dash missed it. Pip’s scarf is by the pond, and all he has is...

Beyond chat with llamadart · Part 3 Pip’s scarf is missing. I wanted Dash to be able to fly over to...