AIQB
TutorialsOrdinary

VRAM for local LLMs: why memory bandwidth sets your tokens per second

Source: DEV Community·

Summary

VRAM for LLMs is a bandwidth problem: every token streams the whole model from memory. Bandwidth per tier, the 20x offload cliff, and what fits in 16, 24 or 48 GB.
CategoryAI Tutorials & Practice
TierOrdinary
Published
Indexed by AIQB
SourceDEV Community
AIQB record IDintel-172054ed3676fc25038c933c
Read original source ↗More Tutorials

Related stories

Matched by category, entity tags and source

TutorialsOrdinary
187 live prompts, 27 bugs: what testing my local AI agent against a real 7B model taught me
I've been building CORTEX, an AI agent that runs entirely on my own laptop. It plans a task, runs...
Source: DEV Community
TutorialsOrdinary
What Happens When an AI Agent Goes Through a Real Development Workflow?
I’ve been experimenting with an AI-Driven Development workflow at Relipa, and one thing became clear:...
Source: DEV Community
TutorialsOrdinary
Your AI guardrail is green. It's also catching nothing.
The scariest security failure isn't the guardrail that's down, or the one that's weak. It's the one that's running, passing every health check, and quietly configured to catch nothing — green by construction. I found one in a benchmark of …
Source: DEV Community
TutorialsOrdinary
CodeSmith:Like Jev A -Harness for Cheap Brains
Prologue: Like Jev — A Harness for Cheap Brains Source version of CodeSmith: v0.5.0...
Source: DEV Community
TutorialsOrdinary
Logbook of a Spec-Driven Developer
Building a 45k-line app with AI agents: what worked, and what broke “Now that we have AI, why can’t...
Source: DEV Community
TutorialsOrdinary
N-grams: the language model before language models
Before transformers, before word embeddings, before anyone said "attention is all you need",...
Source: DEV Community
TutorialsOrdinary
14 Personal AI Agents in 2026: A Technical Guide to Architecture, Memory, Tools & Autonomy
The New Personal Agent Era: 14 Architectures Redefining Personal AI For the past few...
Source: DEV Community
TutorialsOrdinary
SignalForge: Building a Memory-Enabled Competitive Intelligence Agent
Competitive intelligence is not simply about knowing what competitors are doing today. The real...
Source: DEV Community
TutorialsOrdinary
Driving Your Real Microsoft Edge Tabs from AI Agents with edge-agent-bridge
Most browser automation tools built for AI agents start from an inconvenient premise: they assume...
Source: DEV Community
TutorialsOrdinary
My Coding Agent Built a Gas Station Monitor
At 02:01 one night, Telegram got a message from my personal portal: 🟢 You can go for 95-octane It...
Source: DEV Community