Running Local LLMs for Coding Agents
Tool use matters more than model size when running coding agents locally.
There are three layers to this thing, and each one will find a way to frustrate you at some point.
- Model weights. Files on disk, loaded into GPU or unified memory. What you download from Hugging Face.
- Inference runtime / server. Takes token requests, runs GPU compute, exposes a local API. llama.cpp, Ollama, vLLM, SGLang. It turns a pile of weights into something your code can talk to.
- Coding agent harness. The scaffolding that reads files, runs shell commands, edits repos, runs tests, and feeds results back to the model. Aider, Cline, OpenHands, Codex.
The harness is doing more work than it gets credit for. It decides how context gets assembled, when tools get called, how errors get recovered, and how much autonomy the agent has between checkpoints. Put a weak harness on a strong model and the whole thing feels broken. The harness is directing the show. The model is just playing its part.
The prerequisite everyone skips: tool use and function calling support. Agentic workloads need a model that can reliably call tools. Bash execution, file editing, MCP integrations. A model that fumbles tool invocation cannot function as an agent, regardless of how it scores on code generation benchmarks. This is your first filter. Not context length. Not parameter count. Tool use. Everything else is downstream of that.
What the Benchmark Landscape Looks Like, and Why SWE-Bench Is the One That Matters for Agents
Most coding benchmarks measure whether a model can complete a function given a docstring. Single-turn, narrow, and honestly not that useful for predicting what happens when you let an agent loose on a real codebase.
SWE-bench Verified is different. It tests whether a model, running with a harness, can resolve a real GitHub issue end-to-end. Read the repo. Understand the bug. Write the fix. Pass the tests. For coding agents, SWE-bench Verified is your primary signal. HumanEval and MBPP are fine for what they are, but they tell you almost nothing about multi-step task resolution, which is the actual job.
The gap between these two benchmark types bites people regularly. A model can look great on HumanEval and then fall apart the moment a task requires staying coherent across a dozen tool calls and three files. The skills are genuinely different. Snippet completion and autonomous repo navigation are not the same event, and the models that look like sprinters sometimes just cannot hold up over distance.
Two extensions worth knowing about:
- SWE-bench Multilingual tests agentic resolution across codebases in multiple languages. Relevant if your team works outside Python-dominant repos.
- SWE-bench Pro is harder than Verified. Scores drop significantly across the board, which is useful for seeing which models degrade under real reasoning pressure.
When a vendor or community post quotes a benchmark score, the first question is always: which benchmark? A high HumanEval score on an agent recommendation is either a confused comparison or a deliberate one. Either way, it is not the number you want.
The Models Worth Running at the Frontier Tier, and What They Actually Cost to Serve
Qwen3-Coder-Next (Alibaba, Apache 2.0)
Released February 2026. A large number of total parameters, with a small fraction active per token via Mixture-of-Experts. The active-parameter design is what makes this model interesting. It is why you do not need a full multi-GPU server to run it.
Benchmark numbers: a leading score on SWE-bench Verified using the SWE-Agent scaffold, a slightly higher score on SWE-bench Multilingual versus DeepSeek-V3.2, a higher score on SWE-bench Pro versus DeepSeek-V3.2. It supports a 256K context window, non-thinking mode, OpenAI-compatible APIs via SGLang and vLLM, and GGUF quantizations for llama.cpp. Apache 2.0 license, no commercial restrictions buried in the fine print.
This is the model that covers IDE agents, CLI tools, and private coding copilots without requiring a server room.
Qwen3-Coder-480B-A35B (Alibaba, Apache 2.0)
480 billion total parameters, 35 billion active per forward pass. 69.6% on SWE-bench Verified. 256K context window.
Serving the full weights realistically means a multi-GPU node. Four to eight high-VRAM cards, vLLM or SGLang. This is a self-hosted inference server model. Before committing to that hardware, it is worth noting that Qwen3-Coder-Next scores nearly the same on SWE-bench Verified with dramatically lower serving requirements. Both options exist. The math on which one makes sense is not complicated.
A Note on the MoE Design
With Mixture-of-Experts models, active-parameter count drives memory and speed. Total-parameter count drives storage and serving overhead. These are different numbers and they matter differently. When a large total-parameter figure shows up in a spec sheet, it looks more intimidating than it usually is. Check which number you actually care about before making a hardware decision.
The Workstation-Tier Models That Run on a Single GPU or a High-End Mac
This is where most individual developers will actually spend their time. The good news is it is a legitimately good place to be right now.
Qwen3-Coder 30B (Alibaba, Apache 2.0)
30.5 billion total parameters, roughly 3.3 billion active per token. 256K context window extending to 1 million tokens with YaRN. Benchmarks: 92.1% on HumanEval, 77.2% on SWE-bench Verified, 84.3% on MBPP. Per promptquorum.com, highest across all three of any locally runnable model in this comparison.
At Q4KM quantization, it runs at 35 tokens per second on an RTX 4090. Fast enough to feel interactive in an agentic loop, which matters more than the raw number suggests. Slow token generation in an agent loop is not just annoying. It compounds across every tool call.
Devstral (Mistral AI, Apache 2.0)
Built on Mistral-Small-3.1-24B-Base and designed specifically for agentic tasks. Multi-step tool use and multi-file refactoring are the target use cases, not an afterthought the team stapled on later. At Q4 quantization, it runs at 40 tokens per second on an RTX 4090, which is the fastest of the workstation-tier options here.
Devstral was built with OpenHands integration in mind. That provenance is worth noting because it means the model and harness were designed to work together, not bolted together after the fact. If you want that kind of coherence between layers, Devstral on OpenHands is the pairing to look at.
DeepSeek-Coder V3 Distilled 16B
Trained by distilling reasoning traces from the larger V3.2 model into a 16 billion parameter dense student. Preserves chain-of-thought behavior at a fraction of the weight. Scores 40.5% on SWE-bench Verified, which is lower than the Qwen and Devstral options but still genuinely agentic-capable.
The significant thing here is that it fits on 12 GB of VRAM. An RTX 4070 Ti, a 12 GB laptop GPU. That is a lot of developers who already own the hardware and do not know it yet.
Gemma 4 31B (Google)
The outlier in this group because it is multimodal. Screenshots, UI diagrams, documentation images alongside code. Useful for debugging visual interfaces or working through codebases with image-heavy documentation. Strong on LiveCodeBench and Codeforces. Noticeably weaker than the Qwen models on SWE-bench Verified.
Right choice when multimodality matters. Not the right choice if large-repository agentic resolution is the primary criterion.
DeepSeek-Coder-V2 16B Lite
16 billion parameters, 2.4 billion active. 128K context. 90.2% on HumanEval. Runs on roughly 12 GB of VRAM via Ollama or vLLM.
One real caveat: this model predates the 2025 agentic wave. On multi-step SWE-bench tasks it trails the Qwen3-Coder and Devstral generation by a meaningful margin. It is better suited to code completion than autonomous agent loops. That is not a knock on the model. It is just a different tool for a different job.
On Licensing
Not uniform, and it matters before you ship anything.
- Qwen3-Coder and Devstral: Apache 2.0. No commercial restrictions.
- Codestral and Llama: carry usage restrictions. Read them before deploying in a product or distributing externally.
- GLM-4.6 and DeepSeek families: MIT-style permissive, but verify the specific variant you are using.
Licensing is less interesting to think about than benchmark scores and more expensive to get wrong. Read the terms once, early, and save yourself the headache later.
How to Match Hardware Tier to the Model You Want to Run
VRAM Is the Actual Constraint
A model that does not fit in VRAM cannot run at usable speed. It offloads to system RAM or disk, and throughput drops off a cliff. The rough planning math: about 2 GB of VRAM per billion parameters at FP16. Q8 quantization roughly halves that. Q4 halves it again. Everything else in your hardware decision is secondary to whether the model fits.
Entry Tier: 8 to 16 GB VRAM
RTX 3060, RTX 4060, M1 or M2 MacBook.
Runs 7B-class models at Q4 to Q8. Mistral 7B, Qwen 2.5 7B, Phi-3, Gemma 2 in the 2B and 9B variants. Expect 15 to 40 tokens per second. Fine for code completion, simple Q&A, and basic workflows. Agentic capability here is genuinely limited because the models that fit do not score well on SWE-bench. This is autocomplete territory, not autonomous agent territory. Know that going in.
Mid Tier: 24 to 48 GB VRAM
RTX 3090, RTX 4090, M2 or M3 Pro or Max.
Runs 13B to 32B models at Q4 to Q8. Qwen 2.5 32B, DeepSeek Coder V2 16B, Devstral, Qwen3-Coder 30B. Expect 10 to 25 tokens per second. Output quality competes with cloud APIs for most coding tasks. Devstral at 40 tokens per second and Qwen3-Coder 30B at 35 tokens per second both live here. One person, one machine, a local coding agent that actually works.
This is the sweet spot for individual developers who want serious agentic capability without a server rack.
High Tier: 64 to 96 GB Unified Memory
M3 Ultra Mac Studio, multi-GPU nodes.
Runs 70B-class models at Q4 to Q5 quantization. Llama 3.3 70B, DeepSeek R1 70B. Expect 5 to 15 tokens per second. Slower, but this is the quality tier where local inference starts genuinely matching cloud APIs on complex tasks. The M3 Ultra Mac Studio is the accessible single-machine option here. Multi-GPU is the server path. Both work. The Mac does not sound like a jet engine.
Frontier Tier: Multi-GPU Server
Four to eight high-VRAM cards. This is where Qwen3-Coder-480B-A35B and similar large MoE models live. The hardware commitment is real. It makes sense for organizations running inference at scale or teams that need frontier-local capability for compliance reasons. It does not make sense for a single developer trying to ship faster.
Pick the Tier That Fits Your Work
A Devstral or Qwen3-Coder 30B on a single RTX 4090 will handle the vast majority of real coding agent tasks faster and cheaper than a cloud API. Two years ago that sentence would have been wishful thinking. Today it is just accurate.
