Felix Laurent
Hermes Agent runs on any model with an OpenAI-compatible endpoint, including one on your own GPU. The official docs suggest Gemma 4 31B as the local default. Independent agent benchmarks point to Qwen instead: Qwen3.8 27B solves 79.8% of Terminal-Bench 2.1 tasks, while Gemma 4 31B solves 43.4%. Both files weigh about 18–19 GB.
This guide covers what a local model needs inside Hermes Agent and what the benchmarks show. It then matches models to the amount of GPU memory you have. Model sizes come from the Ollama library and scores from Artificial Analysis, both checked on November 11, 2026.
What a local model needs to run Hermes Agent
Hermes Agent is the open-source agent from Nous Research, released under the MIT license. Its GitHub repository has 262,700 stars. The agent ships with more than 40 tools and six terminal backends, from local and Docker to SSH and Modal. One gateway connects it to Telegram, Discord, Slack, WhatsApp and Signal.
Every action the agent takes, from editing a file to running a command, goes through a tool call. The docs put it bluntly. Models without tool-call support "can only chat," and they "can't take actions." Gemma 2 and Llama 3.2 fall into that group in the docs' own model list.
The second requirement is context. Hermes needs at least 64,000 tokens for agentic work with tools, and it rejects smaller windows at startup. Ollama starts models with a much shorter context by default. You raise it with OLLAMA_CONTEXT_LENGTH=64000 or a Modelfile line PARAMETER num_ctx 64000, then set the same context_length in the Hermes config.
The serving stack matters too. llama.cpp needs the --jinja flag, or llama-server ignores the tools entirely. vLLM needs --enable-auto-tool-choice --tool-call-parser hermes. The Hermes desktop app can skip this setup. Its Settings → Providers → Local Models screen installs a llama.cpp server and sizes the context for you.
A 64k context also costs memory on top of the model file. A model that barely fits your card at short context will not fit with 64,000 tokens loaded. That is why the size of the file only tells half the story, and the benchmarks tell the other half.
What agent benchmarks say about local models
Nous Research runs its own leaderboard, the Hermes Index, launched in October 2026. It scores models on four suites inside the Hermes Agent harness. Claude Opus 5.5 leads at 63.31. The best open model, GLM 5.3, scores 39.23, but it has 753B parameters and needs a server rack. None of its 18 models is one of the local models in this guide.
For models that do fit, the closest independent signal comes from Artificial Analysis. Its Terminal-Bench 2.1 run gives a model 89 shell tasks in a sandbox, much like Hermes working in a terminal. Each score is the share of tasks solved on the first try, averaged over three runs.
Qwen3.8 27B, released in August 2026, leads the local field at 79.8%. The older Qwen3.6 27B solves 60.7%, and Gemma 4 31B solves 43.4%. Small models drop fast: Qwen3.5 9B reaches 29.2%, and gpt-oss-20b only 13.9%.
Terminal work is only part of an agent's day. τ²-Bench Telecom measures the other part: a customer-service agent has to call the right tools in the right order while talking to a simulated user.
GLM-4.7-Flash tops this board at 98.8%, ahead of Qwen3.6 35B A3B at 95.3% and Qwen3.6 27B at 94.2%. Even Qwen3.5 9B scores 86.8%. Gemma 4 31B manages 59.9%, and Gemma 4 12B 36.3%. Qwen3.8 27B has no score here, since Artificial Analysis moved newer models to a harder banking version of the test.
Benchmarks rank the models, but the card in your machine decides which ones you can actually run.
The best local LLM for 8–12 GB of VRAM
Qwen3.5 9B is the pick for small cards. Its Ollama file weighs about 6.6 GB, which leaves room for a 64k context on a 12 GB GPU. It scores 86.8% on τ²-Bench and 29.2% on Terminal-Bench 2.1, so it calls tools well and struggles with longer shell work.
Gemma 4 12B is the main alternative at 7.7 GB, and it trails on both tests. Small models also choke on the default Hermes setup. The docs warn that the full toolset plus the system prompt can exceed 32k tokens, and they suggest loading fewer tools with -t file,web.
Expect a 9B model to handle chat requests, notes and simple file jobs well. Multi-step coding runs need more memory, and that is where 16 GB cards come in.
The best local LLM for 16 GB of VRAM
At 16 GB the choice gets awkward. Gemma 4 26B A4B weighs 16 GB on its own, and gpt-oss-20b weighs 14 GB. Neither leaves much room for a 64k context without spilling into system RAM.
Of the two, Gemma 4 26B A4B is stronger on terminal work at 39.0% against 13.9%. gpt-oss-20b does better on tool calls, with 60.2% on τ²-Bench against 43.6%. Both trail Qwen3.5 9B on tool calling, so many 16 GB owners will get more from the 9B model with a long context.
A 16 GB card is a compromise. With 24 GB, the 27B models become usable, and the benchmark gap closes quickly.
The best local LLM for 24 GB of VRAM
A 24 GB card, such as an RTX 4090 or 3090, is where local Hermes Agent becomes practical. Qwen3.8 27B weighs 18 GB in Ollama, comes under Apache 2.0 and supports a 256k context. Its 79.8% on Terminal-Bench 2.1 is the best local result in this guide.
GLM-4.7-Flash is the tool-calling specialist at 19 GB. It scores 98.8% on τ²-Bench and ships under the MIT license. It suits a Hermes setup that mostly runs Telegram or Slack requests, searches and calendar jobs.
Qwen3.6 35B A3B weighs 23 GB, which leaves little room for context on a 24 GB card. It activates only 3B of its 36B parameters per token, so it runs fast once loaded. On a 32 GB card or a Mac with 48 GB of unified memory, it becomes a comfortable option.
Gemma 4 31B, the docs' recommendation, also fits at 19 GB. It works, and the docs confirm "strong tool use and reasoning." On both independent tests it trails the Qwen models of the same size.
When to add a cloud model to Hermes Agent
Bigger local models do not always help. gpt-oss-120b needs about 65 GB and scores 26.2% on Terminal-Bench 2.1, far below Qwen3.8 27B. On a Mac with 96 GB or more, a 27B model at higher precision is often the better use of that memory. Our Mac Studio vs RTX 5090 comparison covers that hardware choice.
The Hermes docs say "70B+ or cloud models like Claude Opus are noticeably better" for complex reasoning. Hermes can run a local model by default and fall back to a hosted one through OpenRouter. That split keeps private and routine work on your machine and sends only the hardest tasks out.
Speed is the other limit. The docs estimate a 9B model on a modern 8-core CPU at about 10 tokens per second, and a 31B model at 2–5. Without a GPU, the agent can sit silent for minutes before it answers.
So which local LLM is best for Hermes Agent?
For most people with a 24 GB GPU, Qwen3.8 27B is the best local LLM for Hermes Agent. It leads Terminal-Bench 2.1 at 79.8%, fits in 18 GB and carries an Apache 2.0 license. GLM-4.7-Flash is the better choice when the agent mostly calls tools from chat, with 98.8% on τ²-Bench.
On 8–12 GB, Qwen3.5 9B is the realistic option, with a trimmed toolset and a 64k context. Gemma 4 31B, the default in the Hermes docs, runs fine but trails both Qwen models on the two agent tests. Whatever you pick, set the context to at least 64,000 tokens in both Ollama and Hermes, or the agent will fail before it starts.
FAQ
The questions below come up most often when people set up Hermes Agent with a local model.
Does Hermes Agent work offline?
Yes. Point it at a local server such as Ollama, llama.cpp, vLLM, SGLang or LM Studio. The docs show the setup with Ollama at http://localhost:11434/v1 as a custom endpoint.
Why does Hermes Agent refuse my local model?
The usual cause is context. Hermes requires at least 64,000 tokens and rejects smaller windows at startup. Raise the context in Ollama and set the same context_length in the Hermes config.
Can I run Hermes Agent without a GPU?
You can, but it is slow. The docs estimate about 10 tokens per second for a 9B model on an 8-core CPU. They also suggest raising the timeout with HERMES_API_TIMEOUT=1800.
Do I need a Nous Research Hermes model?
No. Hermes Agent works with any model that supports tool calls. The docs recommend Gemma 4 31B for local use and Nous Portal or OpenRouter for hosted models.
