Written 31 July 2026. The services below have changed since: llama.cpp has taken over from vLLM and the model is Qwen3.6-35B-A3B. The layout with a proxy in the middle stayed exactly as it was, which is rather the point of this piece.
A local language model is not yet an assistant. Between the GPU and something you can actually use sits a layer of configuration that decides whether the whole thing is dependable or merely impressive on paper. My setup consists of three systemd services that each do one job.
The three layers
| Service | Port | Role |
|---|---|---|
| vllm | 8000 | serves the model on the Arc GPU |
| litellm | 4000 | proxy, routes locally or out to the cloud |
| hermes-agent | 9119 | the agent itself, with a dashboard |
The proxy in the middle is the most important decision here. Hermes never talks to vLLM directly, always to LiteLLM. That turns swapping models into a line of configuration rather than a rebuild, and it lets me pick a different model per task without the agent noticing anything.
hermes -> Qwen3-14B locally, with reasoning
hermes-fast -> same model, reasoning off
hermes-cloud-flash -> DeepSeek V4 Flash ($0.14/$0.28 per Mtok, 1M context)
hermes-vision -> Qwen3-VL-2B on the CPUA lesson about reasoning models
That second route, with reasoning switched off, looked like free optimisation: thirty times fewer tokens for a simple answer. Then I pointed it at automatic document tagging. With 412 existing tags in the prompt the model stopped choosing and started extending the list. Seventy-nine tags, no end in sight, until the client gave up after five minutes. A repetition penalty made the tags unique without stopping the flood. Only with reasoning enabled did a tidy answer of five to seven tags come back. The fast route has been reserved for simple work ever since, and never for picking from a long list.
Integrations, and why they ended up as wrappers
The agent talks to Paperless-ngx, Linkwarden, Radarr, Sonarr, Lidarr, Plex and a self-hosted SearXNG. At first I let the model call those APIs itself. That went wrong in a way I had underestimated: it invented endpoints that sound plausible and do not exist, misread the error code, and then advised me to change server settings to fix a problem that was never there.
What fixed it was a smaller assignment rather than a bigger model. Anywhere an invention would go unnoticed there is now a tested command in /usr/local/bin: paperless-upload, linkwarden-search, linkwarden-add. The model no longer produces anything at that point. It only passes things along. Each script reads its keys straight from the configuration file and stops hard on an empty value, because an empty token produces a 401 that looks suspiciously like a server fault.
What I would take away
Two things. Put a proxy between your agent and your model even if you only have one model, because you will want to swap later without breaking anything. And the line between what you let the model do and what you fix in code does not follow difficulty. It follows how visible your errors are. Code you can read back yourself is fine for the model to write. A list of your own data is not.
Leave a Reply