Four weeks ago I closed the first piece about this card with a rule of thumb: dense architecture, AWQ or GPTQ, at least six months old. Anything newer broke at the kernel level and there seemed to be little to do about it.
That rule is now wrong, and I am glad it is. The machine currently runs Qwen3.6-35B-A3B, a mixture-of-experts model, on the same Arc Pro B70. It is two and a half times the size of what I was running in July and it produces tokens faster.
What actually changed
One thing: the runtime. vLLM has been replaced by llama.cpp built against SYCL, serving the model on port 8000 exactly as before. LiteLLM in front of it did not need a single line changed, which is the second article’s argument arriving unannounced.
The vLLM service is still on the box with a Conflicts= line against the llama.cpp unit, so the two can never claim the GPU at once. Switching back is one systemctl start away. It has stayed idle.
| July, on vLLM | Now, on llama.cpp | |
|---|---|---|
| Model | Qwen3-14B-AWQ, dense | Qwen3.6-35B-A3B, mixture of experts |
| Quantisation | AWQ 4-bit | Q4_K_M |
| Throughput | 55 tok/s | 69.7, 75.1 and 75.7 tok/s across three runs |
| Context | 131k | 131k |
| Concurrency | 1.93x | one request at a time |
The rule was about the runtime all along
This is the part worth writing down. Everything in that first article about broken kernels was true, and I measured all of it honestly. What I got wrong was the conclusion I drew from it. I treated a limitation of Intel’s XPU kernels inside vLLM as though it were a property of the card.
A quantised mixture-of-experts model was exactly the sort of thing that used to die on the first real inference. On llama.cpp it loads and runs, all 99 layers on the GPU, flash attention on, key and value cache at q8_0. The hardware never had an opinion about mixture-of-experts. One piece of software did.
If you own one of these cards and have been avoiding newer architectures because someone measured them failing, try the other runtime before you accept that.
What it costs
Concurrency, mainly. llama.cpp runs here with --parallel 1, so requests queue instead of overlapping. vLLM managed 1.93 simultaneous streams at this context length. For one person and a handful of scheduled jobs that has not once been the bottleneck, and I would give up rather more than that for a model of this size at this speed. On a machine serving several people it would be the first thing to reconsider.
One argument from the first article has quietly collapsed as well. Image analysis used to sit on the processor because a vLLM instance needed 130 seconds to produce its first token, and for the occasional photo that wait dominated everything. The current setup keeps the model resident and loads the vision projector alongside it, so images now go through the same GPU-served model. The separate CPU route is gone, and 4.1 GB of RAM came back with it.
Next
Qwen 3.8 MoE, as soon as it is available. If the pattern from this round holds, the interesting question will not be whether the card can handle it but whether llama.cpp has the kernels ready on day one. I will measure it the same way and report whatever comes out, including the case where it does not work.
Leave a Reply