llama rpc exists and works really well

I've been back'n'forth for a while whether to invest in a second 5060Ti to be able to go dual GPU on my main workstation. The target has always been to be able to run Qwen 3.6 27B at a usable model quant, kv-cache quant and context size.

Today I tried out llama.cpp with the rpc option. I sort of knew at the back of my head that there were projects doing cross-computer inference but I didn't know it was built into llama.cpp.

Sure, this requires that you have another machine on your local network (latency matters) with a GPU - but in my case I actually have two. One server that has a 12GB A2000 and one Bazzite box with an AMD something also with 12GB VRAM.

I just compiled the latest llama.cpp HEAD with the rpc option enabled on my headless server and launched my main llama-server pointing to it. With the 12GB of the A2000 in the mix I'm now able to comfortably run the 27B model Q4KM, Q80 kv-cache and 140000 context size, and I've still not maxed out the available VRAM.

PP at 400 (a bit low, will see if that can be tuned) and TG at 25 (MTP and ngram enabled) makes this perfectly usable. That's the token/second during inference I would've expected from a dual GPU setup in the workstation anyway.

So, if you also happen to have another machine on your network with spare VRAM - try out llama rpc. I have no idea why it took this long for me to do.

) This is literally my first run with random defaults - tuning starts now