KV cache streaming - dynamic offloading of KV cache from VRAM to RAM

This branch adds an experimental, block-granular KV cache streaming path to the CUDA llama-server. It is intended for running long contexts when model weights leave too little VRAM for the complete KV cache.

With --kv-stream-stage-mib N, the authoritative KV tensors are stored in pinned host memory while a bounded CUDA pool is shared by resident KV pages and a transfer ring. The runtime adapts that split as the context grows: it keeps as many pages resident as the budget allows, reclaims resident space for staging when more streaming is required, and prefetches later layers while the current layer computes. This avoids relying on uncontrolled Unified Memory page thrashing and preserves exact attention over the full context.

(Not my project)

I just built and tested the above llama.cpp fork and I can attest that it really works. My previous "best Qwen 3.8 27B" on my 5060Ti 16GB and host DDR5 RAM was UD-IQ4XS beellama with KVarN KV cache 5/4 where I got 112000 in context at usable speeds (900 PP, 25 TG).

With this fork I'm now running the same model, KV cache 8.0/4.0 at 131072 context size at the same speed.