P100 + Gemma not working properly?

Hi All, I just got a P100, and although it appears to work, i cant quite get gemma4 to run on it properly.

My setup is:

• Ubuntu 22.04 VM
• PCI passthrough the GPU (Connected via thunderbolt) (I know this is unideal, and I might try cut this out later)
• Nvidia 580 driver installed.
• llama.cpp built from source, with cuda enabled.

I got some mostly normal output from qwen, but whenever I try to load gemma4 I get a lot of <unused49><unused49><unused49><unused49>. I did have it briefly working, no idea what changed, but reboot and its back to unused spam.

Anyone hit this before? Ideas on what I should try change?