LocalLLaMA @sh.itjust.works Shifu @lemmy.world 4w ago Spent two weeks on a kernel that benchmarked 29x faster. End to end it's maybe 6-10%, and it's not even wired in yet.
LocalLLaMA @sh.itjust.works Shifu @lemmy.world 1mo ago BitNet b1.58-2B-4T at 36 tok/s on Xeon - pure C99, no Python/BLAS, single binary for ternary + GGUF