Skip Navigation

InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)B

BeefAndPoultry

@ BeefAndPoultry @lemmus.org

Posts
6
Comments
18
Joined
3 wk. ago

  • Qwen3.8-2.4T-A95B Open Weights Released!

    Jump

  • model : Ling-3.0-tiny (8b)

    Jump

  • my laptop is crappy, so like 5 tokens per second lol, prompt processing of like 20 tokens per second

    I think a decent laptop nowadays, even running CPU only, could probably do like 5x faster

  • model : Ling-3.0-tiny (8b)

    Jump

  • I've run Qwen 3.5 4b and Gemma 4 e2b on CPU only, this should be faster than those I think (fewer active parameters). If you have AVX512 or AVX10 then it should help a bit. Still slow compared to a GPU lol.

  • model : Ling-3.0-tiny (8b)

    Jump

  • Thinking injection to modify model behaviour (making Gemma 4 less lazy)

    Jump

  • Actually funny he's not asking it to work harder (that would be system prompt or user message), he's forcing it to think that it will work harder

  • Thinking injection to modify model behaviour (making Gemma 4 less lazy)

    Jump

  • That's a really cool idea. It's like inception for an LLM, you make it think it was the one that thought of this lol

  • Thinking injection to modify model behaviour (making Gemma 4 less lazy)

    Jump

  • llama.cpp in progress pull request for smart caching of MoE experts, 16% to 35% TPS boost for my RTX 2080

    Jump

  • (Oops I got my Gemma and Qwen speeds mixed up, edited the post to fix it.)

    But now with the new commits they added, with the same number of hot experts, Qwen is up to about 34. If I increase hot experts to 48 then I get around 37.

    Gemma is still around 23 with just 10 hot experts. With 16 hot experts I get about 26 TPS. If I overprovision my VRAM (thanks to GGML_CUDA_ENABLE_UNIFIED_MEMORY=1) then 24 hot experts can give me 29 TPS, and 32 hot experts 34 TPS.

  • llama.cpp in progress pull request for smart caching of MoE experts, 16% to 35% TPS boost for my RTX 2080

    Jump

  • In a few minutes a significant performance improvement incoming

    👀

  • LocalLLaMA @sh.itjust.works
    BeefAndPoultry @lemmus.org

    llama.cpp in progress pull request for smart caching of MoE experts, 16% to 35% TPS boost for my RTX 2080

    github.com /ggml-org/llama.cpp/pull/26563

  • GLM-5.3 Imminent drop

    Jump

  • this has been a crazy few weeks! lol

  • Qwen 3.8 Max (2.4T-a95b) and 27B open weights being released next week

    Jump

  • true, it's not perfectly clear

    also I just saw this

  • LocalLLaMA @sh.itjust.works
    BeefAndPoultry @lemmus.org

    preserve_thinking fix for Gemma 4 templates

  • Qwen 3.8 Max (2.4T-a95b) and 27B open weights being released next week

    Jump

  • Qwen 3.8 Max (2.4T-a95b) and 27B open weights being released next week

    Jump

  • Gemma is probably good for that, as long as it's consistently succeeding at the tool calls.

  • LocalLLaMA @sh.itjust.works
    BeefAndPoultry @lemmus.org

    Qwen 3.8 Max (2.4T-a95b) and 27B open weights being released next week

  • unsloth/DeepSeek-V4-Flash-0731-GGUF · Hugging Face

    Jump

  • make sure that holds up with large context, you might need to step down to Q3 (which I've heard is still good for this model, many people are even using IQ2)

  • unsloth/DeepSeek-V4-Flash-0731-GGUF · Hugging Face

    Jump

  • unsloth/DeepSeek-V4-Flash-0731-GGUF · Hugging Face

    Jump

  • the original model has a lot of parts that were natively trained in 4 bit, so those layers can't go higher

  • LocalLLaMA @sh.itjust.works
    BeefAndPoultry @lemmus.org

    unsloth/DeepSeek-V4-Flash-0731-GGUF · Hugging Face

    huggingface.co /unsloth/DeepSeek-V4-Flash-0731-GGUF

  • How to run Qwen 35b-a3b on 4GB to 8GB of VRAM (with 24+GB system RAM)

    Jump

  • --n-cpu-moe 36 --spec-type draft-mtp --spec-draft-n-max 3 does seem to speed up token generation for me

    Can't use llama-bench for MTP. In a basic tests it seems to improve from about 26 to 30 tokens per second output. But it seems to hurt my input speed from about 1300 pp down to 800.

  • LocalLLaMA @sh.itjust.works
    BeefAndPoultry @lemmus.org

    How to run Qwen 35b-a3b on 4GB to 8GB of VRAM (with 24+GB system RAM)