Skip Navigation

InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)R

robber

@ robber @lemmy.ml

Posts
16
Comments
42
Joined
4 yr. ago

  • Whats a good alternative to Instagram?

    Jump

  • Pixelfed

  • Do you host your own AI?

    Jump

  • Well compared to the strix, 400GB/s is not that bad, I think with fast system RAM and expert offloading you could squeeze quite something out of it when running stuff in the 100b-a10b regions.

    Your bigger problem is going to be future software support.

  • Do you host your own AI?

    Jump

  • In case you missed the Ornith 1.0 release (Qwen and Gemma RL finetunes for agentic / coding workloads), they look interesting to bridge the gap until we see larger 3.6 models or a 3.7 release. I didn't test them yet but according to benchmarks, the 35b MoE seems to be more or less on par with Qwen3.6 27b dense, while ofc a lot faster.

  • Do you host your own AI?

    Jump

  • You can control how much context should be fitted with --fit-ctx and how much space the algorithm should leave unallocated (even on a per-GPU basis) with --fit-target.

  • Do you host your own AI?

    Jump

  • I currently run Qwen3.6-27b on llama.cpp and use it via openwebui. Mostly, I use it for web research via tavily, to a lesser extent for coding and interactively learning about things that are new to me but common in training data (such as basic math or ML concepts).

  • Do you host your own AI?

    Jump

  • Given the 27b is a dense model, I think the numbers are quite ok. Curious about the quant tho.

    The cool thing about the strix is its large unified memory, but it lacks memory bandwith for compute intensive workloads. Something like Qwen3.5-122b MoE with only like 12b active parameters might run at twice the speed if it fits the configuration.

  • Do you host your own AI?

    Jump

  • Since implementation of the --fit parameter and its relatives, and --fit on becoming the default, llama.cpp intelligently decides what to offload. For me, it made --n-cpu-moe obsolete.

  • Gemma4 12b released with "unified" approach to multi-modality

    Jump

  • You might want to check out heretic or similar tools. I did not try it but there are a lot of heretic finetunes available ond HF.

  • Gemma4 12b released with "unified" approach to multi-modality

    Jump
  • LocalLLaMA @sh.itjust.works
    robber @lemmy.ml

    Gemma4 12b released with "unified" approach to multi-modality

    huggingface.co /google/gemma-4-12B

  • Your best local LLM for low-VRAM (6GB)?

    Jump

  • Late to the party, but this was just released: LiquidAI/LFM2.5-8B-A1B-GGUF

    I guess you could fully fit it at Q4 with a little context if you need all the speed you can get, or offload the experts to RAM if you prefer higher quality and/or more context.

  • Hardware for local inference?

    Jump

  • Your biggest issue with 2010 cards will be software (inference engine) support, I assume.

  • Hardware for local inference?

    Jump

  • To add some practical advice:

    It depends on what you mean by more advanced models. I run Qwen3.6-27b on 48GB VRAM across 3 cards (RTX 2000e Ada), and with the recent software optimizations merged into llama.cpp (tensor parallelism & MTP) I get around 30 tokens per second in generation. I use the model through openwebui for (agentic) web research and simple Q&A mostly and I'm quite happy with what it can do.

    If you want something similar, maybe look at one or two second hand V100 PCIE 32GB. Or something from the Intel Arc Pro series, if you don't mind the software support lacking behind a bit (as in less optimized).

    Also it might be worth reading into the difference of dense vs MoE models, if you're new to that. For MoE models, if your system RAM is fast enough, it's often viable to offload the "experts" (largest parts of such models) to RAM, reducing VRAM capacity needs. Note that server motherboards with e.g. octa-channel RAM have a huge advantage over consumer boards (making DDR4 interesting despite slower speed per module).

    And to adress your last question, while I have no direct experience, I've seen posts online about people connecting Strix Halo or DGX Spark devices, but usually via a 10+Gbit/s switch as interconnect is crucial (except if you just want to load balance).

    Self-hosting LLMs is a very fun thing to do, but also a time- and money-consuming rabbit hole. You might wanna check out the LocalLlama community over at shitjustworks.

    Edit: typos

  • Llama.cpp MTP Support merged - up to 2.5x speed increase

    Jump

  • Using MTP combined with tensor parallelism, I was able to go from running Qwen3.6 27b at ~7t/s to ~30t/s which I think is an insane boost (3x RTX 2000e Ada).

  • LocalLLaMA @sh.itjust.works
    robber @lemmy.ml

    llama.cpp: don't sleep on --split-mode tensor

    github.com /ggml-org/llama.cpp/pull/19378

  • Noob here: Why is Google making Gemma open-source?

    Jump

  • A lot has been said, but to add to the list I'd say it gives them access to quite a large pool of free testers.

    LLM architectures and optimization techniques change rapidly and by releasing open-weight models a lot of enthusiasts will evaluate new models for free, help implement support in inference engines, catch bugs etc. (and in turn, ofc, get a new model to run for free, so it's at least somewhat symbiotic).

    We have at least seen this quite obviously when Alibaba released Qwen3-Next, which was a somewhat undertrained but still useful model which introduced the architecture that their latest models now use "in production" (also their paid "Max" models).

  • LocalLLaMA @sh.itjust.works
    robber @lemmy.ml

    Gemma 4 is here

    huggingface.co /collections/google/gemma-4

  • HP's ink-blocking firmware may violate new global sustainability rules

    Jump

  • Global sustainability rules???

  • Smaller qwen3.5 models released

    Jump

  • I don't follow the discussions on this topic very closely, but as I understood, there are different ways to achieve the goal, but all impact quality to some extent. Heretic is discussed as one one of the SOTA methods.

  • LocalLLaMA @sh.itjust.works
    robber @lemmy.ml

    Smaller qwen3.5 models released

    huggingface.co /collections/Qwen/qwen35

  • Qwen3-Coder-Next

    Jump

  • Yeah I enjoy it as well. Just in case you missed it - a fix was merged into llama.cpp two days ago which is said to improve quality.

    Edit: I stand corrected - the fix for the issue you're experiencing has not yet been merged.

  • LocalLLaMA @sh.itjust.works
    robber @lemmy.ml

    Qwen3-Coder-Next

    huggingface.co /Qwen/Qwen3-Coder-Next

  • Forgejo v13.0 is available

    Jump

  • Relevance of GPU driver version for inference performance

    Jump

  • I see. When I run the inference engine containerized, will the container be able to run its own version of CUDA or use the host's version?

  • LocalLLaMA @sh.itjust.works
    robber @lemmy.ml

    Relevance of GPU driver version for inference performance

  • LocalLLaMA @sh.itjust.works
    robber @lemmy.ml

    Magistral-Small-2509 by Mistral has been released

    huggingface.co /mistralai/Magistral-Small-2509
  • LocalLLaMA @sh.itjust.works
    robber @lemmy.ml

    Qwen3-Next with 80b-a3b parameters is out

    huggingface.co /collections/Qwen/qwen3-next-68c25fd6838e585db8eeea9d
  • LocalLLaMA @sh.itjust.works
    robber @lemmy.ml

    ExLlamaV3 adds tensor parallelism support

    github.com /turboderp-org/exllamav3/releases/tag/v0.0.6
  • LocalLLaMA @sh.itjust.works
    robber @lemmy.ml

    New, promising MoE model "Hunyuan" by Tencent

    huggingface.co /tencent/Hunyuan-A13B-Instruct
  • LocalLLaMA @sh.itjust.works
    robber @lemmy.ml

    Do you quantize models yourself?

  • Selfhosted @lemmy.world
    robber @lemmy.ml

    Any experience with Pangolin?

  • Technology @lemmy.world
    robber @lemmy.ml

    More than 140 Kenya Facebook moderators diagnosed with severe PTSD

    www.theguardian.com /media/2024/dec/18/kenya-facebook-moderators-sue-after-diagnoses-of-severe-ptsd
  • Selfhosted @lemmy.world
    robber @lemmy.ml
    Solved

    Chaining routers and GUA IPv6 addresses

  • Selfhosted @lemmy.world
    robber @lemmy.ml

    Any of you have a self-hosted AI "hub"? (e.g. for LLM, stable-diffusion, ...)

  • Selfhosted @lemmy.world
    robber @lemmy.ml

    Migrated my self-hosted Nextcloud to AIO and I absolutely love it