Skip Navigation

InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)D

doodlebob

@ doodlebob @lemmy.world

Posts
1
Comments
30
Joined
3 yr. ago

  • zai-org/GLM-4.5-Air · Hugging Face

    Jump

  • I'm just gonna try vllm, seems like ik_llama.cpp doesnt have a quick docker method

  • zai-org/GLM-4.5-Air · Hugging Face

    Jump

  • IK sounds promising! Will check it out to see if it can run in a container

  • zai-org/GLM-4.5-Air · Hugging Face

    Jump

  • I'll take a look at both tabby and vllm tomorrow

    Hopefully there's cpu offload in the works so I can test those crazy models without too much fiddling in the future (server also has 128gb of ram)

  • zai-org/GLM-4.5-Air · Hugging Face

    Jump

  • Unfortunately i didn't set up nvlink, but ollama auto splits things for models which require it

    I really just a "set and forget" model server lol (that's why I keep mentioning the auto offload)

    Ollama integrates nicely with OWUI

  • zai-org/GLM-4.5-Air · Hugging Face

    Jump

  • omg, I'm retarded. Your comment made me start thinking about things and...I've been using q4 without knowing it... I assumed ollama ran the fp16 by default 😬

    about vllm, yeah I see that you have to specify how much to offload manually which I wasn't a fan of. I have 4x 3090 in an ML server at the moment but I'm using those for all AI workloads so the VRAM is shared for TTS/STT/LLM/Image Gen

    thats basically why I kind of really want auto offload

  • zai-org/GLM-4.5-Air · Hugging Face

    Jump

  • yeah, im currently running the gemma 27b model locally I recently took a look at vllm but the only reason i didnt want to switch is because it doesnt have automatic offloading (seems that it's a manual thing right now)

  • zai-org/GLM-4.5-Air · Hugging Face

    Jump

  • Just read the L1 post and I'm just now realizing this is mainly for running quants which I generally avoid

    I guess I could spin it up just to mess around with it but probably wouldn't replace my main model

  • zai-org/GLM-4.5-Air · Hugging Face

    Jump

  • Thanks, will check that out!

  • zai-org/GLM-4.5-Air · Hugging Face

    Jump

  • I'm currently using ollama to serve llms, what's everyone using for these models?

    I'm also using open webui as well and ollama seemed the easiest (at the time) to use in conjunction with that