Skip Navigation

InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)T

TheMightyCat

@ TheMightyCat @ani.social

Posts
2
Comments
31
Joined
1 yr. ago

  • take their money and arrest those responsible

    Jump

  • Sure there probably is a minimum somewhere, but most people are far above that minimum.

  • take their money and arrest those responsible

    Jump

  • As if there is nothing in between "don't do anything" and "go live in the woods"

  • This is a glass

    Jump 
  •  cpp
        
// This might be a glass
Glass* glass = dynamic_cast<Glass*>(container);
  • Deleted

    Rebuilding a social platform – how to use an AI agent as a pair programmer, not a black box?

    Jump

  • It depends, do you want to learn or do you want to get something fast and don't care how?

    If you want to learn don't let AI write anything. Ask it questions if you don't understand, ask it to explain, but wrte everything yourself.

    If you want to get something that works and don't care how, make extensive tests. Use a minimum test coverage. Automaric AI reviewing the PR of the AI, And keep the agent going until it passes the CI.

    To awnser your question specifically read the plans very carefully. Most of the time the agents execute the plan fairly faithfully, you don't need to correct the code much if you make sure to read the initial plan very carefully and make changes on where it made a mistake.

    This does build on the fact that you can actually understand what its writing, if you can't, go with the first approach and don't use AI at all for writing code.

  • GPU bifurcation - options

    Jump

  • Yes, if you use layer parallelism. When using tensor parallelism PCIe bandwith is a very important factor.

  • Storage?! In this economy!?

    Jump

  • Ah ok, I've had these drives for a while now but ended up using SSDs instead so they just sat in my parts drawer reminding me of a bad purchase xd.

    Crazy how they are like 3x the price now from when i bought them in 2023

  • Storage?! In this economy!?

    Jump

  • Do you happen to live in zuid holland?

    I have 2 unopened Seagate IronWolf ST4000VN006 that I'll be happy selling.

    No idea what a fair price or your budget would be. That is if you are interested in these drives to begin with.

  • She looks so happy with her new magic tome

    Jump

  • Selfhosted & AI

    Jump

  • My 2 cents are that the issue is promotion not AI, if people started promoting stuff made without AI that would still be spam.

    From the rules:

    F/LOSS Exception: If your post is about a project that is completely open source & can be self-hosted in full without payment, your post is exempt from the 10% requirement. The exception does not exempt you from the account age requirement.

    I would propose making this the requirement and not an exception, forbid all promotion of closed source, and allow the 10% requirement for open source projects.

  • Removed

    Wrote up a full guide for running AI locally on Windows (LM Studio + Ollama + Open WebUI)

    Jump

  • Why would anyone pay for this when documentation is readily available, even for non technical users?

    https://github.com/oobabooga/textgen

    Installing something like textgen is as easy as downloading a single executable.

  • Why do native UI frameworks suck so much?

    Jump

  • Either the UI is defined in some DSL, that’s loaded (or compiled) and then you spend most of the time writing getElement(pathToElement) and wiring it up, or you have to boilerplate create each element and parent.addChild(element).

    Im saying "there are no elements" because of what OP says here, the system i'm using now doesn't have that.

    You have a panel+border+text “primitive” drawing functions. Nothing is stopping you from creating a single function that calls all 3 of those. You probably should, since it’s probably a common pattern. You could call it DrawBoxedText. There is no difference between a DrawBoxedText function and a BoxedText element with a draw() method.

    yeah i could make a function for that, but i would argue that "here is no difference between a DrawBoxedText function and a BoxedText element with a draw() method." there is a big difference between these. Personally i find using a function like DrawBoxedText much easier.

  • Why do native UI frameworks suck so much?

    Jump

  • I had my own library for UI that tried to have a element model, but it was bloated and didn't scale well. Now im drawing everything via vulkan compute shaders and its fast and looks great.

    I think the philosophy needs to switch from elements back to pixels, atleast that is what worked for me.

    i dare to say its as easy as webdev

    snippet from my game:

     cpp
        
DrawMenuButton(buf, btnExit, "Exit to title", exitHover, exitHover && window.mouseLeftHeld, false, font, topFontPx, exitCol);
DrawMenuButton(buf, btnSave, "Save changes", saveHover, saveHover && window.mouseLeftHeld, false, font, topFontPx, saveCol);
PushPanel(buf, titleRect, panelBgColor, 6.0f * s);
PushBorder(buf, titleRect, borderColor, 6.0f * s, 1.5f * s);
DrawText(buf, "OPTIONS", titleRect.x + titleRect.w * 0.5f, titleRect.y + titleRect.h * 0.5f + topFontPx * 0.32f, font, topFontPx, textUnfocusedColor, TextAlign::Center);

    this is in the draw function, and along with the other code produces this menu:

  • Replaced $40/month in AI API subscriptions with self-hosted Ollama + n8n

    Jump

  • I'm running 2x4090, the 35B fits very comfortable in that.

    For large models like the 397B without a ton of money there are several ways, ive seen posts of people using arrays of used 3090s with good results.

    The other option is CPU inference although with current RAM prices that is less cost effective.

    I was looking at maybe an array of Milk-V JUPITER2 since vllm added riscv support which could be very cost effective.

  • Replaced $40/month in AI API subscriptions with self-hosted Ollama + n8n

    Jump

  • Depending what OP was using before but going from something like GPT5.2 to LLama 3 8B will be a massive difference (Although OP says to use it only for basic tasks so that does offset it)

    LLama 3 already being a very old model doesn't help either

    I run Qwen3.5-35B-A3B-AWQ-4bit which while leagues ahead of LLama 3 8B still is a very noticeable difference.

    This is not to say open source is bad, if one had the resources to run something like Qwen3.5-397B-A17B it would also be up there.

  • Building a Linux Phone

    Jump

  • I think everyone has once thought of the idea of taking an sbc and a touchscreen and making a linux phone, cool to actually see one!

  • My 8gb vram system as i try to load GLM-4.6-Q0.00001_XXXS.gguf:

    Jump

  • i found the reason, somehow setting --max_num_seqs 1 makes vllm way more efficient.

    Not sure exactly what it does but i think its because vllm batches requests and the api was using with exlamav3 doesn't

    Now im doing 100k with vllm too

     
        
(Worker_TP0_EP0 pid=99695) INFO 11-03 17:34:00 [gpu_worker.py:298] Available KV cache memory: 4.73 GiB
(Worker_TP1_EP1 pid=99696) INFO 11-03 17:34:00 [gpu_worker.py:298] Available KV cache memory: 4.73 GiB
(EngineCore_DP0 pid=99577) INFO 11-03 17:34:00 [kv_cache_utils.py:1087] GPU KV cache size: 103,264 tokens
(EngineCore_DP0 pid=99577) INFO 11-03 17:34:00 [kv_cache_utils.py:1091] Maximum concurrency for 100,000 tokens per request: 1.03x
(EngineCore_DP0 pid=99577) INFO 11-03 17:34:00 [kv_cache_utils.py:1087] GPU KV cache size: 103,328 tokens
(EngineCore_DP0 pid=99577) INFO 11-03 17:34:00 [kv_cache_utils.py:1091] Maximum concurrency for 100,000 tokens per request: 1.03x

    I would say exlamav3 is still slightly more efficient but this explains the huge discrepancy, exlamav3 also allows setting GB per gpu which allows me to get a view more GB then vllm which spreads it evenly because a bunch of memory on gpu 0 is used for other stuff

    As for the T/s its about the same, in the 80-100 range, this is what im getting with vllm:

     
        
(APIServer pid=99454) INFO 11-03 17:36:31 [qwen3coder_tool_parser.py:76] vLLM Successfully import tool parser Qwen3CoderToolParser !
(APIServer pid=99454) INFO 11-03 17:36:32 [qwen3coder_tool_parser.py:76] vLLM Successfully import tool parser Qwen3CoderToolParser !
(APIServer pid=99454) INFO:     127.0.0.1:32968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=99454) INFO:     127.0.0.1:32968 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=99454) INFO 11-03 17:36:32 [qwen3coder_tool_parser.py:76] vLLM Successfully import tool parser Qwen3CoderToolParser !
(APIServer pid=99454) INFO 11-03 17:36:34 [qwen3coder_tool_parser.py:76] vLLM Successfully import tool parser Qwen3CoderToolParser !
(APIServer pid=99454) INFO:     127.0.0.1:32968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=99454) INFO:     127.0.0.1:32968 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=99454) INFO 11-03 17:36:34 [qwen3coder_tool_parser.py:76] vLLM Successfully import tool parser Qwen3CoderToolParser !
(APIServer pid=99454) INFO 11-03 17:36:34 [qwen3coder_tool_parser.py:76] vLLM Successfully import tool parser Qwen3CoderToolParser !
(APIServer pid=99454) INFO:     127.0.0.1:32968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=99454) INFO:     127.0.0.1:32968 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=99454) INFO 11-03 17:36:34 [qwen3coder_tool_parser.py:76] vLLM Successfully import tool parser Qwen3CoderToolParser !
(APIServer pid=99454) INFO 11-03 17:36:34 [loggers.py:127] Engine 000: Avg prompt throughput: 461.4 tokens/s, Avg generation throughput: 17.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 66.9%
(APIServer pid=99454) INFO 11-03 17:36:35 [qwen3coder_tool_parser.py:76] vLLM Successfully import tool parser Qwen3CoderToolParser !
(APIServer pid=99454) INFO:     127.0.0.1:32968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=99454) INFO:     127.0.0.1:32968 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=99454) INFO 11-03 17:36:35 [qwen3coder_tool_parser.py:76] vLLM Successfully import tool parser Qwen3CoderToolParser !
(APIServer pid=99454) INFO 11-03 17:36:35 [qwen3coder_tool_parser.py:76] vLLM Successfully import tool parser Qwen3CoderToolParser !
(APIServer pid=99454) INFO:     127.0.0.1:32968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=99454) INFO:     127.0.0.1:32968 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=99454) INFO 11-03 17:36:35 [qwen3coder_tool_parser.py:76] vLLM Successfully import tool parser Qwen3CoderToolParser !
(APIServer pid=99454) INFO 11-03 17:36:36 [qwen3coder_tool_parser.py:76] vLLM Successfully import tool parser Qwen3CoderToolParser !
(APIServer pid=99454) INFO:     127.0.0.1:32968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=99454) INFO:     127.0.0.1:32968 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=99454) INFO 11-03 17:36:36 [qwen3coder_tool_parser.py:76] vLLM Successfully import tool parser Qwen3CoderToolParser !
(APIServer pid=99454) INFO 11-03 17:36:36 [qwen3coder_tool_parser.py:76] vLLM Successfully import tool parser Qwen3CoderToolParser !
(APIServer pid=99454) INFO:     127.0.0.1:32968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=99454) INFO:     127.0.0.1:32968 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=99454) INFO 11-03 17:36:36 [qwen3coder_tool_parser.py:76] vLLM Successfully import tool parser Qwen3CoderToolParser !
(APIServer pid=99454) INFO 11-03 17:36:43 [qwen3coder_tool_parser.py:76] vLLM Successfully import tool parser Qwen3CoderToolParser !
(APIServer pid=99454) INFO:     127.0.0.1:32968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=99454) INFO:     127.0.0.1:32968 - "POST /tokenize HTTP/1.1" 200 OK
(APIServer pid=99454) INFO 11-03 17:36:44 [qwen3coder_tool_parser.py:76] vLLM Successfully import tool parser Qwen3CoderToolParser !
(APIServer pid=99454) INFO 11-03 17:36:44 [loggers.py:127] Engine 000: Avg prompt throughput: 1684.4 tokens/s, Avg generation throughput: 96.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 83.4%

    Now that i have found this out ive switched back to vllm because the API i'm using with exlamav3 doesn't support qwen 3 tools yet :(

  • My 8gb vram system as i try to load GLM-4.6-Q0.00001_XXXS.gguf:

    Jump

  • I would suggest trying exllamav3 once, i have no idea what kind of black magic they use but its very memory efficient.

    i can't load Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8 with 16K using vllm

    but using exlamav3 i can SOMEHOW load ArtusDev/Qwen_Qwen3-Coder-30B-A3B-Instruct-EXL3:8.0bpw_H8 at its full context of 262.144 with still 2GiB to spare.

    I really feel like this is too good to be true and im doing something wrong but it just works so i don't know.

  • Relevance of GPU driver version for inference performance

    Jump

  • If I were to guess the biggest peformance impact is if your driver doesn't support newer types like FP4 and FP8 while your hardware does, but I am not sure.

  • The only way one should code C btw.

    Jump

  • Forget that the real evil is that the first character of Int or Ganz in this case is capitalized

  • LocalLLaMA @sh.itjust.works
    TheMightyCat @ani.social

    Very large amounts of gaming gpus vs AI gpus

  • Selfhosted @lemmy.world
    TheMightyCat @ani.social

    Very large amounts of gaming gpus vs AI gpus