Skip Navigation

InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)A

afk_strats

@ afk_strats @lemmy.world

Posts
4
Comments
161
Joined
3 yr. ago

  • My 8gb vram system as i try to load GLM-4.6-Q0.00001_XXXS.gguf:

    Jump

  • That fixed it.

    I am a fan of this quant cook. He often posts perplexity charts.

    https://huggingface.co/ubergarm

    All of his quants require ik_llama which works best with Nvidia CUDA but they can do a lot with RAM+vRAM or even hard drive + rams. I don't know if 8gb is enough for everything.

  • Am I the only one who is really impressed by Granite4 from IBM?

    Jump

  • You are not alone. It blew my mind at how good it is per billion parameters. As an example, I can't think of another model that will give you working code at 4B or less. I havent tried it on agentic tasks but that would be interesting

  • My 8gb vram system as i try to load GLM-4.6-Q0.00001_XXXS.gguf:

    Jump

  • Im not sure if it's a me issue but that's a static image. I figure you posted where they throw a brick into it.

    Also, if this post was serious, how does a highly quantitized model compare to something less quantitized but with fewer parameters? I haven't seen benchmarks other than perplexity which isn't a good measure of capability?

  • Meet Mico, Microsoft’s AI version of Clippy

    Jump

  • Rover back on XP

  • VST 3 now open source under MIT license

    Jump

  • Does this mean VSTs will have better Linux compatibility or is that implementation-dependent?

  • Deleted

    What's the weirdest hobby you've heard of?

    Jump

  • Sea legs [Danby Draws]

    Jump

  • Came here to post this.

  • Upside Down [OC Edit]

    Jump

  • Am I you?

  • Removed Deleted

    Permanently Deleted

    Jump

  • Accept that quality matters more than velocity. Ship slower, ship working. The cost of fixing production disasters dwarfs the cost of proper development.

    This has been a struggle my entire career. Sometimes, the company listens. Sometimes they don't. It's a worthwhile fight but it is a systemic problem caused by management and short-term profit-seeking over healthy business growth

  • Is Taylor Swift's New Album Bad Enough To Trigger Class Consciousness?

    Jump

  • This article has a unique-looking link and I couldn't find it anywhere on on the HuffPo site. Why?

  • What is your internet White-Whale?

    Jump

  • One time in the early 2000s, I saw a commercial during Adult Swim on Cartoons Network and I've wanted to see it again ever since. It was for Virgin Mobile and it featured Ode to Joy or 1812 Overture plating over quick cuts of mundane objects which turn out to be innuendo... for body parts. I know it wasn't delusion because I was at a friends house and also though it was epic.

  • Best LLMs for PC/Tech Troubleshooting?

    Jump

  • I don't know if this is still useful for you, but I tried this out, mostly because I wanted to make sure I wasn't crazy. Here's my gpt-oss setup running on cheap AMD Instinct VRAM:

     
        
./llama-server \
  --model {model}.gguf
  --alias "gpt-oss-120b-mxfp4" \
  --threads 16 \
  -fa on\
  --main-gpu 0 \
  --ctx-size 64000 \
  --n-cpu-moe 0 \
  --n-gpu-layers 999 \
  --temp 1.0 \
  -ub 1536 \
  -b 1536 \
  --min-p 0.0 \
  --top-p 1.0 \
  --top-k 0.0 \
  --jinja \
  --host 0.0.0.0 \
  --port 11343 \
  --chat-template-kwargs '{"reasoning_effort": "medium"}'

    I trimmed the content because it wasn't relevant but left roughly the shape of the replies to give a sense of the verbosity.

    Test 1: With default system message

    user prompt: how do i calculate softmax in python

     
        
What is softmax
1 python + numpy
...
quick demo
...
2 SciPy
...
...
...
8 full script
...
running the script
...
results
...
TL;DR
...

    followup prompt: how can i GPU-accelerate the function with torch

     
        
1 why pytorch is fast
...
...
**[Headers 2,3,4,5,6,7,8,9]**
...
...
TL;DR
...
Recap
...
Table Recap
...
Common pitfalls
...
Going beyond float32
...
10 Summary
...

    Overall 6393 Tokens including reasoning

    TEST 2 with this system prompt: You are a helpful coding assistant. Provide concise answers, to-the point answers. No fluff. Provide straightforward explanations when necessary. Do not add emoji and only provide tl;drs or summaries when asked.

    user prompt: how do i calculate softmax in python

     
        
Softmax calculation in Python
...
Key points
...

    followup prompt: how can i GPU-accelerate the function with torch

     
        
GPU‑accelerated Softmax with PyTorch
...
What the code does
...
Tips for larger workloads
...

    Overall 1103 Tokens including reasoning

  • Best LLMs for PC/Tech Troubleshooting?

    Jump

  • Totally. I think OSS is outright annoying with its verbosity. A system prompt will get around that

  • Best LLMs for PC/Tech Troubleshooting?

    Jump

  • Qwen 3 or Qwen 3 Coder? Qwen3 comes in a 235B, 30B and smaller sizes. Qwen 3 Coder comes in a 30B or 480B size.

    Open Router has multiple quant options and, for coding, I'd try to only use 8bit int or higher.

    Claude also has a ton of sizes and deployment options with different capabilities.

    As far as reasoning, the newest Deepseek V3.1 Terminus should be pretty good.

    Honestly, all of these models should be able to help you up to a certain level with docker. I would double check how you connect to open router, making sure your hyperparams are good, making sure thinking/reasoning is enabled. Maybe try duck.ai and see if the models there are matching up to whatever you're doing in open router.

    Finally, not being a hater, but LLMs are not intelligent. They cannot actually reason or think. They can probabilistically align with answers you want to see. Sometimes your issue might be too weird or new for them to be able to give you a good answer. Even today models will give you docker compose files with a version number at the top, a feature which has been deprecated for over a year.

    Edit: gpt-oss 120 should be cheap and capable enough. Available on duck.ai

  • Are there actually any good mobile games?

    Jump

  • 'Read' and its past tense are spelled the same. How should they be spelled?

    Jump

  • The Chaos by Gerard Nolst Trenité (1922)

    https://ncf.idallen.com/english.html

    Dearest creature in creation Studying English pronunciation, I will teach you in my verse Sounds like corpse, corps, horse and worse.

    I will keep you, Susy, busy, Make your head with heat grow dizzy; Tear in eye, your dress you'll tear; Queer, fair seer, hear my prayer.

    Pray, console your loving poet, Make my coat look new, dear, sew it! Just compare heart, hear and heard, Dies and diet, lord and word. ...

    Very long. Highly recommended

  • getting started in hard mode with very wimpy hardware

    Jump

  • Chiming in to say this is a very reasonable starting place and wanted to highlight to op that this solution is 100% self-hosted

  • Removed

    the past was so advanced

    Jump

  • Cetus-Lupeedus!

  • llama.cpp now has a Sveltekit based UI

    Jump

  • Yep. That was bleak.