Skip Navigation

InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)S

SuspiciousCarrot78

@ SuspiciousCarrot78 @aussie.zone

Posts
19
Comments
326
Joined
3 mo. ago

  • Some general questions on AI setup

    Jump

  • Understood, so my iGPU is too old, and i am better off with my NVIDIA only.

    Sadly, yes

    Also, putting two might be just a waste

    It...depends. What are you planning on doing with em?

    i doubt my PSU can handle them even if i had the proper PCIe slot.

    Technically, PSU and PCIe slots can be shared or upgraded....which goes back to "it depends". But if your gut is already telling you nah...

    So, using llama.cpp if i use a model bigger than my VRAM will offload to CPU?

    Yep. Technically, the other option is better (tell it to auto adjust / put as much as it can on GPU and then spill over onto CPU / RAM) The setting for that should be something like -ngl auto (in the latest versions of llama.cpp)

    The NVIDIA RTX A4000 is nothing to sneeze at, 16GB or not - it's just...old, so support for it is middling. You should definitely try one of the MoE models on that. See if this helps:

    https://www.youtube.com/watch?v=8F_5pdcD3HY

    The setting he uses are in his first comment.

    If I had to pull a number out of my butt, you should be able to triple his thru put (given that the A4000 is almost 3x the bandwidth, has actual tensor cores etc)

  • Some general questions on AI setup

    Jump

  • Incidentally, I can tell you that Qwen 3.6 35B-A3B, at Q4_K_M, --ctx 8192, runs at 6-8 tok/s on that CPU…because I tested it myself (llama.cpp, beellama and ik_llama), with llama.cpp being fastest overall.

    Heh...digging around my .bat files, I found the launch params for that. Fucking ouch - 

     
        
cd /d "%LLAMACUDA_DIR%"
start "Qwen3.6-35B-A3B-UD-Q4_K_M" /min llama-server.exe ^
  -m "%MODEL_PATH%" ^
  -ngl 0 ^
  --n-cpu-moe 35 ^
  --no-mmap ^
  --mlock ^
  --cache-type-k q4_0 ^
  --cache-type-v q4_0 ^
  -c 8192 ^
  --host 0.0.0.0 ^
  --port %PORT%

  • Some general questions on AI setup

    Jump
    1. No, not really. For a start, that CPU / Igpu could at best have Vulkan support...except the i7-8700 is AXV2...which is about 4 gen out of date, so have dicey vulkan support in Windows / a little better in linux.. Ask me how I know.

    But even if you could, it would be slower than a real GPU for AI compute due to lack of tensors in the Igpu. You'd probably (definitely?) be better off just running straight on CPU in that case. And this is before even talking about different OS issues in windows / Linux...

    Save yourself the pain; don't try to force inference via Igpu; you're gonna have a bad time.

    1. It'd be dynamically allocated (generally speaking)...but see 1.
    2. Sadly, it won't help so much as you think...because you are memory bandwidth bound. Though I suppose it does allow you to run larger models (slowly) on CPU...if you hate yourself :)

    Incidentally, I can tell you that Qwen 3.6 35B-A3B, at Q4_K_M, --ctx 8192, runs at 6-8 tok/s on that CPU...because I tested it myself (llama.cpp, beellama and ik_llama), with llama.cpp being fastest overall.

    If you already have a 16GB, DDR6 GPU, find something that plays nicely with it.

    If you're asking about spill over or offloading a MoE, that should be automatic / with launch flags.

    1. The very easy answer to this is "anything by Qwen", because they are the community darlings. But...is your use case specifically AI radio? Because that maybe speaks to something like ACE-Step 1.5 (...which is a Qwen 3 derivative)
    2. Maybe ... but you'll then have to deal with split inference across 2 GPU (or I suppose run different things on different GPUs). Possible but...you'll have to look into orchestration.
    3. They do, but much less than at peak. For example, if your GPU sucks down 250w at peak, it might idle at 100w. So if you have 2 GPUs + everything else, it might idle at 300w and peak near 650w. I suppose that's OK if you're not hammering it 24/7

  • PlayStation fans brutally reject Sony's disc decision, eye Windows 11 PCs over Xbox

    Jump

  • Reddit will require you to log in to use old.reddit.com

    Jump

  • Selfhosted & AI - Part 2: The Results

    Jump

  • Hmm. Not sure I agree with "just mark the parts that are AI generated" because that obfuscates the parts that were human made, skewing perception towards "it's all AI gen".

    Require the full accounting - human, clanker, level.

    • Design - Human
    • Implementation - Pair
    • Testing - Assist
    • Documentation - Human
    • Review - Human
    • Deployment - Human

    Reads differently to

    • Implementation - Pair
    • Testing - Assist

    4/6 human vs ?? / Human is a different trust signal (which is what this is actually about, right?)

    PS: I'm a fan of acronyms, so how about "show us the STACK or show us the DIRTY"

    • Spec (Design)
    • Testing
    • Assembly (Implementation)
    • Checks (Review)
    • Knowledge (Documentation)

    Or

    • Design
    • Implementation
    • Review
    • Testing
    • Yeet (deployment)

  • PlayStation 6 bill of materials nears $1,000 as RAM shortages worsen

    Jump

  • Those are rookie numbers. They need to pump that up. $10K a console, and you need to scan your retina to use it.

    Ya know... for a luxury good with literally dozens of other, better options... "Make it impossible to buy our stuff" is a dumb hill to die on.

    Oh well.

    laughs in 40 years of non PlayStation games

  • Bring Back Crappy Forums: Web forums were rough around the edges and faded in relevance as seemingly better options emerged. But what if we had stuck with them?

    Jump

  • You could not live with your failure. And where did that bring you? Back to (Lem)my.

  • Lumo 2.0 is out

    Jump

  • I use proton for email and cloud drive (non critical files obvs), so I'm playing around with this a bit out of curiosity (and because I can't run 27B or GLM at speed locally).

    For basic, throw away tasks (ala super Google search, fact check what I writing, edit this email etc), it seems to work OK. Worth dicking around with while building out own infra.

    One interesting thing (above ZDR) that I've noticed is that the actual prompts seem to be obfuscated cryptogenically. Can't say I've seen that before. If it's truly E2E, that's something.

    Given that GLM is the brains of the operation, I'm willing to poke around with Proton a bit as an alternative to OAI and Anthropic for simple tasks. I'm suspicious of their claims but willing to give them a fair shake and do my own due diligence.

    On that topic, some discussions uncovered

    https://discuss.privacyguides.net/t/proton-markets-lumo-as-open-source-but-support-calls-it-a-long-term-intention-not-the-instantaneous-state/30004/2

    https://forum.qubes-os.org/t/lumo-protons-ai-assistant/35373

    EDIT: scuttle butt has is that Lumo 2.0 Max (which is GLM 5.2 undoubtedly) has --ctx 128K. They also have compaction ; I hit auto compaction today at 68K, which is somewhat conservative.

    Still no idea on quant etc but it does seem that answers a few questions - lite is most likely Qwen 3.5 27B thinking, Max is GLM5.2, context is 128K. I haven't noticed any of the famous Qwen thinking loops yet; perhaps that explains why compaction is set to 50% total ctx.

    Total usage / daily requests are still obfuscated and quants not known. The android app seems to use a very poor version of Vosk for STT (basically unusable). Image gen and OCR are decent.

    In a perfect world, preference would be given to running GLM and Qwen 3.5 at home and then accessing via wireguard or head scale. This may be an interesting middle ground for (we) GPU peasants.

    Not sure where I sit with this; am documenting what I find for others / not endorsement of proton (I have no relationship to them).

    Given that most of us use cloud AI also, this might be worth considerations.

    NB: there appears to be a Lumo plug in for VS codium; I have no indication as to if there are separate usage pools for chat and code (like ChatGPT) or all roled in one (Claude Pro). Probably the latter.

  • Price hikes pushing consumers to abandon their favorite brands

    Jump
    • Brands increase prices
    • People stop buying brands
    • Brands cry foul
    • Oh no! Anyway....

    The trick, William Potter, is to bleed the people just enough to satiate your parasitism without exsanguinating them, eh?

    Gee, we've never seen that trick before.

  • Reddit will require you to log in to use old.reddit.com

    Jump

  • Does shreddit not work any more?

  • Reddit will require you to log in to use old.reddit.com

    Jump

  • Ah but redreader parses old.reddit.com. So with the proposed changes, you will probably need an account to use even the excellent RedReader.

    IIRC, when the APIpocolypse happened, Reddit tried really hard to nerf Redreader. They got shamed into allowing it access again on accessibility grounds.

    It's only a matter of time before it stops working.

    Take solace in knowing that "lol, fuck reddit then" is a perfectly cromulent response to this.

    I just hope Lemmy doesn't become Reddit 2.0

  • PlayStation will no longer be offering games on disc

    Jump

  • Oh no! Anyway...

    EDIT: Just to be clear, I am not dismissing OP. I'm saying "fuck Sony" and "this would be a good time to convert to PCmasterrace or go retro"

  • Lumo 2.0 is out

    Jump

  • I have a strong feeling that lite is Qwen 3.5 27B and Max is GLM 5.2, based on the AAII scores on protons announcement post. 27B thinking is closest to 34 AAII and only GLM hits 51. So as fingerprints go....

    But the fact that Lumo won't reveal specifics / inhibits that at system prompt level (despite broad family id being cooked into weights) sort of belies the transparency angle for me. What else is being obfuscated? I can't even uncover what fair use is - how many messages per hour etc

    Logical inference then - opacity is driven by profit margin preservation and competitive insulation. If you can't calculate the mark up by query (vs OR) then you might be paying 5-20x mark up.

    Sits funny amidst all the privacy and transparency cosplay.

    At least they're not building autonomous weapons but treating open weight models as commercial secrets is sorta off putting.

  • Storage?! In this economy!?

    Jump

  • Ive seen LTO for sale in person at the $300 USD mark...but I admit that's a rare occurrence / fire sale. So that "as little as" is probably not fair. Sorry and retracted.

    EBay shows some in the $1200-1500 USD range (and maybe closer to $3-4K brand new).

    That now makes me feel stupid for walking past one at $300 ("tape drive? Who the fuck needs that ancient shit") but I'm willing to bet that wasn't a lto-8, in hindsight.

    Even my LTO-3 claim is not as remembered; I can find a lto-3 for $60 USD here locally (not the $5 I jokingly claimed), with cartridges in the $15 USD range.

    https://ebay.io/m/1UST3Y

    https://ebay.io/m/Gcf77T

    Not bad, but not 12TB per cartridge.

    All of this to say; the DVD shuffler + pi intermediary (+ NVIDIA shield if needed) is probably the genuinely better version of this. Bizarrely.

  • Commodore listened, and the Callback 8020 is now $100 cheaper

    Jump

  • Huh. Thought it was stock standard AOSP - perhaps the Aussie version is different? There are a few rebranded versions of the same hardware; you might be able to find something non proprietary. I think the underlying model is UNIWA if you want to go spelunk direct listings

    https://opelmobile.com.au/wp-content/uploads/2023/08/OM-TouchFlip-A4.pdf

    Where did you find the info that the TTFone uses a fork - XDA? Possibly it can take CFW?

    In any case, if it's no go, it's no go.

    PS: The other target might be a the Cat 22 flip but that thing has a face only a mother could love. I have seen clean CFW of the Duoquin models too - multiple threads on XDA - but that's candybar not flip

    PPS: let me spelunk the 8020 for a minute. I have to imagine it's an off the shelf re-badge. EDIT: Hmm...looks like it's bespoke enough to NOT be a simple Shenzhen rebadge job.

  • Commodore listened, and the Callback 8020 is now $100 cheaper

    Jump

  • Yes - I have the rebranded Aussie version

    https://www.officeworks.com.au/shop/officeworks/p/opel-mobile-touchflip-4g-flip-phone-optouchfp

    It runs Android 8.1 and had no issue running Molly the last time I tried it. I really like that little nugget - let me know if you want a list of apps or launchers for it. Note: you'll have to use Fdroid, Droidify, Aurora store or direct apks, as it doesn't meet play store compliance.

    If the TT990 is android 14, then it should work even better.

    I can also confirm that my Duoqin F21 runs either just fine, but that's cheating

    https://qinphone.com/products/qin-f21-pro-smart-keypad-phone-compact-2-8-inch-touchscreen-android-11-4g-lte-single-camera-google-play-support-ideal-backup-work-phone-porcelain-white-iron-grey

    Back to the TT970; the keyboard is fantastic (download the true TT9 app) and it even runs futo voice (albeit a touch slowly). If you get one, try to get the 1750mah battery - it helps.

    Standby is just bang on 2 days for me.

  • Commodore listened, and the Callback 8020 is now $100 cheaper

    Jump

  • Storage?! In this economy!?

    Jump

  • Some people just want to watch the world burn