Interests: programming, video games, anime, music composition
I used to be on kbin as e0qdk@kbin.social before it broke down.
I finally got around to trying this model today. My initial impression is that it's ok conversationally and decently fast (even at Q8), but unfortunately it repeatedly fails when I ask it to do tasks that require using multiple tools in sequence -- like listing files and then reading through them to explore a codebase. (Qwen3.6 and Gemma4 both manage that fine usually.) In some of the failures, I was seeing it get stuck in loops with bad thinking tags.
I haven't pushed it hard on non-tool tasks yet though. If it turns out to be particular good at that I might try using it as a fast code generator driven by another model maybe...
If you just pulled the default version of qwen3.5 from ollama's repo you downloaded a mediocre one that only uses ~6GB.
Check
ollama show qwen3.5and see if you get something like this in the result:Model architecture qwen35 parameters 9.7B context length 262144 embedding length 4096 quantization Q4_K_MThis is the default version I got when I first tried using ollama without any experience. It worked, but it's a heavily quantized, lower parameter version of the model -- i.e. it's pretty dumb -- compared to what you can actually run on your hardware.
I started running LLMs a couple months ago on my own hardware. I have a Framework Desktop that I ordered last year and also recently picked up a refurbished 24GB AMD RX 7900 XTX which I'm doing some performance testing against. The dGPU is much better for dense models, and slightly faster for MoE if I'm willing to run them at a lower quant -- but uses more power and has annoying coil whine. The Framework Desktop uses ~100W under load, is quieter, and for the MoE models already runs them fast enough for most of my needs -- so most of my LLM use happens on that system still.
For software: I'm using ollama on the Framework currently, but I want to replace it with just using llama.cpp directly eventually. I've been using llama-cli for testing the dGPU. I wrote my own chat client to interact with ollama as well as a few other programs for specific tasks.
I've been using the LLMs for a mix of research (both personal and professional), entertainment, practical coding tasks (mostly debugging and brainstorming, plus a bit of UI prototyping, automatic generation of sequence diagrams for documentation, and light scripting), as well as automation of tedious tasks.
As an example of the latter, people often send me requests to prepare data sets by email but don't specify the sources they want precisely so I have to go match the name against the real name in our archives; LLMs are great for mapping the imperfect name -- with typos, missing prefixes, incorrect addition of spaces, addition/removal of hyphens, etc. -- to the exact name I actually need to pull the data off disk when given a lookup table to compare against.
As far as models go, I'm mostly using various Qwen 3.6 and Gemma4 variants. I have multiple versions of each for different purposes. llmfan46's uncensored Qwen 3.6 35B-A3B @ Q6_K (from Hugging Face) is my default model currently.
More like a conceptual search. e.g. I've used my source code explorer to get a survey of how the template handling works in llama.cpp since the sample chat code doesn't apply the same logic that llama-cli actually does.
What the LLM can do that grep can't is that it can find things by imperfect description. You need to know a text string that's exactly in the file to get grep/ack/etc. to locate it; you can be vague with an LLM and it may still be able to figure it out. It's the difference between searching for
FooBarFactoryalready knowing the exact name and trying to find the file it's in (where grep, etc. are great) and "find the code that instantiates FooBar objects infooproject and tell me what it's called" when you don't know if it was FooBarManager or FooBarFactory or it's actually a function calledmake_foo_bar()instead of factory class or there are actually three different ways to do it because of legacy code.Are you thinking to make something like a
quote_snippettool that you give a file and line range to and have it (deterministically) present that to the user as part of the response?I implemented a system for exploring my own source code via tools a while back. I have
find_files,read_source_codeand a few others that allow putting in a project name and/or filename as parameters (restricted heavily based on permissions I've set in my custom harness). It's been pretty good at following tasks like "Read the source code in thefooproject and update the documentation insuch-and-such.md" -- which I have mermaid.js sequence diagrams embedded into. (I don't give it direct file write access; it just gives my output in my chat client and I copy over what it spits out and diff against what was in git then tweak if needed.)Quoting directly from the code works well with no particular special effort. It is absolutely terrible at giving line numbers though (hallucinates everything when it tries to do that). I have a few ideas on how I might be able to improve that -- the most straightforward is to just inject comments with the line number into the return from the tool call (so that it can quote the number instead of trying to estimate position). If that's not good enough, I've also got an AST-based source code reader (only for JS and Python though) that can return line numbers, It was intended for skeletonizing code so that I could throw larger files at an LLM without it having to read the entire thing and then just pull chunks out with
read_source_codebased on line number ranges -- but it hasn't been particularly effective at making good use of that capability. Maybe that concept could be repurposed for quoting code to the user though if the simpler approaches aren't good enough... 🤔️TL;DR: This is relevant to my interests and I might build my own too!
It's what I grew up with. Tastes like childhood. 🤷️
- Toast bread in pan with butter
- Put a slice of American cheese on one piece of the toast
- Add a strip or two of (already cooked) bacon if I have some on hand
- Fry egg in the pan, deliberately breaking the yolk and cooking hard
- Put egg on top of bacon and cheese.
- Salt, pepper, squirt of ketchup
- Close sandwich and microwave briefly to get the cheese melty
That's the most common variation I do when I want a fried egg sandwich. Sometimes I'll skip the ketchup if I'm not in the mood for it though, or mix the ketchup with hot sauce if I want something with a bit of kick instead.
I'd suggest replacing the shell script with a Python script as a first step. If nothing else, it will probably be a hell of a lot easier to read... Your bots ought to be able to do a first pass at that. If it's actually tripping up on some particular bash/dash-ism like you think, that might even just bypass the issue entirely; if not it should be easier to debug with better readability.
I mean, if you want to retroactively classify the PornHub jingle as "cerebral catwave", I'm not gonna stop you... 🤔️
I've got an AMD system so that probably won't work for me, but glad it's working for you and maybe it will help others!
How does the model compare to Qwen and Gemma4 so far?
cerebral catwave
That really sounds like it could plausibly be some niche internet microgenre of music even if you pulled it out of your butt, so maybe they just rolled with it?
(If it isn't one yet, someone will probably make it one now that you've posted this... 🙃️)
Interesting. Looks like I'd need to build a special llama.cpp to get it to run on my system currently, and I think I could get lost for a long time if I start digging up that rabbit hole... so maybe not today, but I'll keep an eye out and give it a try if support lands in main.
Is it doing any better than Qwen at avoiding getting stuck in thinking loops?
Could you recommend the best AI's to use on mobile and on free tier to write ?
I only use open weight models on my own hardware, so I can't give recommendations there from personal experience. If you want suggestions on open models though, there are uncensored Qwen and Gemma4 models that are pretty decent when run at higher quants. I use llmfan46's
Qwen3.6-35B-A3B-uncensored-heretic-GGUFas my default and I've just downloadedgemma-4-26B-A4B-it-ultra-uncensored-heretic-GGUFthis past weekend to experiment with. (Gemma4 stock Q4_K_M has usually been annoyingly thick-headed compared to Qwen so I haven't used Gemma4 as much, but the Q8 uncensored is pretty sharp, fast on my hardware, and unlike Qwen doesn't get stuck in loops constantly, so I may switch over to that as my default... Need to evaluate more first though.)The sidebar gives a link-to-a-link to https://aihorde.net/ though -- haven't used it personally, but might be up your alley.
Looking at your writing guide for the LLM, I think my advice about changing workflow was right.
Break the problem down into much smaller steps and then have the LLM do editing passes and self-review from multiple perspectives after getting a crappy rough draft back instead of trying to get it to match your style one-shot.
Using something like this to provide context for the scene when generating a rough draft will likely be helpful:
[SCENE STATE] - POV: [Name] - Location/Time: - Known to POV: - Unresolved/Secrets: - Active Props/Details: - Goal/Conflict:Also, are you dumping in the full PDF of the entire work so far? Try doing a pass to skeletonize it down into an outline -- maybe chapter by chapter or even scene by scene -- along with getting organized about characters by making profile summaries so that the LLM can understand your story in an already partially digested form instead of trying to do everything all together at once with the huge amount of context reading a full story implies. It's probably getting lost in the weeds.
You may need to be very specific about how you prompt to get decent results in a particular style. Any wiggle room for ambiguity and the bots'll drive a bulldozer through what you meant with insane interpretations.
I don't use cloud based LLMs at all -- local only -- so I'm not sure what they give you to work with there, but in my own usage, setting the system prompt with clearly defined ROLE, CONSTRAINTS, and other other details has been critical for getting them to do anything useful instead of spitting out generic analysis slop. (Even so, be mindful of context limits.)
LLMs aren't magic and don't have infinite memory. In fact, the amount they can hold in their heads effectively is pretty small and you must manage context carefully if you don't want them to get lost going down irrelevant paths.
What they are very good at is style transfer. If you have text and need it reworded, they can do that trivially.
They are not very good at dealing with large structural reasoning. You really have to break things down and hand-hold them if you want to get them to do stuff like that.
I haven't done much directly with creative writing in natural language (I've mostly done code gen, critical news analysis, image analysis, and a lot of context experimentation) -- but the path I'd try out if I wanted to get into that is actually using a lot of writing "self-help" strategies. e.g. Snowflake Method. Lots of outlines. Character profile drafting. Scene and sequel. Apply Vonnegut's rules for writing. Etc. Then set up multiple LLMs as test readers tuned to different aspects of the writing that you want -- like character consistency, diction, appeal to your audience, etc. and have them review it and iterate.
I'm not familiar with the tool you're using and my big, traditionally written projects have largely been organically grown over the years as unpredictable requirements change -- mostly I just try to keep things readable enough that I can jump back in when I need to do so on old projects...
It sounds like you want to incorporate LLMs into brainstorming though. Maybe try using the word "elicitation" in your prompts for early phases? e.g. "ROLE: Requirements Elicitation Specialist" or "ROLE: Architecture Design Elicitation Specialist". You can also meta-prompt to get a system prompt tailored to your specific needs.
Imagine some future PugJesus type excavating our memes from the archives and reposting this 1000 years from now. 🧐️
Have you tried benchmarking prefill/prompt processing with large context? Usually that's more compute-limited so your improvements may show up better there. (Decode is bandwidth limited as you realized.)