oh and I'm running Ollama/OpenWebUI
this post was submitted on 21 Aug 2025
1 points (100.0% liked)
LocalLLaMA
4926 readers
2 users here now
Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.
Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.
As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.
Rules:
Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.
Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.
Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.
Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.
founded 3 years ago
MODERATORS
What are you using it for? Coding? How big context do you need?
some coding yeah but also want one that's just good 'general purpose' chat.
Not sure how much context... from what I've heard models kinda break down at super large context anyway? Though I'd love to have as large of a functional context as possible. I guess it's somewhat a tradeoff in ram usage as the context all gets loaded into memory?
With 128GB of ram on a Mac, GLM 4.5 Air is going to be one of your best options. You could run it anywhere from Q5 to Q8 depending on how you wanna manage your speed to quality ratio.
I have a different system that likely runs it slower than yours will, and I get 5 T/s generation which is just about the speed I read at. (Using q8)
I do hear that ollama may be having issues with that model though, so you may have to wait for an update to it.
I use llamacpp and llama-swap with openwebui, so if you want any tips on switching over I'd be happy to help. Llamacpp is usually one of the first projects to start supporting new models when they come out.
Edit: just reread your post. I was thinking it was a newer Mac lol. This may be a slow model for you, but I do think it'll be one of the best your can run.
oh I didn't realize I could use llamacpp with openwebui. I recall reading something about how ollama was somehow becoming less FOSS so I'm inclined to use llamacpp. Plus I want to be able to more easily use sharded ggufs. You have a guide for setting up llamacpp with openwebui?
I somehow hadn't heard of GLM 4.5 Air, I'll take a look thanks!
update: I tried GLM4.5 air and it was awesome until I remembered how censored it is by the Chinese government. Which I guess is fine if I'm just coding but just on principal I didn't like running a model that will refuse to talk about things China doesn't like. I tried Dolphin-Mistral-24B which will answer anything but isn't particularly smart.
So I'm trying out gpt-oss-120b which was running at an amazing 5.21t/s but the reasoning output was broken and it seems the way to fix it was to switch from the llamacpp python wrapper to pure llamacpp
...which I did, and it fixed the reasoning output... but now I only get .61t/s :|
anyway, I'm on my journey :) thanks y'all
? they are just trying to protect themselves from western propaganda, just as all nations should. Its working great. Everything you've heard about China comes from the western propaganda apparatus, and I doubt you have discovered how insanely polluted the western information sphere is - amongst other things, propaganda towards enemies of the US Plutocracy. All models are trained on that nonsense, and you can't say "Hi" to a western model without being influenced by western ideological pollution/propaganda..