SuspiciousCarrot78

FYI Lumo 2.0 just dropped and it seems to be comprised of open weight models (GLM 5.2 as the big model, Qwen 3.6 35B as the small, few others to boot)

https://proton.me/support/lumo-privacy

Having played around with it a bit...I dunno. It seems pretty locked down / security theatre heavy / refusals on benign things.

I can't run GLM locally (who can lol) but all things given I dunno how good value for money it is vs other options.

(Security and privacy promises notwithstanding that is - which perhaps, are the whole point with Lumo).

Has anyone used Lumo for any period of time? What quants do they use?

Do you host your own ML / AI / LLM? What do you use, and what do you use it for?

Based on recent comments this feels like a discussion we should have. So..topic, basically.

I'm not looking to be chief noisemaker on this, but I stand by what I wrote in !privacy and what's in my post history.

https://lemmy.ml/post/48724623/26190950

Let's have at; do we want a [AI] and [NOT AI] tag. Why or why not?

Re: the recent meta discussion and ongoing chats about self hosting, open vs closed source, AI etc, I wanted to share some food for thought.

I'll explain why this is related to self hosting at the bottom (section bolded): our lovely new mods can call it. I hope it inspires some out loud thinking.

Disclosure: I am not the content creator nor am I paid by them to signal boost. I just like their stuff and think this is an important topic, from multiple angles.

"The future of AI depends on the moral compass of five people."

I've been watching "AI in context" for a few weeks (they make long form biopic content on current state of AI - really good stuff).

This dropped today; it's about the wheeling and dealing behind closed doors at OpenAI re: Sam Altman's firing. It's a lot more watchable than that sounds :)

https://www.youtube.com/watch?v=_eYTkvZqbnQ

The line that brought me to a stand still was "the future of AI depends on the moral compass of like 5 people".

I think folks here (and Lemmy generally) are more savvy about AI then the gen pop (though Lemmy is famously FuckAI)....but even if you're training a nanoGPT model from scratch on hardware you own ...you're still beholden to outside forces.

Eg: the people's champion - Qwen - seems to have split or gone closed weights for 3.7. That's not a good sign.

Reason for post

In the recent [Meta] chat, I noted an undercurrent of "no man is an island" - that is, yes, you might host your own X, but you're still dependant on external Y (eg: SearXNG).

Self hosting / FOSS / forking mitigates some of the "we changed the terms of service after the sale" enshittification we see occurring in related spaces (eg: right to repair). But there's only so much leverage you can enact before it becomes pyrrhic.

I would like to believe "fuck you, I won't do what you told me" is our bulwark against market forces.

At the same time, it's sad to see so many "Don't be Evil" mission statements not survive contact with reality (watch the vid: OAi was founded on the ideal of "don't let AGI kill us")

I think what happens upstream has effects down stream too (see prior X vs Y examples)...

Not sure where this leaves us. It's a weird time to be alive.

Enjoy the video (and their others - Ai2027 is eye opening). Look forward to any productive chat this post might inspire.

Ive been watching "AI in context" for a few weeks (they make long form biopic content on current state of AI - really good stuff).

This dropped today; it's about the wheeling and dealing behind closed doors at OpenAI re: Sam Altman''s firing. It's a lot more watchable than that sounds :)

https://www.youtube.com/watch?v=_eYTkvZqbnQ

The line that brought me to a stand still was "the future of AI depends on the moral compass of like 5 people".

I know we're all about local LLMs here...but it's sad to see yet another "Don't Be Evil" mission statement get swallowed up market forces. OpenAI was meant to bring balance to the Force, not leave it in ruins, and yet....

I think folks here (and Lemmy generally) are more savvy about AI then the gen pop....but even if you're training a nanoGPT model from scratch on hardware you own ...you're still beholden to outside forces. Eg: the people's champion - Qwen - seems to have split or gone closed weights for 3.7.

Things that make you go hmm.

Anyway...just signal boosting a cool video.

I'm starting to develop arthritis in my fingers, which makes typing an interesting challenge.

I'm wondering, what is the best self-hosted solution for speech-to-text generally? Dragon dictate use to be the thing, but is there anything open source, self-hostable that's superseded it?

I would love to be able to have something that I can speak into that can interact with pretty much any app, be that notepad++, or my web browser when I'm entering stuff or even when I'm creating this Lemmy post (which I actually made using futo voice on my phone).

Windows and/or Linux ideally.

Any leads? Getting old sucks.

Recent post re: AI as utility

https://www.tomsguide.com/ai/people-will-buy-intelligence-from-us-on-a-meter-chatgpts-ceo-sam-altman-has-critics-worried-with-his-ai-vision

Myself, I'm a fan of local LLM / self hosted ML.... but if you ever needed a clarion call that a hard pivot is coming (soon) for online/ cloud based AI...Altman et al are making some concerning mouth noises (to say nothing of broader concerns with OAI, Anthropic etc).

Right now, I'm sketching out a plan where my Raspberry Pi (always on, 2-3w) uses a magic packet to wake up my modest AI server (Lenovo P330 with Tesla P4) if/when needed (Qwen 3.6-35B-A3B); no point in chugging down 80-100w, 24/7 for no good reason.

If the trend continues the direction it appears to be (increasing costs, environmental impacts etc) then I'd feel a lot better hosting my own as port of first call and replacing simpler tasks with more traditional programs. YMMV.

More often than not, AI and LLM gets conflated in the public consciousness...and then gets mixed with "Agentic", "SaaS" and other well...slop. So, here is a farmer in Japan, using a raspberry pi, to sort cucumbers.

https://www.newsweek.com/artificial-intelligence-cucumber-farm-raspberry-pi-495289

PS: 2016 article. I expect by now the tractor is self driving and named Betty.

If you have any other "dude does cool AI shit with a box of scraps in a cave", I'm all EARS.md

I was browsing Reddit (yetch) while waiting for some stuff to finish when I came across this post

https://old.reddit.com/r/LocalLLM/comments/1tek00h/why_is_llm_is_so_expensive/

The author make a (very) interesting claim: if table stakes are $6K (they're not...but go with it for now), then most folks are cooked from the get go.

Personally, I have been figuring out how to get more from less. For example, people have found ways to run Qwen3.6 35B on a 6GB VRAM GTX 1060 at ~20tok/s (--ctx 64K IIRC, but go check the vids yourself)

https://youtu.be/8F_5pdcD3HY

I think there's a lot of juice to squeeze by turning LLMs from "all seeing sages" into basically mouth pieces for shit that actually runs fast on regular silicon - but that's just me and my crazy brain. YMMV.

https://mikeveerman.github.io/tokenspeed/?rate=20&mode=agent&think=15

Exactly what it says on the tin :)

Pretty good simulator this. May it cause you to reconsider your expensive GPU upgrade :)