Selfhosted

60026 readers

1260 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam.
Posts here are to be centered around self-hosting. Please ensure it is clear in your post how it relates to self-hosting.
Don't duplicate the full text of your blog or git here. Just post the link for folks to click.
Submission headline should match the article title.
No trolling.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago

MODERATORS

curbstickle@anarchist.nexus

curbstickle_lw@lemmy.world

Hardware for local inference? (lemmy.ca)

submitted 3 weeks ago by droopy4096@lemmy.ca to c/selfhosted@lemmy.world

27 comments fedilink hide all child comments

I want to host some LLM's locally and use more advanced models. Since new hardware is out of the question, I think I should be able to pull something off buying some yesteryear equipment on ebay etc. Did anybody attempt such a project? Does it scale horizontally? (I.e. can I connext two boxes to overcome single box slowness?)

you are viewing a single comment's thread
view the rest of the comments

[–] iceberg314@slrpnk.net 3 points 3 weeks ago

For really lightweight models, Qwen3:8b is pretty good for an 8gb graphics card.

Gemma 4:e4b is pretty good too. I usually sun that on my 16gb gpu.

Obviously the little ones aren't as good as big ones, but you can always rely on real intelligence to fill in the gaps