Selfhosted

I self-host my own neural network, all within the confines of my own skull.

I'm self-hosting immich (with ML-based facial recognition, semantic search and OCR) and psiTTSa, an offline text-to-speech engine.

(Prob not the answer you were looking for, but a reminder that "AI" is more than just LLMs.)

• Qwen3.6 35b a3b Q4_K_M running on llama.cpp
• Open-WebUI as a frontend
• integrations with an embedding model and searxng & crawl4ai
• doing lots of different stuff with it, asking basic stupid questions, recipes, tech stuff, etc

Running Qwen 3.8 27B dense on some R9700s, and Qwen 3.8 flash next on dgx sparks. I use it for code gen outside work, including an automated Dev pipeline that rubs on its own. When I did the math comparing to token API pricing of Opus 4.8, it covered its own costs in a few months. With the sparks I get about 40tg/s, 1500pp/s, but can get concurrency up to 5 stably. With the R9700s, I get ~100tg/s, 3000pp/s and can also get up to 5 concurrency.

I access it over wireguard to the network that it is on, and use termux to ssh into a build machine for dev over CLI using https://pi.dev/

I also started building https://fcli.dev/ to make integrations easier; the idea being that LLMs should just implement simple reusable functions instead of repeatedly making complex curl requests to integrate with a service.

I also use LLMs heavily for work; Qwen 3.8 seems almost on par to Opus 4.8 IMO. I also don't have to worry about token costs, and it often responds faster than Claude would because there usually isn't a queue on my machines. At this point, I wouldn't complain if my work provided me 2x B6000 instead of a Claude subscription; I'd still be able to get all my work done.

What a timely comment.

I recently gave my kid a gaming PC for his birthday...which I used as an excuse to upgrade my GPU from a 1660 Super to a 5060ti 16GB

Combined with a 5600x3d and 32gb of system memory.

I just started messing with local LLMs yesterday. "Getting started" is rather easy. pacman -S lm-studio (I use arch, btw), launch it, download a model that your hardware supports, and start talking.

Right now I'm working on getting it to talk to vscode.

Not only do I run and share it. I've built an entire service to allow everyone else to do so safely. https://aihorde.net/

Yes, were all on c/LocalLLaMa

I'm running qwen3.8-next-flash on a rx7900xtx plus 128gb of ram. It serves my hermes agent. I direct it through matrix. I also forward an endpoint to my VPS to use for anything. Mostly opencode.

I wish I could. I don't have the hardware for it.

We did the maths. Didn't seem worthwhile and the payback period was a very long time

24GB GPU

Qwen 3.8 ( 2 variants )

Pangolin>OpenWebUI>lama.cpp for general use and available outside the homelab

Zoo Code>lama.cpp for coding , mostly scripts and html

Qwen 3.8 was a game changer for me.

Yes i do.

I esperiment a lot, but my stable setup is Qwen 3.8 27B Q6_K on my home server. Hardware is Ryzen 9 5950x with 64gb ddr4 and two old but effective NVIDIA RTX A4000 for a total of 32gb VRAM.

Qwen 3.8 is the best model I can run in terms of output quality. It can perform all tasks pretty well and it's reasonably fast at 500ts prefill and 30/40ts processing. There are much faster models but honestly not as good.

I can also run gemma4 all on CPU, with no GPU at all.

The key point is having both GPU on pci-e CPU, not on chip set bridge, so I am using a PCI riser 8x8x to share the 16x slot between the two GPUs. Inference speed is unaffected, but prefill benefits immensely.

Software side, use llama.cpp on Linux, you get the best performance possible at the expense of a bit more complexity compared to other engines.

From a power consumption perspective, I tined down my CPU governor and maximum frequency, and also the GPUs maximum watts. At ide the system is around 80w. At peak, it's about 280w with the limiters, and inference speed doesn't suffer too much. Without limiters I can easily reach 650w.

Edit: I run llama-swap in front of llama-cpp, and openwebui as agent manager/ web GUI. I also have ComfyUI installed because why not but seldomly use it.

You can read more of my setup on my wiki https://wiki.gardiol.org/8-llm-and-ai (no ads, no monetization whatsoever because fuck a hobby should not be a side gig)

Not me. Call me old-fashioned, but I prefer organic.

Yes! I’ve got quite a few projects going right now. Probably too many of them to keep track

I am the only user, so it’s relatively easy to run multiple models for what ever task. Ollama is pretty good at balancing which models are working at what time to not bog down they system

I’m running gpt-oss 120 for a general purpose Hermes agent. Out of the box, it was ok. Not very snappy, but it was one of the best ones for orchestrating other agents.

Glm flash (4.something?) does pretty good as well.

I’ve found a hybrid of cloud and local is the best solution. Use local for smaller or more defined tasks. Use cloud for orchestrating, planning, and…fixing the local agents mistakes lol

I’ve got an LLM maintained Obsidian wiki that is written and edited by a few Ollama agents based on chats via my own self hosted matrix instance. I’ve taken it to a point where I’m pretty happy with the results. But I think if I had enough time and energy, I could really dial it in.

One of my favorite use cases for this is when I’m researching topics like UFOs/high strangeness/conspiracy theories. I have my very own LLM maintained Obsidian vault filled with curated notes and links to sources.

Literally, no one cares about this and they think it’s stupid and I’m stupid and I should feel bad for spending time on this or having the audacity of sharing it… they just don’t appreciate the amount of work and knowledge it took to get what I have set up.

I think it’s cool as fuck. I geek out every time that it works.

Oh, forgot to mention I am running it on a Mac studio with 96 gig of unified memory. I think it’s an M3 ultra chip.

I use Qwen 3.8 27b Q4 with 100k context (llama.cpp and the Mistral AI harness) on a 24 GB AMD Radeon 7900 XTX and it's been great. It can handle pretty much anything I throw at it and I can let it work indepedently overnight / over days on long tasks.

I’d like to, as I currently drop $20/month on Claude without perhaps using it to the extent that my money would let me, but what’s “good” that can run on a 16GB RAM, i5 12th gen, 3070-equipped desktop?

I have a Qwen model running on a Mac Mini with Ollama - it’s reasonable for Home Assistant, responding to iMessages and similar.

What does it do? I thought about trying this but I literally never have a need to generate text.

Yes, I installed an LLM at home. I use it for many useful things:

1. reading novels by translating them from English to my language on the fly via machine
2. for communicating on lemmy by translating posts and comments
3. For HomeAssistant
4. And very rarely for development and searching the internet

I have a RX9070XT and llama-swap to switch models on the fly. I'm thinking about installing a second RX9070XT so that I don't take away the translation capability from my lemmy instance while I stress the other neural network with other tasks.

Yes, I have a desktop with a radeon 9700 ai pro with 32GB of VRAM. Its more than doubled since I bought it but its basically a down clocked 9070xt with double the VRAM. I also have a little arc b50 in my server.

Mostly yeah use it for privacy reasons. I run llama.CPP directly and access via open webui. I host a searx instance for it to use as web search. Mostly run qwen3.6/3.8 or gemma4 on the radeon and almost exclusively gptoss 20b on the B50 along with a second small model for tags and titles in open webui. Yeah I just use it mostly as a basic chatbot or as an agent in obsidian using the engram plugin. Helps search and summarize years of notes and works pretty well.

I've tried to use it as a coding assistant in zed without much luck so usually just post my code into open webui for feedback and it works alot better for me.

Yeah I can access it with other machines on my network including phone with conduit app. I can get to it on the go by connecting to home network with wireguard and sending a magic packet to wakeup desktop.

I havent done any image gen nor have much use for it outside of maybe using it to make app icons in the style of papirus icons for programs they don't and won't support.

It definitely feels more comprable to first ai wave than current frontier models but the privacy and price trade off is worth it to me. I'd say using it in obsidian has been the best use of it so far and one I certainly wouldn't trust to an external AI.

Three models my self, qwen 27b as a dev assistant, a small model rigged up to handle voice input as an assistant that's connected to my intercom system though out the house. Mostly to run my lights/music/audiobook play back. It's fun having jarvis control the house after all. Then a image generator.

Llama and swap on a 7900xtx.

I haven't gotten around to setting up phone access but wireguard for remote access form work in my laptop.

Honestly it's all basically over priced toys. I'm baffled that people rely on this stuff. I look at it the same way people do with model airplanes or RC cars.

Over priced toys that are hell of a lot of fun. Break frequently and require more care then they are probably worth to use safely lol

If you haven't seen it already, take a look at !localllama@sh.itjust.works

90€/mo to self-host something? Lol

I recently gave Strata a shot for my local AI. I'm very impressed with the performance they have eeked out of my hardware. I've got a rather beefy gaming desktop I built as an AI server back when the current craze started, before prices started going up, but models up until this point really failed to take full advantage of that hardware. Strata (which uses a Qwen model) both simplified the setup process, and it maxes out the hardware when running.

Strata acts like an OpenAI compatible API server, I then have Hermes as a "harness" to make an "agent", which I interact with on "buzz", which is like discord, but made for bots. Although Hermes also supports discord and several other chat platforms.

If you have the hardware to run Strata, I recommend setting it up, it comes with a basic chat interface, which you can then grill to assist you with setting up Hermes and whatever else.

One thing I didn't mention, is security. I highly recommend you treat AI as unpredictable, unreliable, and capable of being taken over by bad actors. This isn't a problem for a pure chat interface, but Hermes gives the AI tools to do things, and those tools could be used to take over your computer, scan your network, and create all kinds of havoc. For my part, Hermes is running in a virtual machine which is on an isolated network. I keep tight control on what it does and doesn't have access to in my network main network. That may not be something you have the hardware or knowledge to do, at the very least I recommend running Hermes in a docker container, and not giving it access to your personal desktop nor credentials to things like your email account.