Self Hosted - Self-hosting your services.

20901 readers

1 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules

No harassment
crossposts from c/Open Source & c/docker & related may be allowed, depending on context
Video Promoting is allowed if is within the topic.
No spamming.
Stay friendly.
Follow the lemmy.ml instance rules.
Tag your post. (Read under)

Important

Lemmy doesn't have tags yet, so mark it with [Question], [Help], [Project], [Other], [Promoting] or other you may think is appropriate. This is strongly encouraged!

Cross-posting

!everything_git@lemmy.ml is allowed!
!docker@lemmy.ml is allowed!
!portainer@lemmy.ml is allowed!
!fediverse@lemmy.ml is allowed if topic has to do with selfhosting.
!selfhosted@lemmy.ml is allowed!

If you see a rule-breaker please DM the mods!

founded 5 years ago

MODERATORS

dogmuffins@lemmy.ml

Zoe8338@lemmy.ml

testman@lemmy.ml

LLM with Web Search functionality (lemmy.ml)

submitted 2 months ago by SubArcticTundra@lemmy.ml to c/selfhost@lemmy.ml

26 comments fedilink hide all child comments

Conducting deep web searches and gathering sources is one of the main things I've been using LLMs for. How far away are we from being able to self-host something like Claude's web search capabilities? Or even just a service where I'd pay with my money instead of my data?

you are viewing a single comment's thread
view the rest of the comments

[–] TropicalDingdong@lemmy.world 0 points 2 months ago (1 children)

Do you have a walk through for setup?

I'm on the strix halo 128 gb variant and while I got ollama working fine, i haven't gotten any of these multi headed setups working

[–] vapeloki@lemmy.world 0 points 2 months ago (2 children)

I am on Gentoo for it, but everything with a decent rocm should work.

Have a look for llama-swap, that handles multi head endpoints.

Also, as you are on a big board, you can quantize yourself, as the BF16 version of qwen has only 72gb.

I will try and post a full writeup next days. But feel free to dm me, if you need some guidance on quantize or more.

I am using this fork currently: https://github.com/charlie12345/ROCmFPX

Stuff happens fast currently, so may be worth to wait a week or two ig you need something super stable, but if you are up for experimenting, that's the way to go

[–] Shimitar@downonthestreet.eu 0 points 2 months ago

Great man! Gentoo lover and long time addicted here.... Keep it the good work!

[–] TropicalDingdong@lemmy.world 0 points 2 months ago

THis is great, thanks. I'm on the z-13 and needed to use it for a work project, which is wrapping up soon. I'm planning on re-building it as a locally hosted agent support machine.