moonshine69

Oops, realized I didn't answer your question about actually crawling, dig into common crawl documentation, they provide a bunch of technical data and stats that show you the scale..2-4billion pages per month

And note CC just does a sample of the pages it finds. So the more monthly dumps don't contain all of the data afaik

And the number above are for one of the monthly dumps

https://commoncrawl.github.io/cc-crawl-statistics/plots/crawlermetrics

The index (i.e just the list of urls) of common crawl is 250gb. The actual crawl is 500tb ish. Then pull out just the text you get down to 30tb ish. And then you need that all to be in a fts index.. (something like opensearch/elasticsearch) idk how big the cluster you would need for that is.. maybe 2tb per data node max. So 15 pretty beefy vms (8cpu w/64gb memory) with 2tb nvme for data and a few more as controller nodes

Common crawl numbers pulled from this paper

https://aclanthology.org/anthology-files/anthology-files/pdf/L/L18/L18-1286.pdf

Cluster sizing numbers pulled from tea leaf divination

Are you the llm though...this is a mangled rephrasing of the punchline.. was your prompt something like rephrase a common from a similar thread

Better Nate than lever?

Is this an llm bugging out by rephrasing?

Nice :)

Nice, ok hmm.. sounds like you are using the router for internal DNS, did you set all the services to point at caddy?

Eg you could try a quick hack in hosts (assuming the caddy ip is 192.168.10.10)

192.168.10.10 grafana.domain.example
192.168.10.10 prometheus.domain.example
192.168.10.10 jellyfin.domain.example

And then navigate to one of the services over https

Main thing is all your internal DNS records need to point at caddy, and then it forwards to the real container IP

Not strictly required but could be worthwhile to get a dedicated internal dns server up.. I use technitium, or there's pi hole or a few other options ..

Fair enough bud

I'm trying to highlight the absurdity of tieing buggyness to sloppyness, they are two distinct axis, not a spectrum. There are plenty of problems with LLMs but bugginess is not one of them. Idk if you've been watching the news lately, bug squashing in old code is up big time, can you guess why? And I wonder have you ever worked on a real codebase? Because the only perfect code is the code you wrote 5minites ago.. go look at Linux kernel code.. arguably the most used but of software out there, find me anything perfect in there.. like what is going on here in this artisanal, hand rolled, organic code?! You're arguing devs need more time to work on security, well guess what's giving them more time to work on security..

Ps. Sorry to the Bluetooth dev, didn't mean for you to catch strays, I love your code anyway!

If you are doing a dns challenge then You need to give caddy access to your domain to perform the dns challenge. Depending on domain provider that is easy or hard. It will briefly change some records that tells let's encrypt you own the domain and then issue the cert

Or you can do http challenge, this requires you to manually configure your domain to point to a publicly accessible caddy endpoint which is probably not what you want, maybe there is a way to make it work through a cloud flare tunnel

Who is your domain provider?

Its so annoying that all code has been perfect and bug free until these silly LLM people came and made buggy code

Me: spruke gimp to artist friends
Them: gimp? Like a gimp suit?
End of conversation

Dunno why you're getting downvotes

Envoy is what i use, or squid is an option