Selfhosted

60054 readers

979 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam.
Posts here are to be centered around self-hosting. Please ensure it is clear in your post how it relates to self-hosting.
Don't duplicate the full text of your blog or git here. Just post the link for folks to click.
Submission headline should match the article title.
No trolling.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago

MODERATORS

curbstickle@anarchist.nexus

curbstickle_lw@lemmy.world

115

How to combat large amounts of Ai scrapers (sh.itjust.works)

submitted 11 months ago by DrunkAnRoot@sh.itjust.works to c/selfhosted@lemmy.world

54 comments fedilink hide all child comments

everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions

you are viewing a single comment's thread
view the rest of the comments

[–] Cyber@feddit.uk 6 points 11 months ago

If you're able to, use GeoIP ranges to only allow access from the countries you want.

That immediately limits a lot of everything

Then - again if you're able to - use a block list that covers known scrapers in case they're in your country.

I use pfBlockerNG on my pfSense firewall for exactly this.