this post was submitted on 04 Jul 2025
115 points (96.0% liked)

Selfhosted

60542 readers
1473 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

  1. Be civil.

  2. No spam.

  3. Posts are to be related to self-hosting.

  4. Don't duplicate the full text of your blog or readme if you're providing a link.

  5. Submission headline should match the article title.

  6. No trolling.

  7. Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the link in Rule 8 for details.

  8. AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post. )

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago
MODERATORS
 

everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions

you are viewing a single comment's thread
view the rest of the comments
[–] fubarx@lemmy.world 11 points 1 year ago (2 children)

If nginx, here's an open-source blocker/honeypot: https://github.com/raminf/RoboNope-nginx

If you have it set up to be proxied or hosted by Cloudflare, they have their own solution: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/

[–] ikidd@lemmy.world 3 points 1 year ago* (last edited 1 year ago)

I wonder why that RoboNope doesn't just make a fail2ban entry for anything that accesses a disallowed url and drop them entirely.

Actually this look like it would do something similiar, then dumps them to fail2ban after the re-access the honeypot page too many times: https://petermolnar.net/article/anti-ai-nepenthes-fail2ban/

[–] DrunkAnRoot@sh.itjust.works 2 points 1 year ago

ill check robonope out seems promising