This is a most excellent place for technology news and articles.
The article writer kind of complains that they're having to serve a 10MB file, which is the result of the gzip compression. If that's a problem, they could switch to bzip2. It's available pretty much everywhere that gzip is available and it packs the 10GB down to 7506 bytes.
That's not a typo. bzip2 is way better with highly redundant data.
I believe he's returning a gzip HTTP reaponse stream, not just a file payload that the requester then downloads and decompresses.
Bzip isn't used in HTTP compression.
For scrapers that not just implementing HTTP, but are trying to extract zip files, you can possibly drive them insane with zip quines: https://github.com/ruvmello/zip-quine-generator or otherwise compressed files that contain themselves at some level of nesting, possibly with other data so that they recursively expand to an unbounded ("infinite") size.
Brotli is an option, and it's comparable to Bzip.
Brotli gets it to 8.3K, and is supported in most browsers, so there's a chance scrapers also support it.
Gzip encoding has been part of the HTTP protocol for a long time and every server-side HTTP library out there supports it, and phishing/scrapper bots will be done with server-side libraries, not using browser engines.
~~Further, judging by the guy's example in his article he's not using gzip with maximum compression when generating the zip bomb files: he needs to add -9 to the gzip command line to get the best compression (but it will be slower).~~ (I tested this and it made no difference at all).
You can make multiple files with different encodings and select based on the Accept-Encoding header.
When I was serving high volume sites (that were targeted by scrapers) I had a collection of files in CDN that contained nothing but the word "no" over and over. Scrapers who barely hit our detection thresholds saw all their requests go to the 50M version. Super aggressive scrapers got the 10G version. And the scripts that just wouldn't stop got the 50G version.
It didn't move the needle on budget, but hopefully it cost them.
How do you tell scrapers from regular traffic?
Most often because they don't download any of the css of external js files from the pages they scrape. But there are a lot of other patterns you can detect once you have their traffic logs loaded in a time series database. I used an ELK stack back in the day.
That sounds like a lot of effort. Are there any tools that get like 80% of the way there? Like something I could plug into Caddy, nginx, or haproxy?
My experience is with systems that handle nearly 1000 pageviews per second. We did use a spread of haproxy servers to handle routing and SNI, but they were being fed offender lists by external analysis tools (built in-house).
Dang, I was hoping for a FOSS project that would do most of the heavy lifting for me. Maybe such a thing exists, idk, but it would be pretty cool to have a pluggable system that analyzes activity and tags connections w/ some kind of identifier so I could configure a web server to either send it nonsense (i.e. poison AI scrapers), zip bombs (i.e. bots that aren't respectful of resources), or redirect to a honey pot (i.e. malicious actors).
A quick search didn't yield anything immediately, but I wasn't that thorough. I'd be interested if anyone knows of such a project that's pretty easy to play with.
Not exactly what you asked, but do you know about ufw-blocklist?
I've been using this on my multiple VPSes for some time now and the number of fail2ban failed/banned has gone down like crazy. Previously, I had 20k failed attempts after a few months and 30-50 currently-banned IPs at all times; now it's less than 1k failed after a year and maybe 3-ish banned at any time.
There was also that paid service where users share their spammy IP address attempts with a centralized network, which does some dynamic intelligence monitoring. I forgot the name and search these days isn't great. Something to do with "Sense"? It was paid, but well recommended as far as I remember.
Edit: seems like the keyword is " threat intelligence platform"
Before I tell you how to create a zip bomb, I do have to warn you that you can potentially crash and destroy your own device.
LOL. Destroy your device, kill the cat, what else?
It'll email your grandmother all if your porn!
Ah yes, the infamous "stinky cheese" email virus. Who knew zip bombs could be so destructive. It erased all of the easter eggs off of my DVDs.
outstanding reference
Haven’t thought about that Weird Al song in a while
Sadly about the only thing that reliably helps against malicious crawlers is Anubis
That URL is telling me "Invalid response". Am I a bot?
I’m sorry you had to find out this way.
https://anubis.techaro.lol/docs/user/known-broken-extensions
If you have JShelter installed, it breaks the proof of work from anubis
You're using a VPN, right?
Nope. Just using Vivaldi on my Android device.
Now you know why your mom spent so much time with the Amiga
And if you want some customisation, e.g. some repeating string over and over, you can use something like this:
yes "b0M" | tr -d '\n' | head -c 10G | gzip -c > 10GB.gz
yes repeats the given string (followed by a line feed) indefinitely - originally meant to type "yes" + ENTER into prompts. tr then removes the line breaks again and head makes sure to only take 10GB and not have it run indefinitely.
If you want to be really fancy, you can even add some HTML header and footer to some files like header and footer and then run it like this:
yes "b0M" | tr -d '\n' | head -c 10G | cat header - footer | gzip -c > 10GB.gz
I'd be amazed if this works, since these sorts of tricks have been around since dinosaurs ruled the Earth, and most bots will use pretty modern zip libraries which will just return "nope" or throw an exception, which will be treated exactly the same way any corrupt file is - for example a site saying it's serving a zip file but the contents are a generic 404 html file, which is not uncommon.
Also, be careful because you could destroy your own device? What the hell? No. Unless you're using dd backwards and as root, you can't do anything bad, and even then it's the drive contents you overwrite, not the device you "destroy".
Yeah, this article came across as if written by a complete beginner. They mention having their WordPress hacked, but failed to admit it was because they didn't upgrade the install.
On the other hand, there are lots of bots scraping Wikipedia even though it's easy to download the entire website as a single archive.
So they're not really that smart....
I've been thinking about making an nginx plugin that randomizes words on a page to poison AI scrapers.
There are "AI mazes" that do that.
I remember reading and article about this but haven't found it yet
Probably only works for dumb bots and I'm guessing the big ones are resilient to this sort of thing.
Judging from recent stories the big threat is bots scraping for AIs and I wonder if there is a way to poison content so any AI ingesting it becomes dumber. e.g. text which is nonsensical or filled with counter information, trap phrases that reveal any AIs that ingested it, garbage pictures that purport to show something they don't etc.
There have been some attempts in that regard, I don’t remember the names of the projects, but there were one or two that’d basically generate a crapton of nonsense to do just that. No idea how well that works.
When it comes to attacks on the Internet, doing simple things to get rid of the stupid bots means kicking 90% of attacks out. No, it won't work against a determined foe, but it does something useful.
Same goes for setting SSH to a random port. Logs are so much cleaner after doing that.
I don't know as to poisoning AI, but one thing that I used to do was to redirect any suspicious bots or ones that were hitting their server too much to a simple html page with no JS or CSS or forward links. Then they used to go away.
❤️
This is why I use things like Docusaurus to generate static sites. Vulnerability injections are pretty hard when there's no code to inject into.
I want to know he they built that visualization