The unfair match between good and bad has most likely been a conundrum for thousands of years. Multiple solutions has been proposed as to how to deal with the bad ones, including; kill them, forgive them, educate and reform them - but nothing really seems to work.
One solution might be to gather them all up and send them to another planet were they can live as they please without bothering the good ones - and some might suggest that this is exactly what they did and here we are :)
I think I saw an article recently where someone used the http protocol to serve gzipped content that was specially crafted to explode in size on the receiver side. This could be a good preventative to crawlers as they don't typically have that much space dedicated to each instance.
It’s worth noting that the “zip bomb” was at a resource location specified in the Disallow section of robots.txt, meaning the server specifically told the bot not to go there and it did anyway.
(Not that the parent commenter seems confused, just that it hadn’t been noted.)
> One solution might be to gather them all up and send them to another planet were they can live as they please without bothering the good ones - and some might suggest that this is exactly what they did and here we are :)
One solution might be to gather them all up and send them to another planet were they can live as they please without bothering the good ones - and some might suggest that this is exactly what they did and here we are :)