Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

So these companies broke the internet





Which companies?

OpenAI, Anthropic, Google? No, their bots are pretty well behaved.

The smaller AI companies deploying bots that don't respect any reasonable rate limits and are scraping the same static pages thousands of times an hour? Yup


Anecdote, but at least for tiny little server hosting single public repository, none of these companies had 'well behaved' bots. It may be possible that they learned to behave better but I wouldn't know since my only possible recourse was to blacklist them all AND take the repo private.

Those are the small companies spoofing their user agent as the big companies to dodge countermeasures.



Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: