Web developer. Lead developer of PieFed
Technology @lemmy.world AI hiring tools may be filtering out the best job applicants
Technology @lemmy.world Google gets its way, bakes a user-tracking ad platform directly into Chrome
Technology @lemmy.world OpenAI wants to raise 5-7 trillion dollars. Yes, Trillion
Fediverse @lemmy.world FediForum, an online unconference for discussing the future of the Fediverse, 19-20 March




I used fail2ban as an anti-scraper strategy for a few months. It helped with the really basic scrapers that only use one or a few IP addresses.
But then they adapted and started using hundreds or thousands of different IP addresses, each one only making a couple of requests. fail2ban became counter-productive as it slowed my server down - the log files it was monitoring became massive under high load and it needs to scan the log files often to check if an IP has been seen before.
I'm not saying don't try it, I'm saying it's only effective in some situations.