You may have noticed that my various sites were nearly unusable for the last 4+ days, and are still maybe not doing great. The current attack by AI scrapers is somehow of an
even more harmful character than previous ones.
During the last storm, I was able to weather 80+req/sec. But this time with only 32 req/sec, responsiveness went completely to shit. Possibly because thousands of connections were staying in LAST_ACK? I dunno.
After trying many things, the only thing that alleviated it somewhat was to start blackholing entire countries by /16.
And having done so, I'm... still getting 32/req sec. But now the site is responsive. So there's some thing special about this latest attacker that makes things fall over, that is not merely "hammering you with page loads as fast as you respond". And this was the case even if I fed the scrapers a plain-old Apache 404 or 503, without dropping down into PHP at all.
To whom do I send the invoice for the lost productivity of my staff, who could not access our own web site to do their jobs for most of the week? Asking for an everybody.
Anyway, some things I have been finding helpful this week:
My approach (which had been working great last week, but not so much this week) was to create separate php-fpm pools for "real" web sites and for scrapers; and a corresponding fcgi proxy for each. The idea here being that under load, the "scraper" responder might become unresponsive, but the "real" site will be fine. And that was working pretty well... last month.
Something like:
<Proxy "unix:/run/php-fpm/jwz.sock|fcgi://php-jwz"> ProxySet </Proxy> ... <Proxy "unix:/run/php-fpm/dadadodo.sock|fcgi://php-dadadodo"> ProxySet timeout=10 </Proxy>
But the fact that under catastrophic load, I couldn't even get the php-fpm status pages to load, despite using a pm.status_listen "invisible pool" to bypass the queue, suggest to me that there's some other shared resource contention I don't know about:
<Location /fpm-status-dadadodo> SetHandler "proxy:unix:/run/php-fpm/dadadodo-status.sock|fcgi://php-dadadodo" Require ip ... </Location>
I also have conflicting experience with whether things are better with a "reasonable" number of Apache workers, or going hog wild on it: ServerLimit 16, MaxRequestWorkers 400 versus ServerLimit 400, MaxRequestWorkers 10000.
In summary: "AI is the future, don't get left behind." Also in this future you can't have a web site.
Previously, previously, previously, previously.