During the last storm, I was able to weather 80+req/sec. But this time with only 32 req/sec, responsiveness went completely to shit. Possibly because thousands of connections were staying in LAST_ACK? I dunno.
After trying many things, the only thing that alleviated it somewhat was to start blackholing entire countries by /16.
And having done so, I'm... still getting 32/req sec. But now the site is responsive. So there's some thing special about this latest attacker that makes things fall over, that is not merely "hammering you with page loads as fast as you respond". And this was the case even if I fed the scrapers a plain-old Apache 404 or 503, without dropping down into PHP at all.
To whom do I send the invoice for the lost productivity of my staff, who could not access our own web site to do their jobs for most of the week? Asking for an everybody.
Anyway, some things I have been finding helpful this week:
- country-block;
- ipsum = level 3;
- blocklist.de = all.
My approach (which had been working great last week, but not so much this week) was to create separate php-fpm pools for "real" web sites and for scrapers; and a corresponding fcgi proxy for each. The idea here being that under load, the "scraper" responder might become unresponsive, but the "real" site will be fine. And that was working pretty well... last month.
Something like:
<Proxy "unix:/run/php-fpm/jwz.sock|fcgi://php-jwz"> ProxySet </Proxy> ... <Proxy "unix:/run/php-fpm/dadadodo.sock|fcgi://php-dadadodo"> ProxySet timeout=10 </Proxy>
But the fact that under catastrophic load, I couldn't even get the php-fpm status pages to load, despite using a pm.status_listen "invisible pool" to bypass the queue, suggest to me that there's some other shared resource contention I don't know about:
<Location /fpm-status-dadadodo> SetHandler "proxy:unix:/run/php-fpm/dadadodo-status.sock|fcgi://php-dadadodo" Require ip ... </Location>
I also have conflicting experience with whether things are better with a "reasonable" number of Apache workers, or going hog wild on it: ServerLimit 16, MaxRequestWorkers 400 versus ServerLimit 400, MaxRequestWorkers 10000.
In summary: "AI is the future, don't get left behind." Also in this future you can't have a web site.
SANAA (ANP/AFP) - Het dodental als gevolg van de Houthi-aanvallen op Jemenitische militairen is opgelopen tot 58. Dat meldt persbureau AFP op basis van een militaire bron. Eerder op de dag werd nog een dertigtal doden gemeld.
De Houthi-rebellen vielen donderdag met ballistische raketten en drones Jemenitische overheidsdoelen aan. Het zou gaan om een reactie op de vermeende opbouw van het staatsleger, gesteund door Saudi-Arabië. Jemen zei donderdagmiddag de aanval, naar verluidt de dodelijkste sinds 2022, te zullen vergelden "op een gepast moment".
Hoewel Jemen officieel een regering met staatsleger heeft, hebben de separatistische, aan Iran gelieerde Houthi's in de praktijk al jaren de macht in Jemen, met name in grote delen van het westen van het land. Daar vormen zij een directe bedreiging voor de commerciële vaart, vanwege de ligging aan de Rode Zee en de belangrijke zeestraat Bab-el-Mandeb.
The End of Google Search. The internet giant is “changing the formula” of its products and media folks are beginning to assume that there will be no search traffic in the future…and fretting about their survival.