Forum has been very slow 2 days now...

[Jouni]

...but works just.

[Eelco de Groot]

When you see there are 3889 guests visiting you know something is wrong. It's an old problem if you search the history, badly designed (or maliciously designed?) crawler bots and they can crash the forum if not enough capacity. But you have to ask Srdja or other administrators if they can restrict access further? Forum does not really need crawlers of course, the forum might as well be private these days no commercial need but opinions might differ on that. AI needs those bots to crawl the internet, for instance.

[Graham Banks]

Ridiculously slow. Almost unusable.

[royb]

I agree. It has been very slow. 10 seconds at a minimum to see the text of a post. Very painful.

Edit: And just a few minutes after I posted this, it seems much faster - back to normal even perhaps.

Happy to see that and I think we all hope it stays that way!

[Eelco de Groot]

In case there is a political factor to it I do not think we are that important but The Netherlands is holding a very large NATO summit in The Hague, less than two months from now, so I don't think we are entirely safe. I just read that 29th (yesterday) there were numerous attacks, DDOS related, on government related websites and Russian hackers claiming responsibility. "If the US thinks this is all okay, we can expect more of it", cybersecurity expert Inge Bryan told on Dutch television NPO 1 https://wnl.tv/2025/04/29/weer-grootsch ... -in-gevaar

But this is probably not related to our forum, I hope not

[Graham Banks]

Seems better now. Super fast!

[Eelco de Groot]

Thumbs up if anyone repaired this, it works great now.

[smatovic]

Disclaimer: I am just an ordinary member, have no access to the server anymore.

On the main page at bottom you can see how many users/guests are visiting the
site, usually it is about 100+, now it is about 1000, during the page slowdown
it was 3000+.

Either some post on TalkChess went viral on social media, causing a "DDOS" by
real user requests (a so called Reddit hug of death), or, a handful of
crawlers/bots with multiple IP-addresses started to crawl the 800+K posts of TC
at once, causing overload on the lil VPS server. The server can handle some
bots, but if too many, or misconfigured, lil VPS server is brought into its
knees.

The Linux web-server was hardened in a fist and second step, if the situation
prevails, we might need a third step, with further tuning of defensive methods.

--
Srdja

[smatovic]

Further note:

The good crawlers/bots respect robots.txt, these can be disallowed to visit the
site, but they usually do not cause the trouble with too many requests. The bad
scrapers/bots do not respect robots.txt and can fake their user agent, not
distinctable from an normal user, these one you can only limit by hardening the
web-server and Linux server configuration, or, by adding a CDN, content
delivering network, like Cloudflare in front.

--
Srdja

[royb]

And we're back to being very slow again. Is there some way to stop bots from dragging down the server's performance? Can they be "niced" (Linux term) so they get only "leftover" cycles after all other needs are handled for regular users?