Task force TalkChess access

[smatovic]

...

I disagree, just tried lynx, get logouts after some time.

--
Srdja

[hgm]

It seems (from studying the server log) the crash last Saturday morning was caused by the server being overloaded by requests from crawler bots. In itself there was no indication that the bots would be malicious. (Although it was a bit strange that one of the bots wasn't requesting any content from the forum, but desperately tried to register, log on and contact the admin all the time.) But there was no file robots.txt in the talkchess.com domain to put any restrictions on such bots.

The simplest remedy is of course to disallow access for all bots. But that would also mean forum posts will no longer can be found by search machines. An alternative is to throttle down the access rate of bots we want to allow, so they cannot do any damage.

What would be the bots that we would want to allow crawling the forum?

[Eelco de Groot]

I had not noticed that the (the old I asume)forum had crashed, maybe it is a good sign if that means that at the moment no IP blocking from the host or less blocking, so more bots from East European IP addresses got through, but it shows that in my opinion the old forum can't really handle 300 to 400 bots all the time. Just a little bit more than that everyday load of 300-400 and the forum crashes. A flood of bots early Saturday morning, when usually there are not a lot of people I find it a bit suspicious. If blocking all those 300, 400, 500 or more bots would mean the webhost does not have to do IP blocking anymore and the forum is stable, with some help from the cloudservices, I think most users would welcome that?

Maybe have a poll on the new forum about that but it is not yet working for everybody, but could be a test how many can now vote there. I think Rybka forum for instance also blocked Wayback Machine bots. At least for some months maybe more. Those are the only bots I personally would consider giving an exception.

[ydebilloez]

(Although it was a bit strange that one of the bots wasn't requesting any content from the forum, but desperately tried to register, log on and contact the admin all the time.) But there was no file robots.txt in the talkchess.com domain to put any restrictions on such bots.

This behaviour is typical for hacker/spammer. Trying to filter bad Vs good bots is cat and mouse game. Not a good idea.

Version 3.3 is supporting reCaptcha 3 on registration. Is it enabled? A fail2ban could help avoiding brute-force attacks. (But no DDOS)

Why not disabling contact admin in the forum software and move it to a custom solution. The phpBB software seems rather weak in this anyway.

[mvanthoor]

What would be the bots that we would want to allow crawling the forum?

Google, Bing, Yahoo, and DuckDuckGo. I think you've got about 95% of the search engine market with that.

[smatovic]

What would be the bots that we would want to allow crawling the forum?

Google, Bing, Yahoo, and DuckDuckGo. I think you've got about 95% of the search engine market with that.

95% of the Western market maybe, what about China, India, Russia?

One common method in the past was to limit the concurrent connections from an single IP by Linux IP Tables or Apache mod_qos, not sure if this helps nowadays against crawler induced load,

***edit***
There must be some kind of equivalent setup in Cloudflare for this.

--
Srdja

[Tibono]

Hello,

since the current week (or so) the access that was previously restored (after the captcha topic remediation, if I remember correctly) fails again for me, requiring to enable a VPN workaround.

If any useful, the symptom I have without VPN is:
1) first hit to the forum succeeds to get bare data, but fails to load the page setup/styles; resulting in what looks like:

As one can see, my credentials were validated, as my user name and avatar picture are present.

2) next click on URLs to access any further content fails with usual 403 error. A full reload (refresh) of the page fails as well.
Only the first hit loads some partial data.

Do others get a similar behavior?

Hope this helps a better spotting of the issue,
Kind regards,
Tibono

[Joost Buijs]

When I use FireFox I have the same issues, somehow Chrome and Edge are giving these problems less frequently.

[Marek Soszynski]

When I use FireFox I have the same issues, somehow Chrome and Edge are giving these problems less frequently.

I have similiar issues, but not always. It's random for me.

[mehmet123]

I look at the number of people how many people are online at this website time to time. I don't know why, there seems to be a very serious decrease in the number of online people in the last 1-2 weeks.