Forum has been very slow 2 days now...

[smatovic]

[..]
There is an 'sid' in the query string (session ID?), which suggests the requests come from someone who is logged on. But for each request from the same IP all sid are different.
[..]

Thanks for taking a look.

When you open the page for the first time or reject cookies then you get an sid in your url, sid for session id, after further browsing or log in it will be stored as sid in a cookie so it odes not appeasr in the url anymore.

--
Srdja

[smatovic]

Beside the apache log file one should take a look into the Munin monitoring tool (I gave the URL to the founders) to figure what the server bottleneck is. Munin monitors CPU load, RAM, disk, network, all four could cause the slowdowns we experience.

--
Srdja

[smatovic]

It seems the times of poor forum performance coinside with a huge number of GET requests (some 4/sec) from the IP gange 47.246.164.*. I now made an attempt to block those IPs by adding to the apache2.conf file:

Code: Select all

<RequireAll>
    Require all granted
    Require not ip 47.246.64.0/24
</RequireAll>

I am not sure whether this will have any effect without restarting the Apache server (which I am not sure how to do), or whether indeed it will work as intended, and whether this will then solve the problem.

Smart arse alert

One should not block manually single IP addresses or ranges (this is what ChessUSA did) but harden the Linux firewall against these. If one IP or IP range causes too many requests you can limit IP requests by configuring the firewall accordingly. I wrote back then a lil documentation in a text file on this and stored it on the the server plus emailed it to you founders.

--
Srdja

[hgm]

I now also posted a similar file (at normal ) for the busiest minute in an hour where trafic was low (13:00 - 14:00). I assume this reflects normal use.

[hgm]

[..]
There is an 'sid' in the query string (session ID?), which suggests the requests come from someone who is logged on. But for each request from the same IP all sid are different.
[..]

Thanks for taking a look.

When you open the page for the first time or reject cookies then you get an sid in your url, sid for session id, after further browsing or log in it will be stored as sid in a cookie so it odes not appeasr in the url anymore.

--
Srdja

I don't get it. If you request a page first time, you cannot have gotten anything yet. At best you get something in the data that you get in reply. But how would that end up in the url of a next request for the same page? Does the browser save it as a cookie, and then appends it to the next request to the site?

[smatovic]

[..]
There is an 'sid' in the query string (session ID?), which suggests the requests come from someone who is logged on. But for each request from the same IP all sid are different.
[..]

Thanks for taking a look.

When you open the page for the first time or reject cookies then you get an sid in your url, sid for session id, after further browsing or log in it will be stored as sid in a cookie so it odes not appeasr in the url anymore.

--
Srdja

I don't get it. If you request a page first time, you cannot have gotten anything yet. At best you get something in the data that you get in reply. But how would that end up in the url of a next request for the same page? Does the browser save it as a cookie, and then appends it to the next request to the site?

I am not into the details how phpBB code implements it, but this is what I can verify with my browser:

1. You request https://talkchess.com
2. In all web site links there is a sid in the url, the session id, e.g.:

viewforum.php?f=2&sid=ad23fbca1efef0640e1d1159bb169e48

3.a) If your client accepts cookies, and you click on any of those links, then the sid is stored in a cookie in your browser which is transmitted with every http header and there will be no sid anymore in the url
3.b) If your client rejects cookies, then the sid will be present in all following links in the url.

That being said, I do not know, how crawlers handle session ids in urls and cookies.

***edit***
You can google about session highjacking, therefore it is not recommended to share urls with parameters, but usually the session is bound to an IP address server wise, which helps.

--
Srdja

[hgm]

Unfortunately I don't know anything about firewalls, or how to configure those. What seems needed is selective temporary blocking of IPs based on the number of requests they made in the recent past, especially requests of the same page (index.php). I don't know if the Linux firewall can be configured to do that.

It seems there exists a tool to do this on the server level: mod_evasive. I am not sure how this would handle URLs with query strings, though. The accesses to index.php that consitute the attack all have different sid in the query string, even when they come from the same IP address. For viewtopic.php it is usual to be requested many times from the same IP with different query strings, though, identifying different topics.

It seems strange to request index.php explicitly anyway. I extracted requests from my own IP address from today's log (uploaded here), and when I access the home page, (www.talkchess.com in the address bar), the request is just "GET /".

[smatovic]

There are several Apache modules present to limit requests, mod_qos and mod_evasive for example.

There is the "ufw" frontend present for "iptables" Linux firewall.

As mentioned, the Linux server was hardened in a fist and second step, now with aggressive bots a third step of config tuning is probably necessary.

I documented server settings in text files in a folder named "howto" on the server and emailed those files to you founders.

I had from 1999 til 2010 the profession of an web-programmer and Linux system-administrator, and reached with the second tuning step of the TC server my limit, my knowledge is meanwhile outdated.

Hence I suggest to the founders to seek for skilled volunteers or to engage a professional for tuning Apache modules and iptables against crawlers and bots.

--
Srdja

[hgm]

This mod_evasive is a very simple piece of C code: only 699 lines in a single file, most of it for handling the necessary infrastructure for maintaing hash tables. The routine that actually decides what it does is very straightforward:

Code: Select all

static int access_checker(request_rec *r) 
{
    int ret = OK;

    /* BEGIN DoS Evasive Maneuvers Code */

    if (r->prev == NULL && r->main == NULL && hit_list != NULL) {
      char hash_key[2048];
      struct ntt_node *n;
      time_t t = time(NULL);

      /* Check whitelist */
      if (is_whitelisted(r->connection->remote_ip)) 
        return OK;

      /* First see if the IP itself is on "hold" */
      n = ntt_find(hit_list, r->connection->remote_ip);

      if (n != NULL && t-n->timestamp<blocking_period) {
 
        /* If the IP is on "hold", make it wait longer in 403 land */
        ret = HTTP_FORBIDDEN;
        n->timestamp = time(NULL);

      /* Not on hold, check hit stats */
      } else {

        /* Has URI been hit too much? */
        snprintf(hash_key, 2048, "%s_%s", r->connection->remote_ip, r->uri);
        n = ntt_find(hit_list, hash_key);
        if (n != NULL) {

          /* If URI is being hit too much, add to "hold" list and 403 */
          if (t-n->timestamp<page_interval && n->count>=page_count) {
            ret = HTTP_FORBIDDEN;
            ntt_insert(hit_list, r->connection->remote_ip, time(NULL));
          } else {

            /* Reset our hit count list as necessary */
            if (t-n->timestamp>=page_interval) {
              n->count=0;
            }
          }
          n->timestamp = t;
          n->count++;
        } else {
          ntt_insert(hit_list, hash_key, t);
        }

        /* Has site been hit too much? */
        snprintf(hash_key, 2048, "%s_SITE", r->connection->remote_ip);
        n = ntt_find(hit_list, hash_key);
        if (n != NULL) {

          /* If site is being hit too much, add to "hold" list and 403 */
          if (t-n->timestamp<site_interval && n->count>=site_count) {
            ret = HTTP_FORBIDDEN;
            ntt_insert(hit_list, r->connection->remote_ip, time(NULL));
          } else {

            /* Reset our hit count list as necessary */
            if (t-n->timestamp>=site_interval) {
              n->count=0;
            }
          }
          n->timestamp = t;
          n->count++;
        } else {
          ntt_insert(hit_list, hash_key, t);
        }
      }

      /* Perform email notification and system functions */
      if (ret == HTTP_FORBIDDEN) {
        char filename[1024];
        struct stat s;
        FILE *file;

        snprintf(filename, sizeof(filename), "%s/dos-%s", log_dir != NULL ? log_dir : DEFAULT_LOG_DIR, r->connection->remote_ip);
        if (stat(filename, &s)) {
          file = fopen(filename, "w");
          if (file != NULL) {
            fprintf(file, "%ld\n", getpid());
            fclose(file);

            LOG(LOG_ALERT, "Blacklisting address %s: possible DoS attack.", r->connection->remote_ip);
            if (email_notify != NULL) {
              snprintf(filename, sizeof(filename), MAILER, email_notify);
              file = popen(filename, "w");
              if (file != NULL) {
                fprintf(file, "To: %s\n", email_notify);
                fprintf(file, "Subject: HTTP BLACKLIST %s\n\n", r->connection->remote_ip);
                fprintf(file, "mod_evasive HTTP Blacklisted %s\n", r->connection->remote_ip);
                pclose(file);
              }
            }

            if (system_command != NULL) {
              snprintf(filename, sizeof(filename), system_command, r->connection->remote_ip);
              system(filename);
            }
 
          } else {
            LOG(LOG_ALERT, "Couldn't open logfile %s: %s",filename, strerror(errno));
	  }

        } /* if (temp file does not exist) */

      } /* if (ret == HTTP_FORBIDDEN) */

    } /* if (r->prev == NULL && r->main == NULL && hit_list != NULL) */

    /* END DoS Evasive Maneuvers Code */

    if (ret == HTTP_FORBIDDEN
	&& (ap_satisfies(r) != SATISFY_ANY || !ap_some_auth_required(r))) {
        ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r,
            "client denied by server configuration: %s",
            r->filename);
    }

    return ret;
}

This opens the possibility of modifying it for our own purpose. E.g. the DDoS attack we are syffering disguises accesses to the same page by appending a query string with a different sid. This would thwart the recoginition of "too many requests for the same page", which takes the full query string as part of the page identity. But we could add code to eliminate the sid parameter of a query string, before calculating the hash key from it.

[towforce]

I had from 1999 til 2010 the profession of an web-programmer and Linux system-administrator, and reached with the second tuning step of the TC server my limit, my knowledge is meanwhile outdated. Hence I suggest to the founders to seek for skilled volunteers or to engage a professional for tuning Apache modules and iptables against crawlers and bots.

I think you'll be impressed how helpful today's top chatbots are with knotty technical issues where the knowledge is widely known. It's definitely worth a try.