Help request for debugging ICS
Posted: Sat Jan 07, 2017 5:46 pm
After many months of flawless running, the ICS on which I organize the monthly blitz tournaments suddenly became crash prone in October, and now typically crashes within the hour. Nothing was changed, neither in the ICS nor in the OS of the VPS it is running on. My experience with Linux is quite limited, and I am not sure how I can figure out what causes these crashes.
The ICS itself is so friendly that it catches segfaults, through the following handler:
Indeed many segv_* files were created in the home directory, corresponding to times where we suffered crashing:
(Note that in the crash-free periods since October the ICS was simply not running. According to the change log on May 8, 2015 I fixed a buffer overrun.) Unfortunately the segv* files do not contain any useful information; the all contain the single line
Apparently I am missing a tool called 'backtrace' that ancient operators of the ICS still had, but now is no longer part of the project.
How could I figure out where the ICS is crashing? I suppose I could run it under gdb. Unfortuately, gdb is not installed on the VPS where the ICS is running. "Apt-get install" does not work on that machine, because repositories for the OS (Ubuntu 10.04) no longer exist.
Are there other ways to get a working gdb on that system? Or other ways to get a stack trace? Would it be possible to force a core dump from the segfault handler, which could be taken to another machine for post-mortem debugging? If so, how?
The ICS itself is so friendly that it catches segfaults, through the following handler:
Code: Select all
/*
give a decent backtrace on segv
*/
static void segv_handler(int sig)
{
char cmd[100];
snprintf(cmd, sizeof(cmd), "/home/mics/bin/backtrace %d > /home/mics/chessd/segv_%d 2>&1",
(int)getpid(), (int)getpid());
system(cmd);
_exit(1);
}
Code: Select all
mics@www:~/chessd$ ls -lt
total 140
-rw-r--r-- 1 mics mics 40 2016-12-24 14:13 segv_9841
-rw------- 1 mics mics 8192 2016-12-23 21:30 config.tdb
-rw------- 1 mics mics 696 2016-12-23 21:30 news.tdb
-rw-r--r-- 1 mics mics 40 2016-12-03 12:32 segv_15471
-rw-r--r-- 1 mics mics 40 2016-11-19 23:09 segv_1092
drwxr-xr-x 2 mics mics 20480 2016-11-19 23:05 spool
-rw------- 1 mics mics 27360 2016-11-19 21:10 admin.log
-rw-r--r-- 1 mics mics 40 2016-11-19 20:38 segv_628
-rw-r--r-- 1 mics mics 40 2016-11-19 19:24 segv_15291
-rw-r--r-- 1 mics mics 40 2016-11-19 18:27 segv_11847
-rw-r--r-- 1 mics mics 40 2016-11-19 15:43 segv_5887
-rw-r--r-- 1 mics mics 40 2016-11-16 23:07 segv_28079
-rw-r--r-- 1 mics mics 40 2016-10-23 08:35 segv_19347
-rw-r--r-- 1 mics mics 40 2015-07-16 03:33 segv_28315
-rw-r--r-- 1 mics mics 40 2015-05-08 12:56 segv_27428
-rw-r--r-- 1 mics mics 40 2015-05-08 10:55 segv_27290
-rw-r--r-- 1 mics mics 40 2015-05-08 10:43 segv_26709
-rw-r--r-- 1 mics mics 40 2015-05-06 17:33 segv_13933
-rw-r--r-- 1 mics mics 40 2015-02-24 02:38 segv_28503
drwxr-xr-x 2 mics mics 4096 2014-04-29 12:24 bin
drwxr-xr-x 10 mics mics 4096 2014-04-29 12:18 data
drwxr-xr-x 5 mics mics 4096 2014-04-29 12:18 games
drwxr-xr-x 2 mics mics 4096 2014-04-29 12:18 lib
drwxr-xr-x 28 mics mics 4096 2014-04-29 12:18 players
Code: Select all
sh: /home/mics/bin/backtrace: not found
How could I figure out where the ICS is crashing? I suppose I could run it under gdb. Unfortuately, gdb is not installed on the VPS where the ICS is running. "Apt-get install" does not work on that machine, because repositories for the OS (Ubuntu 10.04) no longer exist.
Are there other ways to get a working gdb on that system? Or other ways to get a stack trace? Would it be possible to force a core dump from the segfault handler, which could be taken to another machine for post-mortem debugging? If so, how?