Fathom, munmap issue

[Tearth]

Hi, I know a lot of people are using Fathom here - last time I'm trying to incorporate this library into my engine and found an issue that I can't explain. The thing happens on Linux (Windows works ok) when I'm calling tb_init two times (so this is basically a scenario when SyzygyPath is changed two times through UCI) - in the second call, Fathom tries to free tables initialized earlier:

Code: Select all

// if pathString is set, we need to clean up first.
if (pathString) {
  free(pathString);
  free(paths);

  for (int i = 0; i < tbNumPiece; i++)
    free_tb_entry((struct BaseEntry *)&pieceEntry[i]);
  for (int i = 0; i < tbNumPawn; i++)
    free_tb_entry((struct BaseEntry *)&pawnEntry[i]);

  LOCK_DESTROY(tbMutex);

  pathString = NULL;
  numWdl = numDtm = numDtz = 0;
}

The issue is, every time during this process, I'm getting multiple (like 4 or 7, it's also not constant) "munmap: No error information" messages in the output. The data pointer and size look okayish (so no nulls or some strange values, but can't say it for sure since sadly I almost completely don't understand how the whole algorithm is working).

Code: Select all

static void unmap_file(void *data, map_t size)
{
  if (!data) return;
  if (!munmap(data, size)) {
    perror("munmap");
  }
}

Besides that, everything still works well, so in theory, I can just silence this error and forget about everything (especially considering that SyzygyPath is changed twice rather rarely), but at the same time I'm kinda worried that it's the symptom of some other bug that will crash my engine one day. If anyone experienced anything similar, I would be glad for a little help or explanation.

[syzygy]

Code: Select all

// if pathString is set, we need to clean up first.
if (pathString) {
  free(pathString);
  free(paths);

  for (int i = 0; i < tbNumPiece; i++)
    free_tb_entry((struct BaseEntry *)&pieceEntry[i]);
  for (int i = 0; i < tbNumPawn; i++)
    free_tb_entry((struct BaseEntry *)&pawnEntry[i]);

  LOCK_DESTROY(tbMutex);

  pathString = NULL;
  numWdl = numDtm = numDtz = 0;
}

The problem is that the Fathom code does not set tbNumPiece and tbNumPawn to 0 if a few lines later tb_init() exists because path is zero or "<empty>". tbNumPiece and tbNumPawn are reset only later in the code.

The CFish tb_init() code has this:

Code: Select all

  // if pathString is set, we need to clean up first.
  if (pathString) {
    free(pathString);
    free(paths);

    TB_release();

    LOCK_DESTROY(tbMutex);

    pathString = NULL;
  }

  numWdl = numDtm = numDtz = 0;
  tbNumPiece = tbNumPawn = 0;
  TB_MaxCardinality = TB_MaxCardinalityDTM = 0;

  // if path is an empty string or equals "<empty>", we are done.
  const char *p = path;
  if (strlen(p) == 0 || !strcmp(p, "<empty>")) return;

This prevents the error messages.

[Tearth]

Thanks, while what you wrote wasn't exactly the reason, I found that this condition is just written inverted (CFish doesn't check it at all):

Code: Select all

if (!munmap(data, size)) {
    perror("munmap");
}

https://github.com/jdart1/Fathom/blob/m ... obe.c#L372

Function munmap returns 0 on success and -1 on error, and indeed I've tested that it's always 0 so everything is perfectly fine. That's also why sometimes I was getting nonsensical errno value, which was obviously not set during munmap calls. I will create a pull request, so hopefully it will be fixed for future users.

Source:
https://linux.die.net/man/2/munmap

On success, munmap() returns 0, on failure -1, and errno is set (probably to EINVAL).

[Tearth]

Pull request with the fix has been accepted and merged, so the problem is now resolved for everyone.

[Alexlaw1964]

Hello.
I use Code::Blocks 17.12 is a full-featured IDE (Integrated Development Environment).
I compiled https://github.com/jdart1/Fathom/blob/m ... obe.c#L372 under windows.
If you comment out the lines if ((size & 63) != 16) ... , then everything works

Code: Select all

static bool test_tb(const char *str, const char *suffix)
{
  FD fd = open_tb(str, suffix);
  if (fd != FD_ERR) {
    size_t size = file_size(fd);
    close_tb(fd);
  // if ((size & 63) != 16) {
   //   fprintf(stderr, "Incomplete tablebase file %s%s\n", str, suffix);
   //   printf("info string Incomplete tablebase file %s%s\n", str, suffix);
   //   fd = FD_ERR;
    //}
  }
  return fd != FD_ERR;
}

Otherwise, it doesn't work and gives an error.

Code: Select all

...
error: unable to initialize tablebase; no tablebase files found
Process returned 1 (0x1)   execution time : 1.260 s
Press any key to continue.

Why?

[AndrewGrant]

Hello.
I use Code::Blocks 17.12 is a full-featured IDE (Integrated Development Environment).
I compiled https://github.com/jdart1/Fathom/blob/m ... obe.c#L372 under windows.
If you comment out the lines if ((size & 63) != 16) ... , then everything works

Code: Select all

static bool test_tb(const char *str, const char *suffix)
{
  FD fd = open_tb(str, suffix);
  if (fd != FD_ERR) {
    size_t size = file_size(fd);
    close_tb(fd);
  // if ((size & 63) != 16) {
   //   fprintf(stderr, "Incomplete tablebase file %s%s\n", str, suffix);
   //   printf("info string Incomplete tablebase file %s%s\n", str, suffix);
   //   fd = FD_ERR;
    //}
  }
  return fd != FD_ERR;
}

Otherwise, it doesn't work and gives an error.

Code: Select all

...
error: unable to initialize tablebase; no tablebase files found
Process returned 1 (0x1)   execution time : 1.260 s
Press any key to continue.

Why?

First guess is you have corrupted Syzygy files. I would suggest comparing checksums/sha256s.

[Alexlaw1964]

First guess is you have corrupted Syzygy files. I would suggest comparing checksums/sha256s.

If delete these lines, everything works correctly.

[Ciekce]

If delete these lines, everything works correctly.

yeah, when you delete error-checking code the errors won't appear

doesn't mean it works correctly

[Alexlaw1964]

yeah, when you delete error-checking code the errors won't appear

I wanted to understand what this code checks.

[AndrewGrant]

First guess is you have corrupted Syzygy files. I would suggest comparing checksums/sha256s.

If delete these lines, everything works correctly.

Hopefully I won't regret actually explaining something on Talkchess, but here goes:

That code checks that the file you are opening is a specific size. Namely, the file size must be a multiple of 16. Among other things. It is probably the case that the files have some data format, in which this is guaranteed to be true. So one of the first checks in the code is to make sure that it is infact true.

The condition is false for you. Very likely because you do not have VALID tablebase files downloaded. Either you have partial downloads, or corrupted data, or possibly just downloaded from a source with incomplete/invalid tables.

The correct path forward for you to is compute checksums for your files, and compare them against the known checksums. Something like ChatGPT can help you do this if needed. You could also just delete all your files, and redownload them. I would suggest the lichess download source.

To make it super super super clear: Deleting the line DOES NOT FIX YOUR PROBLEM. It hides it. If you delete this code, you will run into problems somewhere else, almost immediately.