tablebase compression / academic integrity

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
syzygy
Posts: 4395
Joined: Tue Feb 28, 2012 10:56 pm

tablebase compression / academic integrity

Post by syzygy » Thu May 19, 2016 9:41 pm

I found this very interesting paper by the creators of the Lomonosov tablebases:
Compression of underdetermined data in a 7-piece chess table
Unfortunately the paper does not seem to be freely available.

However, I do have a copy of it. Some ideas in it probably have been around for quite some time. But the combination of all the ideas put together including many of the little details... Well, all of that was already known as well, discussed in bits and pieces in various forums since 2008 or so and published in its totality in April 2013:
https://github.com/syzygy1/tb

Interestingly, in this recently published paper the authors make no reference whatsoever to the immediate source of all "their" ideas. They must have forgotten?

mar
Posts: 1937
Joined: Fri Nov 26, 2010 1:00 pm
Location: Czech Republic
Full name: Martin Sedlak

Re: tablebase compression / academic integrity

Post by mar » Fri May 20, 2016 3:34 am

I know nothing about tablebases, but speaking of general compression
- filters? (png uses filters to achieve better compression, executable compressors convert near calls/jumps to absolute addresses, but this only helps LZ-based compression) - I've no idea about the structure of tablebase data though
- Huffman cannot encode using fractional bits (=> away from theoretical optimum)

What I came across recently is called ANS (a paper by Jarek Duda, http://arxiv.org/abs/1311.2540)
It's static coding similar to Huffman but can encode using "fractional bits" - not really, but with states as context...
(table-version also called FSE by Yann Collet, he also has a working implementation https://github.com/Cyan4973/FiniteStateEntropy and describes it on his blog)
Charles Bloom explains ANS in detail on his blog http://cbloomrants.blogspot.cz/2014/01/ ... ans-1.html
Question is if it'd be worth it (in terms of ratio). While it's probably somewhat slower to decode than Huffman, the table-based version may work pretty fast actually.

Michel
Posts: 2003
Joined: Sun Sep 28, 2008 11:50 pm

Re: tablebase compression / academic integrity

Post by Michel » Fri May 20, 2016 4:06 am

syzygy wrote:I found this very interesting paper by the creators of the Lomonosov tablebases:
Compression of underdetermined data in a 7-piece chess table
Unfortunately the paper does not seem to be freely available.

However, I do have a copy of it. Some ideas in it probably have been around for quite some time. But the combination of all the ideas put together including many of the little details... Well, all of that was already known as well, discussed in bits and pieces in various forums since 2008 or so and published in its totality in April 2013:
https://github.com/syzygy1/tb

Interestingly, in this recently published paper the authors make no reference whatsoever to the immediate source of all "their" ideas. They must have forgotten?
Seems indeed rather shocking. No acknowledment, no reference. Quite surprising since the authors are academics themselves.

You could try to write to the Editor-in-Chief... http://www.allertonpress.com/journals/muk.htm .
Ideas=science. Simplification=engineering.
Without ideas there is nothing to simplify.

Michel
Posts: 2003
Joined: Sun Sep 28, 2008 11:50 pm

Re: tablebase compression / academic integrity

Post by Michel » Fri May 20, 2016 4:32 am

A more effective strategy is probably to simply write an article about your own implementation and make it easily available (e.g. at https://github.com/syzygy1/tb or on the arXiv). In it you can point out that essentially the same set of techniques is used in the Lomonosov table bases and that you had multiple interactions with the authors of those in the past. Then Google can do its work.
Ideas=science. Simplification=engineering.
Without ideas there is nothing to simplify.

mar
Posts: 1937
Joined: Fri Nov 26, 2010 1:00 pm
Location: Czech Republic
Full name: Martin Sedlak

Re: tablebase compression / academic integrity

Post by mar » Fri May 20, 2016 4:46 am

Ah sorry I missed the point, they ripped off your ideas and published them as their own ... :shock:

bob
Posts: 20362
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: tablebase compression / academic integrity

Post by bob » Fri May 20, 2016 6:52 pm

syzygy wrote:I found this very interesting paper by the creators of the Lomonosov tablebases:
Compression of underdetermined data in a 7-piece chess table
Unfortunately the paper does not seem to be freely available.

However, I do have a copy of it. Some ideas in it probably have been around for quite some time. But the combination of all the ideas put together including many of the little details... Well, all of that was already known as well, discussed in bits and pieces in various forums since 2008 or so and published in its totality in April 2013:
https://github.com/syzygy1/tb

Interestingly, in this recently published paper the authors make no reference whatsoever to the immediate source of all "their" ideas. They must have forgotten?
This unfortunately happens. It seems that Computer Science programs sometimes fail to provide at least a modicum of "ethics 101". We've seen this with chess programs, we've seen this with assignments and reports, we've seen this with published papers. The latter when the referees for said journal are not "up to the task" of knowing what has happened with this topic in the past.

That has always been one of my criteria for accepting or refusing to review a paper for publication. I might understand the subject, but if I am not familiar with the subject then I usually decline to avoid letting this very kind of thing slip through a crack.

syzygy
Posts: 4395
Joined: Tue Feb 28, 2012 10:56 pm

Re: tablebase compression / academic integrity

Post by syzygy » Fri May 20, 2016 9:54 pm

Michel wrote:Seems indeed rather shocking. No acknowledment, no reference. Quite surprising since the authors are academics themselves.
Yes, I wonder what they were thinking.
You could try to write to the Editor-in-Chief... http://www.allertonpress.com/journals/muk.htm .
Yes, that was my plan.

I do not mind (at all) that they study my code or write about it. But they should not pretend that they came up with everything themselves.

Post Reply