Chessprogams with the most chessknowing

[cdani]

Thanks! I will improve Andscacs with those positions. Sure others will do also.

[Sylwy]

Hi Carl !

They are a lot of private chess engines with a very particular (not to tell fantastic) architecture ! Diep is just one.
Very few persons know something about !
Just two examples:

1.-The Baron (from 3.xx series) is able to play amazing games (even the lost ones) versus very strong engines. Sometimes engines like Stockfish, Komodo, Houdini are KO (20-25 moves) despite their big speed ! I have this engine and I did a lot of experiments !

2.-Lemming - I've heard this engine has only 1kn/s and an huge amount of hash used. And around 2600 Elo. That means a lot of extensions !!!

Stockfish-Komodo-Houdini & co.are only the visible direction of development in computer chess. They are also architectural design in "the dark side" of computer chess programming ! The"black hole" of CC programming from here will strike a day !

Regards,
SilvianR

[bob]

This doesn't work very well. depth=1 does NOT mean the same thing to everyone. To some it is simply a 1 ply search followed by q-search. Some allow check extensions before going to q-search. Some allow check extensions AFTER q-search. Some have a couple of plies of threats between basic search and start of q-search. Some extend at the root if in check, some don't. Some extend if giving check at the root, some don't. Etc. I played a match Crafty vs Stockfish a while back to compare evaluations. It took quite a bit of work to make them search the same basic tree space.

I ended up having to modify code in both programs to reach a consistent meaning for "set depth = 1". After all the work was done, the net conclusion from playing 100K games was "no statistical difference".

Yes, correct, but it seems hard to measure comprehensibly the static eval without messing up with the source, where available.

I agree, it is quite difficult. In fact, effectively impossible. Unless you are willing to get "hands dirty" and look at the source. And then you run the risk of introducing an unintentional bug that biases the results even further.

My opinion? This is hopeless. One can compare engines in their entirety, but trying to isolate and measure just one part of an engine is a really difficult project. You can't even try the ultimate and graft each engine's evaluation onto a common search, because many engines compute some parts of their evaluation incrementally within the search. The ideal would be a chess engine with an evaluation that needs access to no information other than the chess position description. But nobody does that since everyone does some sort of king safety analysis that depends on attack information, some sort of mobility analysis that needs the same, etc...

It's a desirable goal, but one that is likely impossible to reach.

Here is my idea of trying to roughly compare evaluation functions:

1) Determine a time per move for each engine so that they score ~50% against each other in normal play.

2) For each piece of chess knowledge being tested, run a boatload of games. For example, to judge how well each engine handles a particular material imbalance (such as the Exchange), collect a large number of positions featuring that material imbalance (using some care in selecting positions that are not too biased for a particular side), and use those positions as the starting positions in the testing.

The hope is that something can be discerned from the noise.

Nothing wrong with that, other than it is a pretty intractable problem, computationally. The SD=1 idea is actually doable in terms of time, but not in terms of actually measuring what you think you are measuring unless you don't mind digging into large programs and trying to equalize their searches, which is not particularly easy.

The problem would become (a) collecting positions for each specific type of "knowledge" and then (b) running the necessary number of games to find the time-handicap that is needed to equalize them. And something tells me even that is going to be a bit of a headache since engines don't vary skill uniformly as time controls change (some play worse at very fast, better at very slow, etc...

[carldaman]

Hi Carl !

They are a lot of private chess engines with a very particular (not to tell fantastic) architecture ! Diep is just one.
Very few persons know something about !
Just two examples:

1.-The Baron (from 3.xx series) is able to play amazing games (even the lost ones) versus very strong engines. Sometimes engines like Stockfish, Komodo, Houdini are KO (20-25 moves) despite their big speed ! I have this engine and I did a lot of experiments !

2.-Lemming - I've heard this engine has only 1kn/s and an huge amount of hash used. And around 2600 Elo. That means a lot of extensions !!!

Stockfish-Komodo-Houdini & co.are only the visible direction of development in computer chess. They are also architectural design in "the dark side" of computer chess programming ! The"black hole" of CC programming from here will strike a day !

Regards,
SilvianR

hola Ruxy,

Well, I've strongly suspected there was more than met the eye. In the big Infinity Chess freestyle event of the past year there was a strong Cryptic (?) engine being used by someone and it played very human-like chess.

You already told me about the latest Baron, and how it's private (*%%%*$#!!?!) and it appears the author (Richard P.) is unwilling to release it.

Was Lemming ever publicly available? I suspect some of its knowledge found its way into the great Hannibal engine, (the other half of Hannibal being based on the fantastic Twisted Logic). Correct me if I'm wrong.

Regards,
CL

[Sylwy]

Hi Carl !

They are a lot of private chess engines with a very particular (not to tell fantastic) architecture ! Diep is just one.
Very few persons know something about !
Just two examples:

1.-The Baron (from 3.xx series) is able to play amazing games (even the lost ones) versus very strong engines. Sometimes engines like Stockfish, Komodo, Houdini are KO (20-25 moves) despite their big speed ! I have this engine and I did a lot of experiments !

2.-Lemming - I've heard this engine has only 1kn/s and an huge amount of hash used. And around 2600 Elo. That means a lot of extensions !!!

Stockfish-Komodo-Houdini & co.are only the visible direction of development in computer chess. They are also architectural design in "the dark side" of computer chess programming ! The"black hole" of CC programming from here will strike a day !

Regards,
SilvianR

hola Ruxy,

Well, I've strongly suspected there was more than met the eye. In the big Infinity Chess freestyle event of the past year there was a strong Cryptic (?) engine being used by someone and it played very human-like chess.

1.-You already told me about the latest Baron, and how it's private (*%%%*$#!!?!) and it appears the author (Richard P.) is unwilling to release it.

2.-Was Lemming ever publicly available? I suspect some of its knowledge found its way into the great Hannibal engine, (the other half of Hannibal being based on the fantastic Twisted Logic). Correct me if I'm wrong.

Regards,
CL

Hi Carl !

1.-I suspect for a free The Baron 3.xx together with a free Diep (the newest class) !

2.-No, never !
Hannibal hasn't any knowledge based on Lemming architecture. Just some learning elements !

And.........don't suspect every private chess engine to be a diamond ! I have a lot of and most of them are in fact extremely weak & buggy ! But among this class are some stars. Brighten stars !

SilvianR

[Stan Arts]

An old post but a good one when it comes to topics like this.:

If you're a slow (knowledge) program, you can beat a fast one by having essential chess knowledge. You maybe find some theme or weakness or king attack or whatever, go for it, sit on it, exploit it and maybe get a win from it. Also, you can find this stuff, but not be able to convert it.

If you're a slow program, and you get into a game where these exploitation possibilities don't exist for some reason, then, effectively, the game turns into slow bean-counter against fast bean-counter, with the inevitable conclusion.

We all see these games. In fact you don't need my program to show them , because they happen all the time in comp-comp. These game types are the norm for bean vs bean.

Take a scenario. Your program now, Ferret, against your program 4 years ago. Or even your program now against your own program on slow hardware. Result inevitable ? Probably. Game style and type ? Probably predictable like so:

Ferret(fast) will have 1,2,3,4 nominal plies on Ferret(slow). Game style and type will be strongly dependant on the nominal ply gap.

a) High gap. Ferret(slow) will likely go down into rapid material collapse. Ferret(fast) may even have some flashy pyrotechnics to demonstrate it. A naive reviewer could call ferret(fast) a spectacular attacking program. He could call ferret(slow) a stupid bean-counter, typical computer.

b) Medium gap. Ferret(fast) will slowly grind ferret(slow) down. Ferret slow will keep finding at its higher iterations, possible loss of material. It
will go panic time, find a way to avoid material loss by giving double pawns instead, or whatever. A naive reviewer will call Ferret(fast) a great
positional player. He'll call Ferret(slow) dumb, accuse it of not having simple knowledge like double pawns, or whatever.

c) Small gap. Probably you'll get reasonable games. The reviewer can't tell much, so, if he'll likely start making things up. Human style, or plays more interesting, or some other nonsense that says nothing.

What I'm trying to say to you, is that Ferret is none of these things. It has none of these 'naive reviewer properties'. The properties are all emergent from the search gap, and therefore depend on the opponent. It knows everything and nothing, all at the same time.

Which is why Genius was thought to be the greatest thing, and now you all think it is boring. It isn't either, or its both. Schrodinger's cat.

Which is why programs seem to keep making progress on the SSDF list. And why reviewers, either dumb, or with axes to grind, wax lyrical about the latest programs.

It's the search gap. Gettit ? Out of this search gap comes all the naive speculation and nonsense that gets written. The program has every style and no style, it has no consistency to play against, only materialism, you can't learn from it, tomorrow it will be different (found another mine in the search gap), only the difference is just a relection of - whoops, trod on another mine. What can you do with such a program ? Use the take-back key and try again ? - and imagine this helps you improve or learn ?

Now, I claim this search gap has no meaning or understanding possibilities for a human. That a human can't relate his heuristics to it. That you can't extract the knowledge out of it and represent it to a human. That you can't even extract the knowledge out of it and represent it to yourself. You can't get heuristics from it. So I call it counting beans - useless for us humans.

Now, take a knowledge program, you can play it and see the play style. You can try and work out what it does and why. There'll be a reason, based on human chess heuristics. The game has plan, and flow, and doesn't consist of hidden minefields. It won't grind you down by search, it will try speculative ideas which it might, or might not, be able to get to work. You can see the speculative ideas, and try them yourself. I think you can, as a human, relate to this type of program. If you know the programmer, maybe you can see patterns into the program that come from him, and so on. I think these types of programs are infused with some force, in so far as any chunk of silicon can be.

I hate materialists.

Chris Whittington

http://www.thorstenczub.de/ihatematerialists.html

[Ferdy]

An old post but a good one when it comes to topics like this.:

http://www.thorstenczub.de/ihatematerialists.html

Good info thanks.

[pkumar]

1 6k1/8/6PP/3B1K2/8/2b5/8/8 b - - 0 1
2 8/8/r5kP/6P1/1R3K2/8/8/8 w - - 0 1
3 7k/R7/7P/6K1/8/8/2b5/8 w - - 0 1
4 8/8/5k2/8/8/4qBB1/6K1/8 w - - 0 1
5 8/8/8/3K4/8/4Q3/2p5/1k6 w - - 0 1
6 8/8/4nn2/4k3/8/Q4K2/8/8 w - - 0 1
7 8/k7/p7/Pr6/K1Q5/8/8/8 w - - 0 1
8 k7/p4R2/P7/1K6/8/6b1/8/8 w - - 0 1

Nice draw positions for fooling engines! Are there some more?

[Ferdy]

Nice draw positions for fooling engines! Are there some more?

Here 3 more difficult positions to bring the score close to zero. I increased the time to 2 sec/position.

Code: Select all

A. Platform:
System   : Windows
Release  : 7
Version  : 6.1.7601
Machine  : AMD64
processor: Intel64 Family 6 Model 42 Stepping 7, GenuineIntel

B. Engine parameters:
Threads  : 1
Hash     : 64mb
Time/pos : 2000ms

C. Test settings:
Total engine count  : 43
Total positions     : 3 (input file: test2.fen)
Total max points    : 300
Estimated total time: 3 pos x 2000ms/pos = 6000 ms

D. Summary high points is better:
 1 id name Komodo 6 64-bit                    (time 5614 ms, Points  35, ratio  11.7%)
 2 id name Fire 4 x64                         (time 2621 ms, Points  24, ratio   8.0%)
 3 id name Naum 4.6                           (time 5680 ms, Points  24, ratio   8.0%)
 4 id name Bouquet 1.8 x64                    (time 6037 ms, Points  19, ratio   6.3%)
 5 id name Andscacs 0.71                      (time 4997 ms, Points  17, ratio   5.7%)
 6 id name cheng4 0.36c                       (time 5504 ms, Points  15, ratio   5.0%)
 7 id name Nebula 2.0                         (time 3651 ms, Points  15, ratio   5.0%)
 8 id name Stockfish 131214 64 POPCNT         (time 5497 ms, Points  10, ratio   3.3%)
 9 id name Arasan 17.4                        (time 6053 ms, Points   9, ratio   3.0%)
10 id name Fruit reloaded 2.1                 (time 2028 ms, Points   9, ratio   3.0%)
11 id name Protector 1.7.0                    (time 5676 ms, Points   8, ratio   2.7%)
12 id name Maverick 0.51 x64                  (time 4728 ms, Points   7, ratio   2.3%)
13 id name Senpai 1.0                         (time 3697 ms, Points   6, ratio   2.0%)
14 id name spark-1.0                          (time 6037 ms, Points   6, ratio   2.0%)
15 id name Hannibal 1.4x64                    (time 4882 ms, Points   5, ratio   1.7%)
16 id name Octochess revision 5190            (time 3486 ms, Points   4, ratio   1.3%)
17 id name Gaviota v1.0                       (time 5194 ms, Points   2, ratio   0.7%)
18 id name GreKo 12.1                         (time 1592 ms, Points   2, ratio   0.7%)
19 id name Vajolet2 1.45                      (time 5344 ms, Points   2, ratio   0.7%)
20 id name DiscoCheck 5.2.1                   (time 4245 ms, Points   1, ratio   0.3%)
21 id name AnMon 5.75                         (time 5242 ms, Points   0, ratio   0.0%)
22 id name Bobcat 3.25                        (time 4992 ms, Points   0, ratio   0.0%)
23 id name Critter 1.6a 64-bit                (time 4848 ms, Points   0, ratio   0.0%)
24 id name Daydreamer 1.75 JA                 (time 4027 ms, Points   0, ratio   0.0%)
25 id name Deuterium v14.3.34.130             (time 5444 ms, Points   0, ratio   0.0%)
26 id name Deuterium v14.4.35.18 64bit POPCNT (time 5303 ms, Points   0, ratio   0.0%)
27 id name DisasterArea-1.54                  (time 4947 ms, Points   0, ratio   0.0%)
28 id name Equinox 3.30 x64mp                 (time 4494 ms, Points   0, ratio   0.0%)
29 id name GNU Chess 5.60-64                  (time 4739 ms, Points   0, ratio   0.0%)
30 id name Gull 3 x64                         (time 6000 ms, Points   0, ratio   0.0%)
31 id name HIARCS 14 WCSC                     (time 5289 ms, Points   0, ratio   0.0%)
32 id name Houdini 4 x64                      (time 6000 ms, Points   0, ratio   0.0%)
33 id name iCE 2.0 v2240 x64/popcnt           (time 1045 ms, Points   0, ratio   0.0%)
34 id name Nemo SP64o 1.0.1 Beta              (time 6000 ms, Points   0, ratio   0.0%)
35 id name Quazar 0.4 x64                     (time 4961 ms, Points   0, ratio   0.0%)
36 id name Rhetoric 1.4.1 x64                 (time 3590 ms, Points   0, ratio   0.0%)
37 id name Rodent 1.6 (build 6)               (time 4788 ms, Points   0, ratio   0.0%)
38 id name Ruffian 1.0.5                      (time 5160 ms, Points   0, ratio   0.0%)
39 id name Rybka 2.3.2a mp                    (time 2920 ms, Points   0, ratio   0.0%)
40 id name Spike 1.4                          (time 5992 ms, Points   0, ratio   0.0%)
41 id name Strelka 6 w32                      (time 6000 ms, Points   0, ratio   0.0%)
42 id name Texel 1.04 64-bit                  (time 4629 ms, Points   0, ratio   0.0%)
43 id name Yace 0.99.87                       (time 5973 ms, Points   0, ratio   0.0%)

E. Positions:
 1 6k1/6Pp/7P/8/3BK3/8/8/8 w - - 0 1
 2 6k1/7p/5K1P/8/8/7P/6P1/8 w - - 0 1
 3 8/8/8/6k1/2q3p1/4R3/5PK1/8 w - - 0 1

F. Point System:
score <= abs&#40;50&#41;  &#58; 100 points
score <= abs&#40;100&#41; &#58; 61 - 70, points
score <= abs&#40;150&#41; &#58; 51 - 60, points
score <= abs&#40;200&#41; &#58; 41 - 50, points
score <= abs&#40;250&#41; &#58; 31 - 40, points
score <= abs&#40;300&#41; &#58; 21 - 30, points
score <= abs&#40;350&#41; &#58; 11 - 20, points
score <= abs&#40;400&#41; &#58;  1 - 10, points
Other scores      &#58;  0 points

G. Engine that does not report time&#58;
 1 id name Gull 3 x64                  
 2 id name Nemo SP64o 1.0.1 Beta

Log from the top performer.

Code: Select all

[engine: 24]
id name Komodo 6 64-bit 

Pos 1
6k1/6Pp/7P/8/3BK3/8/8/8 w - - 0 1

Engine search score +431, time 1965ms
Points earned in this position: 0
Total points update: 0

Pos 2
6k1/7p/5K1P/8/8/7P/6P1/8 w - - 0 1

Engine search score +371, time 1871ms
Points earned in this position: 6
Total points update: 6

Pos 3
8/8/8/6k1/2q3p1/4R3/5PK1/8 w - - 0 1
Engine search score -259, time 1778ms
Points earned in this position: 29

Total points update: 35

Positions.
[d]6k1/6Pp/7P/8/3BK3/8/8/8 w - - 0 1
[d]6k1/7p/5K1P/8/8/7P/6P1/8 w - - 0 1
[d]8/8/8/6k1/2q3p1/4R3/5PK1/8 w - - 0 1

[carldaman]

Now, take a knowledge program, you can play it and see the play style. You can try and work out what it does and why. There'll be a reason, based on human chess heuristics. The game has plan, and flow, and doesn't consist of hidden minefields. It won't grind you down by search, it will try speculative ideas which it might, or might not, be able to get to work. You can see the speculative ideas, and try them yourself. I think you can, as a human, relate to this type of program. If you know the programmer, maybe you can see patterns into the program that come from him, and so on. I think these types of programs are infused with some force, in so far as any chunk of silicon can be.

Chris Whittington

http://www.thorstenczub.de/ihatematerialists.html

I love the above paragraph. Couldn't have said it any better.

A real shame that CSTAL development ceased.

Regards,
CL