Mysterious Vajolet chess engine

[Werner]

From our 40/20 tests I have following results under these conditions:
Win 10 64bit; Skylake processor 4CPU; Shredder GUI; engine 1 time installed and started Shredder GUI 4 times (4 parallel matches)

Code: Select all

Vajolet2 2.1 x64 1CPU   - Arasan 18.2 x64 1CPU           40.0 - 60.0    +21/=38/-41    40.00%
Vajolet2 2.1 x64 1CPU   - Deuterium v14.3.34.130 x64     37.5 - 62.5    +18/=39/-43    37.50%
Vajolet2 2.1 x64 1CPU   - Komodo 1.2 w32                 41.0 - 59.0    +17/=48/-35    41.00%

Now I repeated these matches on running only 1 Vajolet match each time - but also 4 matches parallel (with other engines like iCE, SmarThink, Stockfish):

Code: Select all

Vajolet2 2.1 x64 1CPU   - Deuterium v14.3.34.130 x64     52.5 - 47.5    +34/=37/-29    52.50%
Vajolet2 2.1 x64 1CPU   - Arasan 18.2 x64 1CPU           43.5 - 56.5    +19/=49/-32    43.50%
Vajolet2 2.1 x64 1CPU   - Komodo 1.2 w32                 50.0 - 50.0    +30/=40/-30    50.00%

Of course I used the same openings. The elo difference are over 100 points - and the new results are like the awaited scores.

So, is this coincidence or is it possible some reason hurts Vajolet under the first conditions? Is it possible, the engine writes and reads values from the same cell of the ram or another hidden file?
Would it be better when I make 4 matches parallel to install the engine 4 times from different folders?

The answers can help me to make better test conditions. I did not see such a behaviour with other engines.

best wishes
Werner

[tpetzke]

Hi Werner,

I would not draw conclusions yet, the sample size is to small. The error bar for 300 games in a gauntlet is huge. Much larger than you would run the old and the new version against each other.

Thomas...

[elcabesa]

Vajolet is not using any external file or resource except the opening book if present.
It should only use the memory allocate for the process.

I don't expect the big variation, let's see what it will happen with longer test, the number of games is not so high.

I haven't experienced your problem.

[bob]

From our 40/20 tests I have following results under these conditions:
Win 10 64bit; Skylake processor 4CPU; Shredder GUI; engine 1 time installed and started Shredder GUI 4 times (4 parallel matches)

Code: Select all

Vajolet2 2.1 x64 1CPU   - Arasan 18.2 x64 1CPU           40.0 - 60.0    +21/=38/-41    40.00%
Vajolet2 2.1 x64 1CPU   - Deuterium v14.3.34.130 x64     37.5 - 62.5    +18/=39/-43    37.50%
Vajolet2 2.1 x64 1CPU   - Komodo 1.2 w32                 41.0 - 59.0    +17/=48/-35    41.00%

Now I repeated these matches on running only 1 Vajolet match each time - but also 4 matches parallel (with other engines like iCE, SmarThink, Stockfish):

Code: Select all

Vajolet2 2.1 x64 1CPU   - Deuterium v14.3.34.130 x64     52.5 - 47.5    +34/=37/-29    52.50%
Vajolet2 2.1 x64 1CPU   - Arasan 18.2 x64 1CPU           43.5 - 56.5    +19/=49/-32    43.50%
Vajolet2 2.1 x64 1CPU   - Komodo 1.2 w32                 50.0 - 50.0    +30/=40/-30    50.00%

Of course I used the same openings. The elo difference are over 100 points - and the new results are like the awaited scores.

So, is this coincidence or is it possible some reason hurts Vajolet under the first conditions? Is it possible, the engine writes and reads values from the same cell of the ram or another hidden file?
Would it be better when I make 4 matches parallel to install the engine 4 times from different folders?

The answers can help me to make better test conditions. I did not see such a behaviour with other engines.

best wishes
Werner

Any chance it uses more than one thread by default???

[elcabesa]

if not specified vajolet should use only one thread for searching.

it use 3 thread for search, one for uci INPUT bloacked waiting for user input and a thread for the timer sleeping almost all the time

[elcabesa]

I'm trying some tests

[elcabesa]

I made a simple test:

vajolet against ice-2.0, 100 round, 40Moves/120s

I repeated the test 4 times, those are the results:
W-L-D result elo
27-33-40 47% -21 elo
24-26-50 49% -7 elo
20-36.44 42% -56 elo
33-24-44 54% 28 elo

Maybe Vajolet has some problem but I think 100 games aren't enough to draw any conclusion.

that said I think vajolet2 2.1 is as strong as vajolet2 2.0 so you can see some test where it's stronger than the older and vice versa.

[Werner]

Thanks Marco,
the question for me was: what happens, when I install the engine 1 time in Shredder and then start the gui 4 times with 4 parallel matches with the engine.
My results with this setting have been very bad. So I will delete these games:

Code: Select all

Vajolet2 2.1 x64 1CPU   - Crafty 25.0 x64 1CPU           52.5 - 47.5    +33/=39/-28    52.50%
Vajolet2 2.1 x64 1CPU   - ExChess v7.88b x64 1CPU        51.5 - 48.5    +28/=47/-25    51.50%
Vajolet2 2.1 x64 1CPU   - Deep Shredder 12 x64 1CPU      40.5 - 59.5    +22/=37/-41    40.50%
Vajolet2 2.1 x64 1CPU   - Arasan 18.2 x64 1CPU           40.0 - 60.0    +21/=38/-41    40.00%
Vajolet2 2.1 x64 1CPU   - Deep Junior 13.3 x64 1CPU      42.0 - 58.0    +21/=42/-37    42.00%
Vajolet2 2.1 x64 1CPU   - Komodo 1.2 w32                 41.0 - 59.0    +17/=48/-35    41.00%
Vajolet2 2.1 x64 1CPU   - Deuterium v14.3.34.130 x64     37.5 - 62.5    +18/=39/-43    37.50%
Vajolet2 2.1 x64 1CPU   - DiscoCheck 5.2 x64             41.5 - 58.5    +21/=41/-38    41.50%

So I repeated these matches running only 1 time Vajolet on a pc - with much better results:

Code: Select all

Vajolet2 2.1 x64 1CPU   - Komodo 1.2 w32                 50.0 - 50.0    +30/=40/-30    50.00%
Vajolet2 2.1 x64 1CPU   - Deep Junior 13.3 x64 1CPU      51.0 - 49.0    +29/=44/-27    51.00%
Vajolet2 2.1 x64 1CPU   - Deep Shredder 12 x64 1CPU      53.0 - 47.0    +30/=46/-24    53.00%
Vajolet2 2.1 x64 1CPU   - Deuterium v14.3.34.130 x64     52.5 - 47.5    +34/=37/-29    52.50%
Vajolet2 2.1 x64 1CPU   - Arasan 18.2 x64 1CPU           43.5 - 56.5    +19/=49/-32    43.50%
Vajolet2 2.1 x64 1CPU   - ExChess v7.88b x64 1CPU        120.5 - 79.5    +72/=97/-31    60.25%
Vajolet2 2.1 x64 1CPU   - Crafty 25.0 x64 1CPU           51.5 - 48.5    +30/=43/-27    51.50%
Vajolet2 2.1 x64 1CPU   - DiscoCheck 5.2 x64             46.5 - 53.5    +25/=43/-32    46.50%

These games are now used for our list. The reason is not clear for me. Perhaps there was something with the pc during the first match (e.g. a crashed engine was running too...)

[bob]

Thanks Marco,
the question for me was: what happens, when I install the engine 1 time in Shredder and then start the gui 4 times with 4 parallel matches with the engine.
My results with this setting have been very bad. So I will delete these games:

Code: Select all

Vajolet2 2.1 x64 1CPU   - Crafty 25.0 x64 1CPU           52.5 - 47.5    +33/=39/-28    52.50%
Vajolet2 2.1 x64 1CPU   - ExChess v7.88b x64 1CPU        51.5 - 48.5    +28/=47/-25    51.50%
Vajolet2 2.1 x64 1CPU   - Deep Shredder 12 x64 1CPU      40.5 - 59.5    +22/=37/-41    40.50%
Vajolet2 2.1 x64 1CPU   - Arasan 18.2 x64 1CPU           40.0 - 60.0    +21/=38/-41    40.00%
Vajolet2 2.1 x64 1CPU   - Deep Junior 13.3 x64 1CPU      42.0 - 58.0    +21/=42/-37    42.00%
Vajolet2 2.1 x64 1CPU   - Komodo 1.2 w32                 41.0 - 59.0    +17/=48/-35    41.00%
Vajolet2 2.1 x64 1CPU   - Deuterium v14.3.34.130 x64     37.5 - 62.5    +18/=39/-43    37.50%
Vajolet2 2.1 x64 1CPU   - DiscoCheck 5.2 x64             41.5 - 58.5    +21/=41/-38    41.50%

So I repeated these matches running only 1 time Vajolet on a pc - with much better results:

Code: Select all

Vajolet2 2.1 x64 1CPU   - Komodo 1.2 w32                 50.0 - 50.0    +30/=40/-30    50.00%
Vajolet2 2.1 x64 1CPU   - Deep Junior 13.3 x64 1CPU      51.0 - 49.0    +29/=44/-27    51.00%
Vajolet2 2.1 x64 1CPU   - Deep Shredder 12 x64 1CPU      53.0 - 47.0    +30/=46/-24    53.00%
Vajolet2 2.1 x64 1CPU   - Deuterium v14.3.34.130 x64     52.5 - 47.5    +34/=37/-29    52.50%
Vajolet2 2.1 x64 1CPU   - Arasan 18.2 x64 1CPU           43.5 - 56.5    +19/=49/-32    43.50%
Vajolet2 2.1 x64 1CPU   - ExChess v7.88b x64 1CPU        120.5 - 79.5    +72/=97/-31    60.25%
Vajolet2 2.1 x64 1CPU   - Crafty 25.0 x64 1CPU           51.5 - 48.5    +30/=43/-27    51.50%
Vajolet2 2.1 x64 1CPU   - DiscoCheck 5.2 x64             46.5 - 53.5    +25/=43/-32    46.50%

These games are now used for our list. The reason is not clear for me. Perhaps there was something with the pc during the first match (e.g. a crashed engine was running too...)

I assume you are watching memory usage? If a program uses more memory that told and you run N copies on the same machine, things will go wrong. Ditto for multiple threads.

[Werner]

I assume you are watching memory usage? If a program uses more memory that told and you run N copies on the same machine, things will go wrong. Ditto for multiple threads.

Yes, of course - the taskmanager is always running in the background. But if 4 engines are running on the quad it is not easy to see when one of them has crashed and runs permanently as 5th engine...
I only see it, when 1 match has finished and there are still 4 threads busy.