Koivisto 2.0

[Luecx]

A while ago, Kim Kåhre and I (Finn Eggers) have published the first version of our engine Koivisto. We have made some very fast progress before that release and we have also had a lot of progress in the last couple of days. That's why we want to publish Koivisto 2.0 today. Besides adding Lazy SMP which scales nicely up to 8 threads and beyond, we fixed a few bugs, improved the speed, and many more things as listed below:

• Added Lazy SMP (artificially limited to 32 threads)

• Added UCI bug fixes to work better with GUI's as it could crash if e.g. "stop" is called without anything running

• Fixed all compiler warnings

• Several speed improvements all over the engine

• evaluation improvements by e.g. retuning the psqt

We also worked day and night on bringing you some binaries, yet for Windows, there seems to be an issue with link time optimizations(LTO) together with static executables.
Removing LTO results in a massive drop in nps so we do not provide binaries yet so this has to wait a little longer unless someone of you could solve these compiling issues.

Our testing has show considerable elo gain over Koivisto 1.0:

Code: Select all

(120s+1s on ryzen 3950x)
Score of Koivisto Dev vs Koivisto 1.0: 143 - 59 - 167 [0.614]
...      Koivisto Dev playing White: 84 - 28 - 72  [0.652] 184
...      Koivisto Dev playing Black: 59 - 31 - 95  [0.576] 185
...      White vs Black: 115 - 87 - 167  [0.538] 369
Elo difference: 80.5 +/- 26.3, LOS: 100.0 %, DrawRatio: 45.3 %

Furthermore, Koivisto 2.0 on 4 Threads seems to be about 70 Elo stronger than Koivisto 2.0 on 1 thread in self-play.

There are still a lot of basic things like Aspiration Windows which have not been implemented yet so there will be a lot of progress in the future. Besides improving the search and classical evaluation functions, we plan on doing our own NN implementation. And with that, we mean 100% code self-written using AVX256 instructions and not copied from SF. This also includes training code for the GPU etc. A lot of this code has already been written, yet developing will take some time and we are open for advice!

The easiest way to contact us would be our Discord: (Finn) Luecx#0540 or (Kim) ChiefWiggum#1154

https://github.com/Luecx/Koivisto/releases/tag/v2.0

We have received immense help from other community members/engine authors and they deserve a shoutout:
- Eugenio Bruno https://github.com/Eugenio-Bruno
- Andrew Grant https://github.com/AndyGrant/Ethereal
- Mk-Chan https://github.com/Mk-Chan
- Terje Kirstihagen https://github.com/TerjeKir
- kz04px https://github.com/kz04px

[supersharp77]

A while ago, Kim Kåhre and I (Finn Eggers) have published the first version of our engine Koivisto. We have made some very fast progress before that release and we have also had a lot of progress in the last couple of days. That's why we want to publish Koivisto 2.0 today. Besides adding Lazy SMP which scales nicely up to 8 threads and beyond, we fixed a few bugs, improved the speed, and many more things as listed below:

• Added Lazy SMP (artificially limited to 32 threads)

• Added UCI bug fixes to work better with GUI's as it could crash if e.g. "stop" is called without anything running

• Fixed all compiler warnings

• Several speed improvements all over the engine

• evaluation improvements by e.g. retuning the psqt

We also worked day and night on bringing you some binaries, yet for Windows, there seems to be an issue with link time optimizations(LTO) together with static executables.
Removing LTO results in a massive drop in nps so we do not provide binaries yet so this has to wait a little longer unless someone of you could solve these compiling issues.

Our testing has show considerable elo gain over Koivisto 1.0:

Code: Select all

(120s+1s on ryzen 3950x)
Score of Koivisto Dev vs Koivisto 1.0: 143 - 59 - 167 [0.614]
...      Koivisto Dev playing White: 84 - 28 - 72  [0.652] 184
...      Koivisto Dev playing Black: 59 - 31 - 95  [0.576] 185
...      White vs Black: 115 - 87 - 167  [0.538] 369
Elo difference: 80.5 +/- 26.3, LOS: 100.0 %, DrawRatio: 45.3 %

Furthermore, Koivisto 2.0 on 4 Threads seems to be about 70 Elo stronger than Koivisto 2.0 on 1 thread in self-play.

There are still a lot of basic things like Aspiration Windows which have not been implemented yet so there will be a lot of progress in the future. Besides improving the search and classical evaluation functions, we plan on doing our own NN implementation. And with that, we mean 100% code self-written using AVX256 instructions and not copied from SF. This also includes training code for the GPU etc. A lot of this code has already been written, yet developing will take some time and we are open for advice!

The easiest way to contact us would be our Discord: (Finn) Luecx#0540 or (Kim) ChiefWiggum#1154

Whoa..Whoa... just a minute my friend...no windows binaries have been produced on v1.0 so far (or 2.0)... and windows searches on Koivisto bring up absolutely nothing.. so who is doing the testing...we need windows binaries for popcont and npopcount...Thank you AR

[Jouni]

Engine's finnish name Koivisto surprises me! Has it any connection to Finland?

[Luecx]

Engine's finnish name Koivisto surprises me! Has it any connection to Finland?

Well Kim lives in Finland and my name is Finn. So we decided to take something Finnish

[Luecx]

A while ago, Kim Kåhre and I (Finn Eggers) have published the first version of our engine Koivisto. We have made some very fast progress before that release and we have also had a lot of progress in the last couple of days. That's why we want to publish Koivisto 2.0 today. Besides adding Lazy SMP which scales nicely up to 8 threads and beyond, we fixed a few bugs, improved the speed, and many more things as listed below:

• Added Lazy SMP (artificially limited to 32 threads)

• Added UCI bug fixes to work better with GUI's as it could crash if e.g. "stop" is called without anything running

• Fixed all compiler warnings

• Several speed improvements all over the engine

• evaluation improvements by e.g. retuning the psqt

We also worked day and night on bringing you some binaries, yet for Windows, there seems to be an issue with link time optimizations(LTO) together with static executables.
Removing LTO results in a massive drop in nps so we do not provide binaries yet so this has to wait a little longer unless someone of you could solve these compiling issues.

Our testing has show considerable elo gain over Koivisto 1.0:

Code: Select all

(120s+1s on ryzen 3950x)
Score of Koivisto Dev vs Koivisto 1.0: 143 - 59 - 167 [0.614]
...      Koivisto Dev playing White: 84 - 28 - 72  [0.652] 184
...      Koivisto Dev playing Black: 59 - 31 - 95  [0.576] 185
...      White vs Black: 115 - 87 - 167  [0.538] 369
Elo difference: 80.5 +/- 26.3, LOS: 100.0 %, DrawRatio: 45.3 %

Furthermore, Koivisto 2.0 on 4 Threads seems to be about 70 Elo stronger than Koivisto 2.0 on 1 thread in self-play.

There are still a lot of basic things like Aspiration Windows which have not been implemented yet so there will be a lot of progress in the future. Besides improving the search and classical evaluation functions, we plan on doing our own NN implementation. And with that, we mean 100% code self-written using AVX256 instructions and not copied from SF. This also includes training code for the GPU etc. A lot of this code has already been written, yet developing will take some time and we are open for advice!

The easiest way to contact us would be our Discord: (Finn) Luecx#0540 or (Kim) ChiefWiggum#1154

Whoa..Whoa... just a minute my friend...no windows binaries have been produced on v1.0 so far (or 2.0)... and windows searches on Koivisto bring up absolutely nothing.. so who is doing the testing...we need windows binaries for popcont and npopcount...Thank you AR

Koivisto v1.0 is currently being tested. The only solution to test it at the moment is to compile it yourself using make. Assuming make is installed, it’s just a single command as you can see on the GitHub page. We have been working very hard to bring you some binaries but we first need to resolve some massive LTO problems

[Guenther]

A while ago, Kim Kåhre and I (Finn Eggers) have published the first version of our engine Koivisto. We have made some very fast progress before that release and we have also had a lot of progress in the last couple of days. That's why we want to publish Koivisto 2.0 today. Besides adding Lazy SMP which scales nicely up to 8 threads and beyond, we fixed a few bugs, improved the speed, and many more things as listed below:

...

The easiest way to contact us would be our Discord: (Finn) Luecx#0540 or (Kim) ChiefWiggum#1154

Whoa..Whoa... just a minute my friend...no windows binaries have been produced on v1.0 so far (or 2.0)... and windows searches on Koivisto bring up absolutely nothing.. so who is doing the testing...we need windows binaries for popcont and npopcount...Thank you AR :) :wink:

You can try those on your own risk from here (bmi2 completely untested - I can only run the 'old' ones on my hardware = no popcnt/core2)
(I noticed that on my slow old hardware Koivisto 2 runs now at around 740kn/s compared to around 600kn/s for version 1 - checked for startpos)
https://rwbc-chess.de/download.htm

[Gabor Szots]

Finn, I am a beginner in compiling so I'd like to ask for your help.

This is your Makefile:

CC = g++
#SRC = *.cpp syzygy/*.c nn/*.cpp nn/data/*.cpp
SRC = *.cpp syzygy/tbprobe.c
LIBS = -lpthread -lm
EXE = ../Koivisto

WFLAGS = -std=c++17 -Wall -Wextra -Wshadow
CFLAGS = -O3 $(WFLAGS) -DNDEBUG -flto -march=native
RFLAGS = -O3 $(WFLAGS) -DNDEBUG -flto -static
TFLAGS = -O3 $(WFLAGS) -DNDEBUG -flto -march=native -fopenmp -DTUNE
PFLAGS = -O0 $(WFLAGS) -DNDEBUG -p -pg
DFLAGS = -O0 $(WFLAGS) -g

POPCNTFLAGS = -DUSE_POPCNT -msse3 -mpopcnt
PEXTFLAGS = $(POPCNTFLAGS) -DUSE_PEXT -mbmi2



release:
$(CC) $(CFLAGS) $(SRC) $(LIBS) $(POPCNTFLAGS) -o $(EXE)

My questions:
1. What does #SRC do? There is no nn directory and I can't see where it is used.
2. What are WFLAGS, RFLAGS, TFLAGS, PFLAGS, DFLAGS and PEXTFLAGS expected to do? No reference to either of them.
3. When CFLAGS already includes -march=native then why do we need POPCNTFLAGS at all?

I think it would be welcomed if you included your own compile of Koivisto in the download package, whatever hardware it requires.

BTW, under what flag shall Koivisto sail? I understand Kim is from Finnland but as far as I know you haven't revealed your nationality.

[Luecx]

My questions:
1. What does #SRC do? There is no nn directory and I can't see where it is used.
2. What are WFLAGS, RFLAGS, TFLAGS, PFLAGS, DFLAGS and PEXTFLAGS expected to do? No reference to either of them.
3. When CFLAGS already includes -march=native then why do we need POPCNTFLAGS at all?

I think it would be welcomed if you included your own compile of Koivisto in the download package, whatever hardware it requires.

BTW, under what flag shall Koivisto sail? I understand Kim is from Finnland but as far as I know you haven't revealed your nationality.

All of those flags are mostly used for debugging and testing purpose. For example when we want to debug the code, we would use DFLAGS.
The #SRC flag is used in the future once the NN code will be included. I actually changed the directory name to "eun" but forgot to adjust that in the makefile. You can mostly ignore all the other flags. Also you are right that march=native and the POPCNTFLAGS are redundant.

I am german. Is it possible to have some international flag... Like EU?

[Luecx]

A while ago, Kim Kåhre and I (Finn Eggers) have published the first version of our engine Koivisto. We have made some very fast progress before that release and we have also had a lot of progress in the last couple of days. That's why we want to publish Koivisto 2.0 today. Besides adding Lazy SMP which scales nicely up to 8 threads and beyond, we fixed a few bugs, improved the speed, and many more things as listed below:

...

The easiest way to contact us would be our Discord: (Finn) Luecx#0540 or (Kim) ChiefWiggum#1154

Whoa..Whoa... just a minute my friend...no windows binaries have been produced on v1.0 so far (or 2.0)... and windows searches on Koivisto bring up absolutely nothing.. so who is doing the testing...we need windows binaries for popcont and npopcount...Thank you AR

You can try those on your own risk from here (bmi2 completely untested - I can only run the 'old' ones on my hardware = no popcnt/core2)
(I noticed that on my slow old hardware Koivisto 2 runs now at around 740kn/s compared to around 600kn/s for version 1 - checked for startpos)
https://rwbc-chess.de/download.htm

We should perhaps adjust the makefile for those type of hardwares. On my hardware, Koivisto gets about 2.5Mnps there.

[Guenther]

A while ago, Kim Kåhre and I (Finn Eggers) have published the first version of our engine Koivisto. We have made some very fast progress before that release and we have also had a lot of progress in the last couple of days. That's why we want to publish Koivisto 2.0 today. Besides adding Lazy SMP which scales nicely up to 8 threads and beyond, we fixed a few bugs, improved the speed, and many more things as listed below:

...

The easiest way to contact us would be our Discord: (Finn) Luecx#0540 or (Kim) ChiefWiggum#1154

Whoa..Whoa... just a minute my friend...no windows binaries have been produced on v1.0 so far (or 2.0)... and windows searches on Koivisto bring up absolutely nothing.. so who is doing the testing...we need windows binaries for popcont and npopcount...Thank you AR :) :wink:

You can try those on your own risk from here (bmi2 completely untested - I can only run the 'old' ones on my hardware = no popcnt/core2)
(I noticed that on my slow old hardware Koivisto 2 runs now at around 740kn/s compared to around 600kn/s for version 1 - checked for startpos)
https://rwbc-chess.de/download.htm

We should perhaps adjust the makefile for those type of hardwares. On my hardware, Koivisto gets about 2.5Mnps there.

How fast is e.g. vanilla SF 11 for your hardware from startpos?
Let's say around depth 25-30, here it is around 640 kn/s.