Koivisto 2.0

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Luecx
Posts: 25
Joined: Thu Jun 18, 2020 7:20 pm
Full name: Finn Eggers

Koivisto 2.0

Post by Luecx » Tue Sep 08, 2020 8:22 pm

A while ago, Kim Kåhre and I (Finn Eggers) have published the first version of our engine Koivisto. We have made some very fast progress before that release and we have also had a lot of progress in the last couple of days. That's why we want to publish Koivisto 2.0 today. Besides adding Lazy SMP which scales nicely up to 8 threads and beyond, we fixed a few bugs, improved the speed, and many more things as listed below:
  • Added Lazy SMP (artificially limited to 32 threads)
  • Added UCI bug fixes to work better with GUI's as it could crash if e.g. "stop" is called without anything running
  • Fixed all compiler warnings
  • Several speed improvements all over the engine
  • evaluation improvements by e.g. retuning the psqt
We also worked day and night on bringing you some binaries, yet for Windows, there seems to be an issue with link time optimizations(LTO) together with static executables.
Removing LTO results in a massive drop in nps so we do not provide binaries yet so this has to wait a little longer unless someone of you could solve these compiling issues.

Our testing has show considerable elo gain over Koivisto 1.0:

Code: Select all

(120s+1s on ryzen 3950x)
Score of Koivisto Dev vs Koivisto 1.0: 143 - 59 - 167 [0.614]
...      Koivisto Dev playing White: 84 - 28 - 72  [0.652] 184
...      Koivisto Dev playing Black: 59 - 31 - 95  [0.576] 185
...      White vs Black: 115 - 87 - 167  [0.538] 369
Elo difference: 80.5 +/- 26.3, LOS: 100.0 %, DrawRatio: 45.3 %
Furthermore, Koivisto 2.0 on 4 Threads seems to be about 70 Elo stronger than Koivisto 2.0 on 1 thread in self-play.

There are still a lot of basic things like Aspiration Windows which have not been implemented yet so there will be a lot of progress in the future. Besides improving the search and classical evaluation functions, we plan on doing our own NN implementation. And with that, we mean 100% code self-written using AVX256 instructions and not copied from SF. This also includes training code for the GPU etc. A lot of this code has already been written, yet developing will take some time and we are open for advice!

The easiest way to contact us would be our Discord: (Finn) Luecx#0540 or (Kim) ChiefWiggum#1154

https://github.com/Luecx/Koivisto/releases/tag/v2.0

We have received immense help from other community members/engine authors and they deserve a shoutout:
- Eugenio Bruno https://github.com/Eugenio-Bruno
- Andrew Grant https://github.com/AndyGrant/Ethereal
- Mk-Chan https://github.com/Mk-Chan
- Terje Kirstihagen https://github.com/TerjeKir
- kz04px https://github.com/kz04px

supersharp77
Posts: 991
Joined: Sat Jul 05, 2014 5:54 am
Location: Southwest USA

Re: Koivisto 2.0

Post by supersharp77 » Wed Sep 09, 2020 3:29 am

Luecx wrote:
Tue Sep 08, 2020 8:22 pm
A while ago, Kim Kåhre and I (Finn Eggers) have published the first version of our engine Koivisto. We have made some very fast progress before that release and we have also had a lot of progress in the last couple of days. That's why we want to publish Koivisto 2.0 today. Besides adding Lazy SMP which scales nicely up to 8 threads and beyond, we fixed a few bugs, improved the speed, and many more things as listed below:
  • Added Lazy SMP (artificially limited to 32 threads)
  • Added UCI bug fixes to work better with GUI's as it could crash if e.g. "stop" is called without anything running
  • Fixed all compiler warnings
  • Several speed improvements all over the engine
  • evaluation improvements by e.g. retuning the psqt
We also worked day and night on bringing you some binaries, yet for Windows, there seems to be an issue with link time optimizations(LTO) together with static executables.
Removing LTO results in a massive drop in nps so we do not provide binaries yet so this has to wait a little longer unless someone of you could solve these compiling issues.

Our testing has show considerable elo gain over Koivisto 1.0:

Code: Select all

(120s+1s on ryzen 3950x)
Score of Koivisto Dev vs Koivisto 1.0: 143 - 59 - 167 [0.614]
...      Koivisto Dev playing White: 84 - 28 - 72  [0.652] 184
...      Koivisto Dev playing Black: 59 - 31 - 95  [0.576] 185
...      White vs Black: 115 - 87 - 167  [0.538] 369
Elo difference: 80.5 +/- 26.3, LOS: 100.0 %, DrawRatio: 45.3 %
Furthermore, Koivisto 2.0 on 4 Threads seems to be about 70 Elo stronger than Koivisto 2.0 on 1 thread in self-play.

There are still a lot of basic things like Aspiration Windows which have not been implemented yet so there will be a lot of progress in the future. Besides improving the search and classical evaluation functions, we plan on doing our own NN implementation. And with that, we mean 100% code self-written using AVX256 instructions and not copied from SF. This also includes training code for the GPU etc. A lot of this code has already been written, yet developing will take some time and we are open for advice!

The easiest way to contact us would be our Discord: (Finn) Luecx#0540 or (Kim) ChiefWiggum#1154
Whoa..Whoa... just a minute my friend...no windows binaries have been produced on v1.0 so far (or 2.0)... and windows searches on Koivisto bring up absolutely nothing.. so who is doing the testing...we need windows binaries for popcont and npopcount...Thank you AR :) :wink:

Jouni
Posts: 2387
Joined: Wed Mar 08, 2006 7:15 pm

Re: Koivisto 2.0

Post by Jouni » Wed Sep 09, 2020 6:14 am

Engine's finnish name Koivisto surprises me! Has it any connection to Finland?
Jouni

Luecx
Posts: 25
Joined: Thu Jun 18, 2020 7:20 pm
Full name: Finn Eggers

Re: Koivisto 2.0

Post by Luecx » Wed Sep 09, 2020 7:05 am

Jouni wrote:
Wed Sep 09, 2020 6:14 am
Engine's finnish name Koivisto surprises me! Has it any connection to Finland?
Well Kim lives in Finland and my name is Finn. So we decided to take something Finnish :)

Luecx
Posts: 25
Joined: Thu Jun 18, 2020 7:20 pm
Full name: Finn Eggers

Re: Koivisto 2.0

Post by Luecx » Wed Sep 09, 2020 7:07 am

supersharp77 wrote:
Wed Sep 09, 2020 3:29 am
Luecx wrote:
Tue Sep 08, 2020 8:22 pm
A while ago, Kim Kåhre and I (Finn Eggers) have published the first version of our engine Koivisto. We have made some very fast progress before that release and we have also had a lot of progress in the last couple of days. That's why we want to publish Koivisto 2.0 today. Besides adding Lazy SMP which scales nicely up to 8 threads and beyond, we fixed a few bugs, improved the speed, and many more things as listed below:
  • Added Lazy SMP (artificially limited to 32 threads)
  • Added UCI bug fixes to work better with GUI's as it could crash if e.g. "stop" is called without anything running
  • Fixed all compiler warnings
  • Several speed improvements all over the engine
  • evaluation improvements by e.g. retuning the psqt
We also worked day and night on bringing you some binaries, yet for Windows, there seems to be an issue with link time optimizations(LTO) together with static executables.
Removing LTO results in a massive drop in nps so we do not provide binaries yet so this has to wait a little longer unless someone of you could solve these compiling issues.

Our testing has show considerable elo gain over Koivisto 1.0:

Code: Select all

(120s+1s on ryzen 3950x)
Score of Koivisto Dev vs Koivisto 1.0: 143 - 59 - 167 [0.614]
...      Koivisto Dev playing White: 84 - 28 - 72  [0.652] 184
...      Koivisto Dev playing Black: 59 - 31 - 95  [0.576] 185
...      White vs Black: 115 - 87 - 167  [0.538] 369
Elo difference: 80.5 +/- 26.3, LOS: 100.0 %, DrawRatio: 45.3 %
Furthermore, Koivisto 2.0 on 4 Threads seems to be about 70 Elo stronger than Koivisto 2.0 on 1 thread in self-play.

There are still a lot of basic things like Aspiration Windows which have not been implemented yet so there will be a lot of progress in the future. Besides improving the search and classical evaluation functions, we plan on doing our own NN implementation. And with that, we mean 100% code self-written using AVX256 instructions and not copied from SF. This also includes training code for the GPU etc. A lot of this code has already been written, yet developing will take some time and we are open for advice!

The easiest way to contact us would be our Discord: (Finn) Luecx#0540 or (Kim) ChiefWiggum#1154
Whoa..Whoa... just a minute my friend...no windows binaries have been produced on v1.0 so far (or 2.0)... and windows searches on Koivisto bring up absolutely nothing.. so who is doing the testing...we need windows binaries for popcont and npopcount...Thank you AR :) :wink:
Koivisto v1.0 is currently being tested. The only solution to test it at the moment is to compile it yourself using make. Assuming make is installed, it’s just a single command as you can see on the GitHub page. We have been working very hard to bring you some binaries but we first need to resolve some massive LTO problems

User avatar
Guenther
Posts: 3809
Joined: Wed Oct 01, 2008 4:33 am
Location: Regensburg, Germany
Full name: Guenther Simon
Contact:

Re: Koivisto 2.0

Post by Guenther » Wed Sep 09, 2020 11:38 am

supersharp77 wrote:
Wed Sep 09, 2020 3:29 am
Luecx wrote:
Tue Sep 08, 2020 8:22 pm
A while ago, Kim Kåhre and I (Finn Eggers) have published the first version of our engine Koivisto. We have made some very fast progress before that release and we have also had a lot of progress in the last couple of days. That's why we want to publish Koivisto 2.0 today. Besides adding Lazy SMP which scales nicely up to 8 threads and beyond, we fixed a few bugs, improved the speed, and many more things as listed below:

...

The easiest way to contact us would be our Discord: (Finn) Luecx#0540 or (Kim) ChiefWiggum#1154
Whoa..Whoa... just a minute my friend...no windows binaries have been produced on v1.0 so far (or 2.0)... and windows searches on Koivisto bring up absolutely nothing.. so who is doing the testing...we need windows binaries for popcont and npopcount...Thank you AR :) :wink:
You can try those on your own risk from here (bmi2 completely untested - I can only run the 'old' ones on my hardware = no popcnt/core2)
(I noticed that on my slow old hardware Koivisto 2 runs now at around 740kn/s compared to around 600kn/s for version 1 - checked for startpos)
https://rwbc-chess.de/download.htm
https://rwbc-chess.de
HGM@'chessqueen' 2018-present, aka: 'George' 2013-2016, 'pichy' 2006-2013, 'Jorge Pichard' 2000-2006 (old forum) wrote: http://talkchess.com/forum3/viewtopic.p ... 79#p789713

Gabor Szots
Posts: 757
Joined: Sat Jul 21, 2018 5:43 am
Location: Szentendre, Hungary
Full name: Gabor Szots

Re: Koivisto 2.0

Post by Gabor Szots » Wed Sep 09, 2020 11:58 am

Finn, I am a beginner in compiling so I'd like to ask for your help.

This is your Makefile:
CC = g++
#SRC = *.cpp syzygy/*.c nn/*.cpp nn/data/*.cpp
SRC = *.cpp syzygy/tbprobe.c
LIBS = -lpthread -lm
EXE = ../Koivisto

WFLAGS = -std=c++17 -Wall -Wextra -Wshadow
CFLAGS = -O3 $(WFLAGS) -DNDEBUG -flto -march=native
RFLAGS = -O3 $(WFLAGS) -DNDEBUG -flto -static
TFLAGS = -O3 $(WFLAGS) -DNDEBUG -flto -march=native -fopenmp -DTUNE
PFLAGS = -O0 $(WFLAGS) -DNDEBUG -p -pg
DFLAGS = -O0 $(WFLAGS) -g

POPCNTFLAGS = -DUSE_POPCNT -msse3 -mpopcnt
PEXTFLAGS = $(POPCNTFLAGS) -DUSE_PEXT -mbmi2



release:
$(CC) $(CFLAGS) $(SRC) $(LIBS) $(POPCNTFLAGS) -o $(EXE)
My questions:
1. What does #SRC do? There is no nn directory and I can't see where it is used.
2. What are WFLAGS, RFLAGS, TFLAGS, PFLAGS, DFLAGS and PEXTFLAGS expected to do? No reference to either of them.
3. When CFLAGS already includes -march=native then why do we need POPCNTFLAGS at all?

I think it would be welcomed if you included your own compile of Koivisto in the download package, whatever hardware it requires.

BTW, under what flag shall Koivisto sail? I understand Kim is from Finnland but as far as I know you haven't revealed your nationality.
Gabor Szots
CCRL testing group

Luecx
Posts: 25
Joined: Thu Jun 18, 2020 7:20 pm
Full name: Finn Eggers

Re: Koivisto 2.0

Post by Luecx » Wed Sep 09, 2020 12:35 pm

My questions:
1. What does #SRC do? There is no nn directory and I can't see where it is used.
2. What are WFLAGS, RFLAGS, TFLAGS, PFLAGS, DFLAGS and PEXTFLAGS expected to do? No reference to either of them.
3. When CFLAGS already includes -march=native then why do we need POPCNTFLAGS at all?

I think it would be welcomed if you included your own compile of Koivisto in the download package, whatever hardware it requires.

BTW, under what flag shall Koivisto sail? I understand Kim is from Finnland but as far as I know you haven't revealed your nationality.
All of those flags are mostly used for debugging and testing purpose. For example when we want to debug the code, we would use DFLAGS.
The #SRC flag is used in the future once the NN code will be included. I actually changed the directory name to "eun" but forgot to adjust that in the makefile. You can mostly ignore all the other flags. Also you are right that march=native and the POPCNTFLAGS are redundant.

I am german. Is it possible to have some international flag... Like EU?

Luecx
Posts: 25
Joined: Thu Jun 18, 2020 7:20 pm
Full name: Finn Eggers

Re: Koivisto 2.0

Post by Luecx » Wed Sep 09, 2020 12:40 pm

Guenther wrote:
Wed Sep 09, 2020 11:38 am
supersharp77 wrote:
Wed Sep 09, 2020 3:29 am
Luecx wrote:
Tue Sep 08, 2020 8:22 pm
A while ago, Kim Kåhre and I (Finn Eggers) have published the first version of our engine Koivisto. We have made some very fast progress before that release and we have also had a lot of progress in the last couple of days. That's why we want to publish Koivisto 2.0 today. Besides adding Lazy SMP which scales nicely up to 8 threads and beyond, we fixed a few bugs, improved the speed, and many more things as listed below:

...

The easiest way to contact us would be our Discord: (Finn) Luecx#0540 or (Kim) ChiefWiggum#1154
Whoa..Whoa... just a minute my friend...no windows binaries have been produced on v1.0 so far (or 2.0)... and windows searches on Koivisto bring up absolutely nothing.. so who is doing the testing...we need windows binaries for popcont and npopcount...Thank you AR :) :wink:
You can try those on your own risk from here (bmi2 completely untested - I can only run the 'old' ones on my hardware = no popcnt/core2)
(I noticed that on my slow old hardware Koivisto 2 runs now at around 740kn/s compared to around 600kn/s for version 1 - checked for startpos)
https://rwbc-chess.de/download.htm

We should perhaps adjust the makefile for those type of hardwares. On my hardware, Koivisto gets about 2.5Mnps there.

User avatar
Guenther
Posts: 3809
Joined: Wed Oct 01, 2008 4:33 am
Location: Regensburg, Germany
Full name: Guenther Simon
Contact:

Re: Koivisto 2.0

Post by Guenther » Wed Sep 09, 2020 12:47 pm

Luecx wrote:
Wed Sep 09, 2020 12:40 pm
Guenther wrote:
Wed Sep 09, 2020 11:38 am
supersharp77 wrote:
Wed Sep 09, 2020 3:29 am
Luecx wrote:
Tue Sep 08, 2020 8:22 pm
A while ago, Kim Kåhre and I (Finn Eggers) have published the first version of our engine Koivisto. We have made some very fast progress before that release and we have also had a lot of progress in the last couple of days. That's why we want to publish Koivisto 2.0 today. Besides adding Lazy SMP which scales nicely up to 8 threads and beyond, we fixed a few bugs, improved the speed, and many more things as listed below:

...

The easiest way to contact us would be our Discord: (Finn) Luecx#0540 or (Kim) ChiefWiggum#1154
Whoa..Whoa... just a minute my friend...no windows binaries have been produced on v1.0 so far (or 2.0)... and windows searches on Koivisto bring up absolutely nothing.. so who is doing the testing...we need windows binaries for popcont and npopcount...Thank you AR :) :wink:
You can try those on your own risk from here (bmi2 completely untested - I can only run the 'old' ones on my hardware = no popcnt/core2)
(I noticed that on my slow old hardware Koivisto 2 runs now at around 740kn/s compared to around 600kn/s for version 1 - checked for startpos)
https://rwbc-chess.de/download.htm

We should perhaps adjust the makefile for those type of hardwares. On my hardware, Koivisto gets about 2.5Mnps there.
How fast is e.g. vanilla SF 11 for your hardware from startpos?
Let's say around depth 25-30, here it is around 640 kn/s.
https://rwbc-chess.de
HGM@'chessqueen' 2018-present, aka: 'George' 2013-2016, 'pichy' 2006-2013, 'Jorge Pichard' 2000-2006 (old forum) wrote: http://talkchess.com/forum3/viewtopic.p ... 79#p789713

Post Reply