Instruction for running Scorpio with neural network on linux

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Daniel Shawul
Posts: 3572
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Instruction for running Scorpio with neural network on linux

Post by Daniel Shawul » Wed Aug 01, 2018 6:05 pm

Hi,

I just added notes on how to run Scorpio with neural network evaluation on a linux machine here https://github.com/dshawul/Scorpio#nns

No idea how to do this on Windows since tensorflow_cc does not support it.

The easiest way is to use the docker images but you could also build and install the tensorflow_cc library.

Let me know of any issues you may encounter.

regards,
Daniel

User avatar
CMCanavessi
Posts: 825
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: Instruction for running Scorpio with neural network on linux

Post by CMCanavessi » Wed Aug 01, 2018 9:51 pm

Daniel, is there any way to avoid having to install tensorflow and all the crap on windows? Something like the leela team has done, you get the binary and a couple of .dll files and that's it.

It would be much more attractive for the end user, and solve a lot of problems. I'm really interested in seeing Scorpio NN play.
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

crem
Posts: 68
Joined: Wed May 23, 2018 7:29 pm

Re: Instruction for running Scorpio with neural network on linux

Post by crem » Thu Aug 02, 2018 10:42 am

Fyi building tensorflow_cc for windows is not very straightforward (easily takes a whole weekend), but possible.

What we did for LCZero is described here:
https://github.com/glinscott/leela-ches ... -382848569

I've just read in TCEC chat that not having windows build is the reason Scorpio NN doesn't participate in TCEC. That's sad to hear. :(
I guess that now it's too late to try to squeeze in, but can help with building it for windows anyway. If I have known about the issue at least one week ago, I think we'd be able to build it.

Milos
Posts: 3295
Joined: Wed Nov 25, 2009 12:47 am

Re: Instruction for running Scorpio with neural network on linux

Post by Milos » Thu Aug 02, 2018 2:17 pm

crem wrote:
Thu Aug 02, 2018 10:42 am
Fyi building tensorflow_cc for windows is not very straightforward (easily takes a whole weekend), but possible.

What we did for LCZero is described here:
https://github.com/glinscott/leela-ches ... -382848569

I've just read in TCEC chat that not having windows build is the reason Scorpio NN doesn't participate in TCEC. That's sad to hear. :(
I guess that now it's too late to try to squeeze in, but can help with building it for windows anyway. If I have known about the issue at least one week ago, I think we'd be able to build it.
As I wrote in the other thread, the best tutorial I found is this one:
https://medium.com/@shiweili/building-t ... 9c90e23e6e
To answer CMCanavessi it is possible to build tensorflow.dll as described in tutorial, however, one cannot distribute already built one without breaking the copyright.

User avatar
CMCanavessi
Posts: 825
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: Instruction for running Scorpio with neural network on linux

Post by CMCanavessi » Fri Aug 03, 2018 11:37 am

I wonder if tensorflow.dll can be compiled under linux and then copied to a windows machine...?
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

Milos
Posts: 3295
Joined: Wed Nov 25, 2009 12:47 am

Re: Instruction for running Scorpio with neural network on linux

Post by Milos » Fri Aug 03, 2018 2:51 pm

CMCanavessi wrote:
Fri Aug 03, 2018 11:37 am
I wonder if tensorflow.dll can be compiled under linux and then copied to a windows machine...?
Crosscompiling TF from Linux to Windows, good luck with that...

Daniel Shawul
Posts: 3572
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Re: Instruction for running Scorpio with neural network on linux

Post by Daniel Shawul » Mon Aug 06, 2018 2:48 am

crem wrote:
Thu Aug 02, 2018 10:42 am
Fyi building tensorflow_cc for windows is not very straightforward (easily takes a whole weekend), but possible.

What we did for LCZero is described here:
https://github.com/glinscott/leela-ches ... -382848569

I've just read in TCEC chat that not having windows build is the reason Scorpio NN doesn't participate in TCEC. That's sad to hear. :(
I guess that now it's too late to try to squeeze in, but can help with building it for windows anyway. If I have known about the issue at least one week ago, I think we'd be able to build it.
Thanks, I am trying to build it on windows machine now. I installed cuda/cudnn/msbuild and went upto the build stage .
The MSBuild.exe is there in the directory so I don't know why it is complaining.
Any ideas ?

Code: Select all

-- Selecting Windows SDK version  to target Windows 10.0.15063.
CMake Error at CMakeLists.txt:5 (project):
  Failed to run MSBuild command:

    C:/Program Files (x86)/MSBuild/14.0/bin/MSBuild.exe

  to get the value of VCTargetsPath:

    Microsoft (R) Build Engine version 14.0.25420.1
    Copyright (C) Microsoft Corporation. All rights reserved.

    Build started 8/6/2018 2:37:50 AM.
    Project "C:\dev\build\CMakeFiles\3.12.0\VCTargetsPath.vcxproj" on node 1 (default targets).
    C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Platforms\x64\PlatformToolsets\v140\Toolset.targets(36,5): error MSB8036: The Windows SDK version 8.1 was not found. Install the required version of Windows SDK or change the SDK version in the project property pages or by right-clicking the solution and selecting "Retarget solution". [C:\dev\build\CMakeFiles\3.12.0\VCTargetsPath.vcxproj]
    Done Building Project "C:\dev\build\CMakeFiles\3.12.0\VCTargetsPath.vcxproj" (default targets) -- FAILED.

    Build FAILED.

    "C:\dev\build\CMakeFiles\3.12.0\VCTargetsPath.vcxproj" (default target) (1) ->
    (Desktop_PlatformPrepareForBuild target) ->
      C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Platforms\x64\PlatformToolsets\v140\Toolset.targets(36,5): error MSB8036: The Windows SDK version 8.1 was not found. Install the required version of Windows SDK or change the SDK version in the project property pages or by right-clicking the solution and selecting "Retarget solution". [C:\dev\build\CMakeFiles\3.12.0\VCTargetsPath.vcxproj]

        0 Warning(s)
        1 Error(s)

    Time Elapsed 00:00:00.43


  Exit code: 1



-- Configuring incomplete, errors occurred!
See also "C:/dev/build/CMakeFiles/CMakeOutput.log".
The log file says it is a Windows 10 ?!

Code: Select all

The system is: Windows - 10.0.15063 - AMD64

Milos
Posts: 3295
Joined: Wed Nov 25, 2009 12:47 am

Re: Instruction for running Scorpio with neural network on linux

Post by Milos » Mon Aug 06, 2018 3:52 am

Daniel Shawul wrote:
Mon Aug 06, 2018 2:48 am
crem wrote:
Thu Aug 02, 2018 10:42 am
Fyi building tensorflow_cc for windows is not very straightforward (easily takes a whole weekend), but possible.

What we did for LCZero is described here:
https://github.com/glinscott/leela-ches ... -382848569

I've just read in TCEC chat that not having windows build is the reason Scorpio NN doesn't participate in TCEC. That's sad to hear. :(
I guess that now it's too late to try to squeeze in, but can help with building it for windows anyway. If I have known about the issue at least one week ago, I think we'd be able to build it.
Thanks, I am trying to build it on windows machine now. I installed cuda/cudnn/msbuild and went upto the build stage .
The MSBuild.exe is there in the directory so I don't know why it is complaining.
Any ideas ?

Code: Select all

-- Selecting Windows SDK version  to target Windows 10.0.15063.
CMake Error at CMakeLists.txt:5 (project):
  Failed to run MSBuild command:

    C:/Program Files (x86)/MSBuild/14.0/bin/MSBuild.exe

  to get the value of VCTargetsPath:

    Microsoft (R) Build Engine version 14.0.25420.1
    Copyright (C) Microsoft Corporation. All rights reserved.

    Build started 8/6/2018 2:37:50 AM.
    Project "C:\dev\build\CMakeFiles\3.12.0\VCTargetsPath.vcxproj" on node 1 (default targets).
    C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Platforms\x64\PlatformToolsets\v140\Toolset.targets(36,5): error MSB8036: The Windows SDK version 8.1 was not found. Install the required version of Windows SDK or change the SDK version in the project property pages or by right-clicking the solution and selecting "Retarget solution". [C:\dev\build\CMakeFiles\3.12.0\VCTargetsPath.vcxproj]
    Done Building Project "C:\dev\build\CMakeFiles\3.12.0\VCTargetsPath.vcxproj" (default targets) -- FAILED.

    Build FAILED.

    "C:\dev\build\CMakeFiles\3.12.0\VCTargetsPath.vcxproj" (default target) (1) ->
    (Desktop_PlatformPrepareForBuild target) ->
      C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Platforms\x64\PlatformToolsets\v140\Toolset.targets(36,5): error MSB8036: The Windows SDK version 8.1 was not found. Install the required version of Windows SDK or change the SDK version in the project property pages or by right-clicking the solution and selecting "Retarget solution". [C:\dev\build\CMakeFiles\3.12.0\VCTargetsPath.vcxproj]

        0 Warning(s)
        1 Error(s)

    Time Elapsed 00:00:00.43


  Exit code: 1



-- Configuring incomplete, errors occurred!
See also "C:/dev/build/CMakeFiles/CMakeOutput.log".
The log file says it is a Windows 10 ?!

Code: Select all

The system is: Windows - 10.0.15063 - AMD64
As it says you are missing Windows SDK 8.1. But you have Visual Studio 2017 installed which goes with Win SDK 10. So in your build script you have badly referenced Windows SDK version.

Daniel Shawul
Posts: 3572
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Re: Instruction for running Scorpio with neural network on linux

Post by Daniel Shawul » Mon Aug 06, 2018 4:35 am

Thanks! Though I was using 2015 community edition, it didn't come with the SDK.
The cmake configuration went through after doing that and also installing python.

Daniel Shawul
Posts: 3572
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Re: Instruction for running Scorpio with neural network on linux

Post by Daniel Shawul » Mon Aug 06, 2018 2:53 pm

The compilation failed after 5 hours while building the Eigen library.
Any ideas ?

Code: Select all

"C:\dev\build\tf_tutorials_example_trainer.vcxproj" (default target) (1) ->
"C:\dev\build\tf_core_gpu_kernels.vcxproj" (default target) (132) ->
(CustomBuild target) ->
  c:\dev\build\external\eigen_archive\eigen\src/Core/arch/CUDA/Half.h(212): error : more than one instance of overloaded function "__hadd" matches the argument list: [C:\d
ev\build\tf_core_gpu_kernels.vcxproj]

    154 Warning(s)
    1 Error(s)

Time Elapsed 05:44:23.34

Post Reply