How to add LP to CorChess?
Moderators: hgm, Rebel, chrisw
-
Ozymandias
- Posts: 1535
- Joined: Sun Oct 25, 2009 2:30 am
How to add LP to CorChess?
I asked I. Ivec about it, but he's at a loss.
-
Eelco de Groot
- Posts: 4567
- Joined: Sun Mar 12, 2006 2:40 am
- Full name:
Re: How to add LP to CorChess?
No experience with Large Pages. Is it difficult because of changes in CorChess relative to Stockfish? If that not the problem, then maybe on Rybka forum somebody could give some advice because several people there made versions of Stockfish with large pages. Not sure but I thought the BYO (Build Your Own) version is also built with LP. And there is a thread from correspondence chessplayer Dragon Mist about compiling Stockfish, it is not totally up to date anymore but he started from the beginning so a tutorial Compiling Large Pages into abrok dev BMI2 SF
Debugging is twice as hard as writing the code in the first
place. Therefore, if you write the code as cleverly as possible, you
are, by definition, not smart enough to debug it.
-- Brian W. Kernighan
place. Therefore, if you write the code as cleverly as possible, you
are, by definition, not smart enough to debug it.
-- Brian W. Kernighan
-
Ozymandias
- Posts: 1535
- Joined: Sun Oct 25, 2009 2:30 am
Re: How to add LP to CorChess?
Thanks for the reply, I already contacted Marco (Amos @ Rybkaforum) and he tells me that changes will need to be made, to the part that deals with adding LP.
BYO is outdated, newest SF development versions started to give error months ago. The same applies to CorChess, which obviously introduces its own changes.
BYO is outdated, newest SF development versions started to give error months ago. The same applies to CorChess, which obviously introduces its own changes.
-
tpoppins
- Posts: 919
- Joined: Tue Nov 24, 2015 9:11 pm
- Location: upstate
Re: How to add LP to CorChess?
BYO from last November works fine with the current SF dev. If you can't get it to work it's probably a case of PEBCAK. There was a new version released a month ago, that one was reported to have problems; I haven't tried it though.
The main problem with BYO is that it tries to operate like a black box; the other is that it uses code from other ppl (most importantly MZ) without giving any credit. MZ's LP code is available on Github. It took me about 10 minutes to adapt it to the current SF dev and I'm not even a programmer. POPCNT is the best I can do ATM, but you can use the following patch to make your own LP+NUMA compile. There are a couple of extras from my end:
- suppress display of fail-highs/lows (Dann Corbit's idea, IIRC, on by default)
- a "Use Syzygy" checkbox à la Komodo (checked by default)
- Syzygy probe limit defaulted to 5
- "x64" in the name to distinguish it from Master
numa.cpp and numa.h should be available from the Brainfish source. If you cannot get NUMA to work for some reason just delete the thread.cpp section from the patch and "numa.o" from the makefile section. You can still expect a few warnings from the compiler, they can be ignored.
opt.cpp follows the main patch.
opt.cpp
The main problem with BYO is that it tries to operate like a black box; the other is that it uses code from other ppl (most importantly MZ) without giving any credit. MZ's LP code is available on Github. It took me about 10 minutes to adapt it to the current SF dev and I'm not even a programmer. POPCNT is the best I can do ATM, but you can use the following patch to make your own LP+NUMA compile. There are a couple of extras from my end:
- suppress display of fail-highs/lows (Dann Corbit's idea, IIRC, on by default)
- a "Use Syzygy" checkbox à la Komodo (checked by default)
- Syzygy probe limit defaulted to 5
- "x64" in the name to distinguish it from Master
numa.cpp and numa.h should be available from the Brainfish source. If you cannot get NUMA to work for some reason just delete the thread.cpp section from the patch and "numa.o" from the makefile section. You can still expect a few warnings from the compiler, they can be ignored.
opt.cpp follows the main patch.
Code: Select all
Index: main.cpp
===================================================================
--- main.cpp (revision 4636)
+++ main.cpp (working copy)
@@ -32,10 +32,15 @@
void init();
}
+void SETUP_PRIVILEGES();
+void FREE_MEM(void *);
+
int main(int argc, char* argv[]) {
std::cout << engine_info() << std::endl;
-
+ #ifndef BENCH
+ SETUP_PRIVILEGES();
+ #endif
UCI::init(Options);
PSQT::init();
Bitboards::init();
@@ -49,6 +54,11 @@
UCI::loop(argc, argv);
+ if (large_use) {
+ FREE_MEM(TT.mem);
+ TT.mem = nullptr;
+ }
+
Threads.exit();
return 0;
}
Index: Makefile
===================================================================
--- Makefile (revision 4636)
+++ Makefile (working copy)
@@ -39,7 +39,7 @@
### Object files
OBJS = benchmark.o bitbase.o bitboard.o endgame.o evaluate.o main.o \
- material.o misc.o movegen.o movepick.o pawns.o position.o psqt.o \
+ material.o misc.o movegen.o movepick.o numa.o opt.o pawns.o position.o psqt.o \
search.o thread.o timeman.o tt.o uci.o ucioption.o syzygy/tbprobe.o
### ==========================================================================
@@ -140,6 +140,7 @@
### 3.1 Selecting compiler (default = gcc)
CXXFLAGS += -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 $(EXTRACXXFLAGS)
+CXXFLAGS += -march=native
DEPENDFLAGS += -std=c++11
LDFLAGS += $(EXTRALDFLAGS)
Index: misc.cpp
===================================================================
--- misc.cpp (revision 4636)
+++ misc.cpp (working copy)
@@ -130,7 +130,7 @@
ss << setw(2) << day << setw(2) << (1 + months.find(month) / 4) << year.substr(2);
}
- ss << (Is64Bit ? " 64" : "")
+ ss << (Is64Bit ? " x64" : "")
<< (HasPext ? " BMI2" : (HasPopCnt ? " POPCNT" : ""))
<< (to_uci ? "\nid author ": " by ")
<< "T. Romstad, M. Costalba, J. Kiiski, G. Linscott";
Index: search.cpp
===================================================================
--- search.cpp (revision 4636)
+++ search.cpp (working copy)
@@ -414,7 +414,8 @@
if ( mainThread
&& multiPV == 1
&& (bestValue <= alpha || bestValue >= beta)
- && Time.elapsed() > 3000)
+ && Time.elapsed() > 3000
+ && Options["Show Fail-highs and Fail-lows"])
sync_cout << UCI::pv(rootPos, rootDepth, alpha, beta) << sync_endl;
// In case of failing low/high increase aspiration window and
@@ -648,7 +649,7 @@
}
// Step 4a. Tablebase probe
- if (!rootNode && TB::Cardinality)
+ if (!rootNode && TB::Cardinality && Options["Use Syzygy"])
{
int piecesCount = pos.count<ALL_PIECES>();
@@ -850,8 +851,8 @@
ss->moveCount = ++moveCount;
- if (rootNode && thisThread == Threads.main() && Time.elapsed() > 3000)
- sync_cout << "info depth " << depth / ONE_PLY
+ if (rootNode && thisThread == Threads.main() && Time.elapsed() > 3000)
+ sync_cout << "info depth " << depth / ONE_PLY
<< " currmove " << UCI::move(move, pos.is_chess960())
<< " currmovenumber " << moveCount + thisThread->PVIdx << sync_endl;
Index: thread.cpp
===================================================================
--- thread.cpp (revision 4636)
+++ thread.cpp (working copy)
@@ -22,6 +22,7 @@
#include <cassert>
#include "movegen.h"
+#include "numa.h"
#include "search.h"
#include "thread.h"
#include "uci.h"
@@ -94,6 +95,8 @@
void Thread::idle_loop() {
+ Numa::instance().bindThisThread(idx);
+
WinProcGroup::bindThisThread(idx);
while (!exit)
Index: tt.cpp
===================================================================
--- tt.cpp (revision 4636)
+++ tt.cpp (working copy)
@@ -26,7 +26,10 @@
TranspositionTable TT; // Our global transposition table
+void CREATE_MEM2(void **,uint64_t);
+void FREE_MEM(void *);
+
/// TranspositionTable::resize() sets the size of the transposition table,
/// measured in megabytes. Transposition table consists of a power of 2 number
/// of clusters and each cluster consists of ClusterSize number of TTEntry.
@@ -40,8 +43,18 @@
clusterCount = newClusterCount;
+ mem = nullptr;
+ FREE_MEM(mem);
+ CREATE_MEM2(&mem, clusterCount * sizeof(Cluster));
+ large_use = true;
+
+ if (!mem)
+ {
+
free(mem);
mem = calloc(clusterCount * sizeof(Cluster) + CacheLineSize - 1, 1);
+ large_use = false;
+ }
if (!mem)
{
Index: tt.h
===================================================================
--- tt.h (revision 4636)
+++ tt.h (working copy)
@@ -83,6 +83,9 @@
/// cache lines. This ensures best cache performance, as the cacheline is
/// prefetched, as soon as possible.
+extern int large_use;
+void FREE_MEM (void *);
+
class TranspositionTable {
static const int CacheLineSize = 64;
@@ -96,7 +99,8 @@
static_assert(CacheLineSize % sizeof(Cluster) == 0, "Cluster size incorrect");
public:
- ~TranspositionTable() { free(mem); }
+ void* mem;
+ ~TranspositionTable() { large_use ? FREE_MEM (mem) : free(mem); }
void new_search() { generation8 += 4; } // Lower 2 bits are used by Bound
uint8_t generation() const { return generation8; }
TTEntry* probe(const Key key, bool& found) const;
@@ -112,7 +116,6 @@
private:
size_t clusterCount;
Cluster* table;
- void* mem;
uint8_t generation8; // Size must be not bigger than TTEntry::genBound8
};
Index: ucioption.cpp
===================================================================
--- ucioption.cpp (revision 4636)
+++ ucioption.cpp (working copy)
@@ -70,10 +70,12 @@
o["Slow Mover"] << Option(89, 10, 1000);
o["nodestime"] << Option(0, 0, 10000);
o["UCI_Chess960"] << Option(false);
+ o["Use Syzygy"] << Option(true);
o["SyzygyPath"] << Option("<empty>", on_tb_path);
o["SyzygyProbeDepth"] << Option(1, 1, 100);
o["Syzygy50MoveRule"] << Option(true);
- o["SyzygyProbeLimit"] << Option(6, 0, 6);
+ o["SyzygyProbeLimit"] << Option(5, 0, 6);
+ o["Show Fail-highs and Fail-lows"] << Option(false);
}
opt.cpp
Code: Select all
/*
Stockfish, a UCI chess playing engine derived from Glaurung 2.1
Copyright(C)2004-2008 Tord Romstad (Glaurung author)
Copyright(C)2008-2014 Marco Costalba, Joona Kiiski, Tord Romstad
Stockfish is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation,either version 3 of the License, or
(at your option) any later version.
Stockfish is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include <stdio.h>
#include "thread.h"
#include "tt.h"
using namespace std;
#define TRUE 1
#define FALSE 0
#define MEMALIGN(a, b, c) a = _aligned_malloc (c, b)
#define ALIGNED_FREE(x) _aligned_free (x)
int large_use;
#ifndef _WIN32 // Linux
#include <sys/ipc.h>
#include <sys/shm.h>
static int num;
void SETUP_PRIVILEGES(){}
void CREATE_MEM(void** A,int align,uint64_t size)
{
large_use=FALSE;
num=shmget(IPC_PRIVATE,size,IPC_CREAT|SHM_R|SHM_W|SHM_HUGETLB);
if(num==-1)
{
printf("info string LargePages FAILED %llu Mb\n",size>>20);
MEMALIGN((*A),align,size);
}
else
{
(*A)=shmat(num,NULL,0x0);
large_use=TRUE;
printf("info string LargePages OK %llu Mb\n",size>>20);
std::cout<<"info string HUGELTB "<<(size>>20)<<std::endl;
}
}
void CREATE_MEM2(void** A,uint64_t size)
{
large_use=FALSE;
num=shmget(IPC_PRIVATE,size,IPC_CREAT|SHM_R|SHM_W|SHM_HUGETLB);
if(num!=-1)
{
(*A)=shmat(num,NULL,0x0);
large_use=TRUE;
printf("info string %llu Mb LargePages\n",size>>20);
}
}
void FREE_MEM(void* A)
{
if(!A)
return;
if(!large_use)
{
ALIGNED_FREE(A);
return;
}
shmdt(A);
shmctl(num,IPC_RMID,NULL);
large_use=FALSE;
}
void SETUP_PRIVILEGES(){}
#else
void CREATE_MEM(void** A,int align,uint64_t size)
{
large_use=FALSE;
(*A)=VirtualAlloc(NULL,size,MEM_LARGE_PAGES|MEM_COMMIT|MEM_RESERVE,PAGE_READWRITE);
if((*A))
{
large_use=TRUE;
printf("info string %llu Mb LargePages\n",size>>20);
}
else
{
printf("info string %llu Mb(no LargePages)\n",size>>20);
MEMALIGN((*A),align,size);
}
}
void CREATE_MEM2(void** A,uint64_t size)
{
large_use=FALSE;
(*A)=VirtualAlloc(NULL,size,MEM_LARGE_PAGES|MEM_COMMIT|MEM_RESERVE,PAGE_READWRITE);
if((*A))
{
large_use=TRUE;
printf("Large Pages enabled. Hash %llu Mb\n",size>>20);
}
}
void FREE_MEM(void* A)
{
if(!A)
return;
if(!large_use)
ALIGNED_FREE(A);
else
{
VirtualFree(A,0,MEM_RELEASE);
large_use=FALSE;
}
}
void SETUP_PRIVILEGES()
{
HANDLE TH,PROC;
TOKEN_PRIVILEGES tp;
PROC=GetCurrentProcess();
OpenProcessToken(PROC,TOKEN_ADJUST_PRIVILEGES|TOKEN_QUERY,&TH);
LookupPrivilegeValue(NULL,TEXT("SeLockMemoryPrivilege"),&tp.Privileges[0].Luid);
tp.PrivilegeCount=1;
tp.Privileges[0].Attributes=SE_PRIVILEGE_ENABLED;
AdjustTokenPrivileges(TH,FALSE,&tp,0,NULL,0);
CloseHandle(TH);
}
#endif
-
Ozymandias
- Posts: 1535
- Joined: Sun Oct 25, 2009 2:30 am
Re: How to add LP to CorChess?
I didn't see it on Github, but I did on his forum; it is indeed easy to use.toppings wrote:MZ's LP code is available on Github.
Thx anyway.