Compiling Stockfish dev for Large Pages

[Milos]

Lol, that's from old time when Robert didn't know how to enable user rights for large pages. In the meanwhile he learnt it so it is not required. Never looked at Sugar source but if there is a function Get_LockMemory_Privileges in tt.cpp it should be able to do it automatically, otherwise its implementation of Large Pages is probably broken.

What exactly do you mean?
Is there an easier way to enable Large Pages than what is specified in the Houdini manual?
I'm interested!

Code: Select all

bool Get_LockMemory_Privileges()
{
    HANDLE TH, PROC7;
    TOKEN_PRIVILEGES tp;
    bool ret = false;

    PROC7 = GetCurrentProcess();
    if (OpenProcessToken(PROC7, TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY, &TH))
    {
        if (LookupPrivilegeValue(NULL, TEXT("SeLockMemoryPrivilege"), &tp.Privileges[0].Luid))
        {
            tp.PrivilegeCount = 1;
            tp.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;
            if (AdjustTokenPrivileges(TH, FALSE, &tp, 0, NULL, 0))
            {
                if (GetLastError() != ERROR_NOT_ALL_ASSIGNED)
                    ret = true;
            }
        }
        CloseHandle(TH);
    }
    return ret;
}

Very similar as to what is described in MSDN support page for large pages (https://msdn.microsoft.com/en-us/librar ... s.85).aspx)

[Houdini]

Very similar as to what is described in MSDN support page for large pages (https://msdn.microsoft.com/en-us/librar ... s.85).aspx)

The Houdini code does exactly that (it's also based on the MSDN example).

To make this work, the user requires the Privilege to lock the pages in memory, which is achieved by the steps listed in the Houdini User's Manual (including running as Administrator or without UAC).

[Milos]

Very similar as to what is described in MSDN support page for large pages (https://msdn.microsoft.com/en-us/librar ... s.85).aspx)

The Houdini code does exactly that (it's also based on the MSDN example).

To make this work, the user requires the Privilege to lock the pages in memory, which is achieved by the steps listed in the Houdini User's Manual (including running as Administrator or without UAC).

If you are already running as administrator you already have privilege to lock the pages therefore explicitly changing rights with gpedit is redundant. And if you open process token with TOKEN_ADJUST_PRIVILEGES you do not need to run the engine as Administrator, being logged as administrator is enough.

[Houdini]

If you are already running as administrator you already have privilege to lock the pages therefore explicitly changing rights with gpedit is redundant.

This is untrue.
It takes 5 seconds do to the test, try it...

[Milos]

If you are already running as administrator you already have privilege to lock the pages therefore explicitly changing rights with gpedit is redundant.

This is untrue.
It takes 5 seconds do to the test, try it...

I have 5 different windows machines (2 desktops and 3 laptops) all running either Windows 7 x64 Professional or Enterprise.
On all 5 I have an administrative account and on all 5 administrative accounts I have my administrator username already set for the Lock Pages in Memory policy without me explicitly ever setting anything up. So I really have no clue what are you talking about.

[Houdini]

Interesting.
Here's a fresh Windows 10 Pro installation. My apologies for the Dutch Windows

Exhibit 1: User "Robert" is Administrator on this PC



Exhibit 2: Local Policy Manager - The "Lock Pages in Memory" privilege is not assigned by default
(in Dutch: "Pagina's in het geheugen vergrendelen")



Exhibit 3: If I run Houdini Pro "as Administrator" I don't get Large Pages
Note that although my account is an Administrator on the computer, applications don't get run as Administrator by default. I have to right-click and explicitly request that. UAC is turned on.



Can you show the same on your computer?
Is your Windows 7 Pro that different from my Windows 10 Pro?

[Milos]

Yes I know that default for Lock pages in memory is not defined, i.e. none. And it is clear that allocating large pages would fail if you didn't have this permission set for active account since you could never get a token. Running engine even as administrator would not help since Lock pages in memory policy is empty.
Here is what I have:



And Lock pages in memory has my account name by default, I never added it there manually. It seams it got there automatically when I first created the account. However, if I create additional admin accounts they don't get this one set.



And finally when I run Brainfish Large Pages get normally allocated (no dreaded "No Privilege for Large Pages" message).

[Milos]

Feel free to point out what makes Brainfish's NUMA implementation better.

The code is smaller, more readable, especially initialization part where number of nodes and processors per node is acquired (even though it's Peter's code).
The actual assignment policy of physical cores is the same, but BF also has assignment in case of logical cores.
In the past BF had exactly the same initialization code, obviously you both copied from the same source.
In addition BF had those nice history move counters per node that are now unfortunately gone.

As to asmFish, I have to assume you have never looked at its code or are simply not able to read code. Apart from the TB code, all of asmFish was hand coded.

I assume you actually never looked into MinGW produced binary or you don't know how to go through one.
The asmFish code (at least the one done by Ahmad) is done by starting from MinGW produced binary with full symbol table, disassembling in IDA breaking to roughly match main cpp files and then rewriting critical sections.
And if you really believe asmFish is done by taking cpp source and then writing the binary by hand from scratch, hm, what can I tell, except that you are clueless beyond hope.

[syzygy]

And if you really believe asmFish is done by taking cpp source and then writing the binary by hand from scratch, hm, what can I tell, except that you are clueless beyond hope.

If you were able to read the code, you would understand it was all done by hand (apart from the syzygy/* code). There is absolutely no doubt about that. Yes, it is an amazing piece of work.

[syzygy]

Feel free to point out what makes Brainfish's NUMA implementation better.

The code is smaller, more readable, especially initialization part where number of nodes and processors per node is acquired (even though it's Peter's code).

It also does less in that BF-NUMA needs Windows 7 or higher while Cfish-NUMA supports Windows XP SP3 and higher.

The actual assignment policy of physical cores is the same, but BF also has assignment in case of logical cores.

I don't know what you are seeing, but Cfish takes into account logical cores. It's just not necessary to count them, since it distributes all threads above the number of physical cores evenly over all the nodes.

In the past BF had exactly the same initialization code, obviously you both copied from the same source.

My primary source was MSDN, but I probably did compare with Mohammed's port of Texel's code to SF.

In addition BF had those nice history move counters per node that are now unfortunately gone.

Cfish still has per-node countermove history tables. And it makes sure that various per-thread data structures are allocated on the right node, which Brainfish very likely does not (since C++'s memory allocator would get in the way).

As usual you don't succeed in backing up your claims.