Tricks for Compiling Sources?

[Onno Garms]

There are several engines that are compiled by someone else than the original author. I would assume that the one who compiled the sources can do this job significantly better than the author.

But what does he do differently? Are there any tricks that he applied? Or is he just the only one who bought some fancy compilier, such as the Intel compiler?

Btw., for Onno I tried the Intel compiler and found the executable slightly slower than to the MS compiler on Windows. On Linux, the Intel compiler produced an executable as fast as on Windows, while GCC resulted in a 10% slower executable those days. A current GCC is as good as Intel and MS were those days, I did not check current Intel and MS. These results are highly specific for what you compile; at work Intel outperforms MS in all projects we have tried so far.

Also my self compiled Linux binary (with GCC without any tricks) of Stockfish is a little faster on my computer than Jim's. This may of course be due to the fact that it is geared to my computer, by makefile switches or by the PGO. (However, with Onno, when I did the comparison MS vs. Intel and Pentium vs. AMD, I found that the performance of the executable does not depend on the computer where it was profiled.)

On Windows, I cannot quite achieve Jim's performance, as my Intel trial license has expired. After fixing this one, things might look different:
http://talkchess.com/forum/viewtopic.php?t=38341

[Jim Ablett]

There are several engines that are compiled by someone else than the original author. I would assume that the one who compiled the sources can do this job significantly better than the author.

But what does he do differently? Are there any tricks that he applied? Or is he just the only one who bought some fancy compilier, such as the Intel compiler?

Btw., for Onno I tried the Intel compiler and found the executable slightly slower than to the MS compiler on Windows. On Linux, the Intel compiler produced an executable as fast as on Windows, while GCC resulted in a 10% slower executable those days. A current GCC is as good as Intel and MS were those days, I did not check current Intel and MS. These results are highly specific for what you compile; at work Intel outperforms MS in all projects we have tried so far.

Also my self compiled Linux binary (with GCC without any tricks) of Stockfish is a little faster on my computer than Jim's. This may of course be due to the fact that it is geared to my computer, by makefile switches or by the PGO. (However, with Onno, when I did the comparison MS vs. Intel and Pentium vs. AMD, I found that the performance of the executable does not depend on the computer where it was profiled.)

On Windows, I cannot quite achieve Jim's performance, as my Intel trial license has expired. After fixing this one, things might look different:
http://talkchess.com/forum/viewtopic.php?t=38341

Hi Onno,

No tricks. Just lots of time spent trying various combinations of switches, and of course using p.g.o, which surprisingly not all authors employ.
The JA compiles you see posted are the ones where I have made some large speed gains over authors original compile, but there are plenty
you don't see where I couldn't improve it at all It depends a lot on the code as well. Authors sometimes include lots of hand-coded inlining
optimizations in their src already and I cannot improve on the speed at all. The Stockfish Linux compile you mentioned was compiled with the
Intel compiler, before I found out that GCC now produces faster binaries on Linux. I'm still a newbie with Linux. On Windows I compile with
Intel & Msvc, really no speed difference between them in 32 bit, but in 64 bit Intel is usually a bit faster. I try to make my compiles as fast and
as backward-compatible as possible, so for 32 bit I use older Msvc libraries & no SSE optimizations. You should be able to run my 32 bit compiles
on a Pentium 2 cpu and Windows 98.

Jim.

[Kempelen]

Jim,

Has you tried MINGW under windows? What is its performance?

thanks

[Jim Ablett]

Jim,

Has you tried MINGW under windows? What is its performance?

thanks

Hi Fermin,

Not so good under Windows. Compiles are around 20% slower than comparable Intel/Msvc ones.

Jim.

[bob]

There are several engines that are compiled by someone else than the original author. I would assume that the one who compiled the sources can do this job significantly better than the author.

But what does he do differently? Are there any tricks that he applied? Or is he just the only one who bought some fancy compilier, such as the Intel compiler?

Btw., for Onno I tried the Intel compiler and found the executable slightly slower than to the MS compiler on Windows. On Linux, the Intel compiler produced an executable as fast as on Windows, while GCC resulted in a 10% slower executable those days. A current GCC is as good as Intel and MS were those days, I did not check current Intel and MS. These results are highly specific for what you compile; at work Intel outperforms MS in all projects we have tried so far.

Also my self compiled Linux binary (with GCC without any tricks) of Stockfish is a little faster on my computer than Jim's. This may of course be due to the fact that it is geared to my computer, by makefile switches or by the PGO. (However, with Onno, when I did the comparison MS vs. Intel and Pentium vs. AMD, I found that the performance of the executable does not depend on the computer where it was profiled.)

On Windows, I cannot quite achieve Jim's performance, as my Intel trial license has expired. After fixing this one, things might look different:
http://talkchess.com/forum/viewtopic.php?t=38341

For me, your assumption is wrong. I don't compile executables, not because I can't make them efficiently, but because there are too many options. 32 vs 64 bits. SEE/noSSE, SSE2/noSSE2, all the way up to popcnt/no_popcnt and such.

I doubt anyone makes a faster executable than I do, as I use Intel's compiler and always do PGO. But trying to provide executables for every platform on the planet is a challenge.

[bob]

There are several engines that are compiled by someone else than the original author. I would assume that the one who compiled the sources can do this job significantly better than the author.

But what does he do differently? Are there any tricks that he applied? Or is he just the only one who bought some fancy compilier, such as the Intel compiler?

Btw., for Onno I tried the Intel compiler and found the executable slightly slower than to the MS compiler on Windows. On Linux, the Intel compiler produced an executable as fast as on Windows, while GCC resulted in a 10% slower executable those days. A current GCC is as good as Intel and MS were those days, I did not check current Intel and MS. These results are highly specific for what you compile; at work Intel outperforms MS in all projects we have tried so far.

Also my self compiled Linux binary (with GCC without any tricks) of Stockfish is a little faster on my computer than Jim's. This may of course be due to the fact that it is geared to my computer, by makefile switches or by the PGO. (However, with Onno, when I did the comparison MS vs. Intel and Pentium vs. AMD, I found that the performance of the executable does not depend on the computer where it was profiled.)

On Windows, I cannot quite achieve Jim's performance, as my Intel trial license has expired. After fixing this one, things might look different:
http://talkchess.com/forum/viewtopic.php?t=38341

Hi Onno,

No tricks. Just lots of time spent trying various combinations of switches, and of course using p.g.o, which surprisingly not all authors employ.
The JA compiles you see posted are the ones where I have made some large speed gains over authors original compile, but there are plenty
you don't see where I couldn't improve it at all It depends a lot on the code as well. Authors sometimes include lots of hand-coded inlining
optimizations in their src already and I cannot improve on the speed at all. The Stockfish Linux compile you mentioned was compiled with the
Intel compiler, before I found out that GCC now produces faster binaries on Linux. I'm still a newbie with Linux. On Windows I compile with
Intel & Msvc, really no speed difference between them in 32 bit, but in 64 bit Intel is usually a bit faster. I try to make my compiles as fast and
as backward-compatible as possible, so for 32 bit I use older Msvc libraries & no SSE optimizations. You should be able to run my 32 bit compiles
on a Pentium 2 cpu and Windows 98.

Jim.

What GCC version did you use to beat icc? I have the latest versions, which includes 4.5.something and 4.6.something. I still have great problems making gcc do a PGO when I try to optimize using more than one thread. The re-compile always fails with "corrupted .dyn (or whatever gcc calls the file)" without fail. And icc has always been faster, up to right now as I just tried my laptop with version 4.5.1 and icc whips it handily...

[Onno Garms]

For me, your assumption is wrong.

Which assumption?

The problem to compile efficiently arises independent of the question if the distributor or the user compiles the sources. So why shouldn't I ask how to do that?

Distributing the sources and distributing binaries both have advantages and disadvantages, which I really don't want to discuss here.

[Onno Garms]

What GCC version did you use to beat icc?

I didn't claim that. It was just a tie of GCC 4.5 against old ICC (with Onno engine)

A current GCC is as good as Intel and MS were those days

The fact that my GCC 4.5 outperformed Jim's compile can mean anything of:
- Jim did something wrong
- my executable was better geared to my computer
- GCC 4.5 is better than Jims ICC

[bob]

What GCC version did you use to beat icc?

I didn't claim that. It was just a tie of GCC 4.5 against old ICC (with Onno engine)

A current GCC is as good as Intel and MS were those days

The fact that my GCC 4.5 outperformed Jim's compile can mean anything of:
- Jim did something wrong
- my executable was better geared to my computer
- GCC 4.5 is better than Jims ICC

Notice I was responding to Jim's post...

[bob]

For me, your assumption is wrong.

Which assumption?

The problem to compile efficiently arises independent of the question if the distributor or the user compiles the sources. So why shouldn't I ask how to do that?

Distributing the sources and distributing binaries both have advantages and disadvantages, which I really don't want to discuss here.

This one:

I would assume that the one who compiled the sources can do this job significantly better than the author.

I spend a lot of time when a new version of icc comes out to see if there are any new things to try with optimizations... I care very much about speed.