GUI Speed Benchmarks

[Sedat Canbaz]

Hello Chess Friends,

Today I have organized a series of GUI Benchmarks...

There were some discussions before (it was not much clear) regarding kN/s values under different GUIs

So I hope the current GUI speed test results will be useful, in case of comparing...

And here are the results by Stockfish 13111907 x64 4c's Highest kN/s values per GUIs (based on approx. 30 seconds)

Ranking (higher better):
1. Fritz 12 GUI (10 Mart 2011) >>>>> 110s = 4147 kN/s (34s = 4235 kN/s)
2. ChessGUI 0.240k >>>>> 99s = 4138 kN/s (29s = 4204 kN/s)
3. Arena 3.0 GUI >>>>> 171s = 4170 kN/s (30s = 4120 kN/s)
4. ShredderClassic 4 GUI >>>>> 92s = 4112 kN/s (37s = 4092 kN/s)
5. ChessPartner 6.0.4.0 GUI >>>>> 97s = 4103 kN/s (35s = 4041 kN/s)
6. Winboard 4.7.1 GUI >>>>> 242s = 4164 kN/s (30s = 4023 kN/s)
7. Aquarium 4.0.5 GUI >>>>> 137s = 4081 kN/s (30s = 4020 kN/s)

More Details (all benchmarks are done on/by):
- QX9650 3.0 GHz, Kingstone DDR2 3GB 1066 MHz, Windows 8.1 64-bit
- A fresh operating system is installed (on 12.02.2014)
- Stockfish 13111907 x64 (Non-SSE42) is played, using 4 cores
- During the benchmarks: no any external programs were in the background
- Stockfish (under Winboard GUI) is played with PolyGlot 2.0.1 adapter
- Before starting each bench: Stockfish's hahtables are cleaned
- The benchmarks are tested only one time per GUI (not several times)

Conclusion:
For a better conclusion, we need more benchmarks on different hardwares...
But however,(looking at current bench results) we see that Fritz GUI's performance is best!!
Also I am quite sutisfied by the performance of ChessGUI, Arena GUI and ShredderClassic GUI too
Actually all GUIs bench results are very good and close to each other, so there is no BIG difference between all of them
The most unfamiliar GUIs for me: (where I spent a lot of time for the configurations): ChessGUI and Aquarium GUI


For Test Conditions:
http://www.sedatcanbaz.com/chess/?page_id=864


Download All GUI Benchmarks:
http://www.mediafire.com/download/fl8ox ... ISPEED.rar

*Note: Benchmarks are with password protected, see for more details:
http://rybkaforum.net/cgi-bin/rybkaforu ... ?tid=28471



Btw,
I hope you enjoy all my latest ComputerChess activities (or maybe some of them... ) for 2014 !?
Note that only for the current GUI test: I spent at least 6 hours and don't forget to say thanks





Greetings,
Sedat

[zullil]

Hello Chess Friends,

Today I have organized a series of GUI Benchmarks...


- Stockfish 13111907 x64 (Non-SSE42) is played, using 4 cores
- The benchmarks are tested only one time per GUI (not several times)

Greetings,
Sedat

Thanks. But if you are searching with 4 threads and taking just one measurement, the results are meaningless.

[Sedat Canbaz]

Hello Chess Friends,

Today I have organized a series of GUI Benchmarks...


- Stockfish 13111907 x64 (Non-SSE42) is played, using 4 cores
- The benchmarks are tested only one time per GUI (not several times)

Greetings,
Sedat

Thanks. But if you are searching with 4 threads and taking just one measurement, the results are meaningless.

Not at all...its my pleasure

Once more I'd like to mention that I spent at least 6 hours for the current GUI Speed Benchmark

Maybe my results are meaningless or maybe not...what about your results ???

And I have no patience to see your useful GUI benchmark results by different engines and running several times per GUI

Can you do that ??? where other chess friends can benefit from your work ?

[ouachita]

And here are the results by Stockfish 13111907 x64 4c's Highest kN/s values per GUIs (based on approx. 30 seconds)
Greetings, Sedat

thanks again Sedat for all your useful chess GUI/engine activities

[zullil]

Can you do that ??? where other chess friends can benefit from your work ?

No. I don't even have a GUI.

Didn't mean to offend, and sorry if I did. But it seems to me that what you're doing is something like this:

I roll a red die one time. It comes up 5. I then roll a green die and get a 3. And then I try to claim that the red die is "better" than the green one.

As I'm sure you know, any multi-threaded search is non-deterministic. Even if you search to a fixed depth, the tree that is examined varies wildly from one search to the next.

I appreciate that you've spent a lot of time and energy setting up this test. But the methodology seems off the mark. Seems to me that what you're measuring has nothing to do with the GUI, and is simply the natural variation in a non-deterministic process.

In any case, I hope many "chess friends" contribute their data. The more contributions, the better your conclusions might be.

[Sedat Canbaz]

Can you do that ??? where other chess friends can benefit from your work ?

No. I don't even have a GUI.

Didn't mean to offend, and sorry if I did. But it seems to me that what you're doing is something like this:

I roll a red die one time. It comes up 5. I then roll a green die and get a 3. And then I try to claim that the red die is "better" than the green one.

As I'm sure you know, any multi-threaded search is non-deterministic. Even if you search to a fixed depth, the tree that is examined varies wildly from one search to the next.

I appreciate that you've spent a lot of time and energy setting up this test. But the methodology seems off the mark. Seems to me that what you're measuring has nothing to do with the GUI, and is simply the natural variation in a non-deterministic process.

In any case, I hope many "chess friends" contribute their data. The more contributions, the better your conclusions might be.

Oh yes... now I remembered that you are not experienced of using GUIs

So you are the last person who can make comments over this issue

But anyway, it smells me something like you are looking for holes in my work.... no problem I wish you good luck to look for holes...
But next time remember me to give you magnifying glass, in this way you can find more holes or something BIGGER...

BTW, (the below message is not for the amateur Louis):
My main goal of the current GUI benchmark speed test is to show that there is no BIG speed different between GUIs
To be sure 100 % we need hundreds/thousands of testings...
The current idea belongs to me (nowadays I have no much free time, due to right now I am working over Perfect 2014 books)
So I hope to see a hero, who can manage to run a similar GUI benchmarks ...but based on hundreds/thousands of testings

Best,
Sedat

[Sedat Canbaz]

And here are the results by Stockfish 13111907 x64 4c's Highest kN/s values per GUIs (based on approx. 30 seconds)
Greetings, Sedat

thanks again Sedat for all your useful chess GUI/engine activities

You are welcome dear Tony

I know very well that in this world, there are also very good and positive people !!!

Kind regards,
Sedat

[Jim Collins]

Thanks Sedat. Very useful!

[hgm]

Note that engines are running on a computer as independent processes, so that it is theoretically impossible for a GUI to affect the speed of an engine (running at a given priority) other than through competing with it for general system resources (in particular, CPU time). So the only thing of interest is to know how much CPU time a GUI would consume in doing what it does while the engine is searching. Of course most GUIs would typically not do much at all; now and then they might receive a line of engine output, and display in on the screen, possible scrolling other engine lines away. Or update the current move / move count / node count. In a long analysis run, where new lines are spit out very infrequently, that would hardly be anything. And how much it is exactly would depend more on how you layed out the GUI on the screen (large / smmall size, how many lines of engine-output you display) than on what GUI it is.

A more reliable way to measure that would be to directly obtain the CPU usage of the GUI process from the OS than trying to deduce it from the performance of an inherently variable other process (such as an SMP engine) running on the same machine. There are hundreds of things going on in the background of today's Operating Systems, and each of them could be the source of speed fluctuations even in a determinsitic engine, and you would never know whether the GUI is responsble for them. But the OS would know, and can tell you that.

Note that GUI could affect apparent performance by setting the priority of the engine process. A process that tries to consume all system resources tends to make the entire system sluggish and unresponsive if it is run at too high a priority. This is good for the knps of the engine, but during fast games it would still hurt itself by this, because it would also make the GUI sluggish by denying it CPU time, and it is dependent on the GUI for transferring the moves. Of course during analysis you would care less about this, as the engine runs pretty much its own show there. OTOH, it would be pretty annoying there if the user wanted to interactively move pieces around, and had sluggish mouse response. So GUIs usually run engine at 'below-normal' priority (as Windows calls it), or can be configured to do so. For a meaningful comparison it would have to be checked at what priority Stockfish was actually running when its knps were measured, and the GUIs would have to be configured to make that the same. Otherwise you would just be measuring the effect of engine priority on engine speed, not that of the GUI.

Other than through setting the engine priority, the idea that GUIs could affect engine speed is more a supersition as anything else. Unless the GUI would have been exceedingly poorly designed, and was burning huge amount of CPU time while doing nothing. But you can be sure none of the mentioned GUIs does that. To get an idea: on my Win8 laptop, the Resource Monitor says that WinBoard uses 0.30% and Polyglot 0.05% of the average CPU time when I am analysing with Fruit 2.1 (using 25%, as I have an i3 with 4 Hyper Threads), and rapidly stepping through a game. That drops to 0.20% and 0.03% in a long analysis where I am doing nothing. Even if other GUIs would be twice as fast, that would only make 0.10% difference, and it seems a completely hopeless task to try to deduce it from the engine knps, which easily fluctuates 10 times as much in absence of any GUI (running it from the command line).

And why would you want to measure it in such an exceedingley indirect and inherently unreliable way, when you can simply read it with an accuracy 0f 0.01 percent-points from the resource monitor? I think Louis' metaphor of throwing dice is even too favorable, because if you repeated that often enough you would eventually see whether the red or green die was better. But this is more like throwing dice in the hope to determine what flowers are in the vase that is standing on the same table...

[Sedat Canbaz]

Hello H.G.Muller,

Thanks for your valuable information

I have benchmarked all GUIs at default settings

And as far as I remember (now I am not in front of QX9650 system):
-Fritz GUI:Stockfish is played with Priority=below normal
Note that Fritz GUI asks for 'Priority' in case of creating new UCI engine
*Shredder Classic: Stockfish is played with Priority=below normal
Note that ShredderClassic's default GUI setting for all engines is Priority=below normal
- Arena GUI:Stockfish is played with Priority=below normal
*Note Arena has a superior 'Priority' option to set for all engines, e.g Priority=below normal or Priority=normal (as you wish...)

Not sure about rest GUIs,
Later I plan to repeat the benchmark test...
But in the next time (i will look more carefully and I will set for all GUIs Priority=below normal) I plan to give a mate position,where the engine will use 1 (one) core...

And now I wonder a lot about which GUI will solve the mate position faster ... ?!

Btw, there are a lot of improvements in your latest Winboard GUI...
I noticed many new options, well-done!!

Best,
Sedat