Ronald de Man is such a hero

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

Ras
Posts: 2487
Joined: Tue Aug 30, 2016 8:19 pm
Full name: Rasmus Althoff

Re: Ronald de Man is such a hero

Post by Ras »

mvanthoor wrote: Sat Mar 27, 2021 3:22 pmRust is linked statically by default and has a "executable has no dependencies (except if you link to external DLL / SO's yourself)" philosophy.
At least no specific Rust dependencies. Sure, it does increase the size of the binary, and Martin is of course right that runtimes don't get fixed as easily with static linkage in each application, but the key right now is to avoid anything like "Rust programs are a hassle for the end user". That would be deadly for a new language. Same for Go. With today's RAM and SSD, 1MB per program isn't that much of a price to pay.

Let's take e.g. static website generators. They are often quite a hassle to set up, particularly under Windows - with one notable exception: Hugo, written in Go (hence the name). That's just one executable, and this gets mentioned in every comparative review.
I would not be surprised if Rust itself is completely independent, and builds against the oldest possible versions of glibc/kernel etc... as it can manage, EVEN if you compile it on the latest Linux (i.e, it includes the old stuff to link against).
That would be a pretty unusual approach.
I would have to test it and see if a binary I build on Debian 10 will run on 8.
Yes, please do so, that would be interesting to know.
Rasmus Althoff
https://www.ct800.net
mar
Posts: 2555
Joined: Fri Nov 26, 2010 2:00 pm
Location: Czech Republic
Full name: Martin Sedlak

Re: Ronald de Man is such a hero

Post by mar »

Ras wrote: Sat Mar 27, 2021 4:11 pm
I never mentioned Rust or Go or managed languages in my original reply, I simply stated that 1M+ hello world programs (in general) aren't cool.
And that's where it's comparing apples to oranges because both are using static linkage. I just checked a hello world in C. Dynamic linkage: 16kb. Static linkage, with -O2 and -flto: 850kB. So it seems what you actually don't like (from a size point of view) is static linkage, independent of the language used.
well, 850kB still seems like a lot (if stripped), I'll try it with microsoft compiler, I'm pretty sure it will be significantly smaller even with static runtime.
I'd expect the liker to remove lots of unused runtime functions.

actually with microsoft compiler I get 120kb (hello world, static runtime) and that even contains a lot of C++ specific bloat for some reason (I get the same binary when I compile it as .c or .cpp => probably missing some compiler switch, anyway it can't get worse than 120k)
You were talking about vendor lock-in. I believe I proved you wrong.
You didn't, but you need to look at quite a bit of history to see why. It started with Sun's Java. Microsoft came along and tried their usual "embrace, extend, extinguish" strategy. That backfired in court, and that's why MS came up with a Microsoft-Java that they couldn't legally call "Java", so it went by "C#". It was over a whole decade later that MS began to opensource it. That was because by that time, MS had understood that the desktop wasn't the relevant battlefield anymore. That doesn't change the fact that originally, it had been intended as vendor lock-in: a Java that would only run on Windows.
yes, the point is that (at present time) .NET core, runtime and roslyn are all open-sourced, ergo no vendor lock-in is possible.

btw - are you sure that C# really is microsof-java? I could only find something like "Visual J++" - whatever that was, it's long dead and discontinued.
I'm not very familiar with C# or Java, but I know for sure that C# supports structs and operator overloads, while Java doesnt.

my only experience with C# is porting tscp to C# for fun, got 1.6x slowdown overall compared to C (which is actually way better than I expected)
Martin Sedlak
User avatar
mvanthoor
Posts: 1784
Joined: Wed Jul 03, 2019 4:42 pm
Location: Netherlands
Full name: Marcel Vanthoor

Re: Ronald de Man is such a hero

Post by mvanthoor »

Ras wrote: Sat Mar 27, 2021 4:57 pm At least no specific Rust dependencies. Sure, it does increase the size of the binary, and Martin is of course right that runtimes don't get fixed as easily with static linkage in each application, but the key right now is to avoid anything like "Rust programs are a hassle for the end user". That would be deadly for a new language. Same for Go. With today's RAM and SSD, 1MB per program isn't that much of a price to pay.
Indeed. As you've noticed, I have a love/hate relationship with cargo and crates.io. I love cargo, because you just put the library you need into cargo.toml, and done. But... what if you have a 15 year old program? I'd love to have some sort of way to have cargo download the dependencies, and then stash them somewhere next to the source for re-use. (It actually caches them, but not next to your source; it does so in the ./target folder.)
Let's take e.g. static website generators. They are often quite a hassle to set up, particularly under Windows - with one notable exception: Hugo, written in Go (hence the name). That's just one executable, and this gets mentioned in every comparative review.
Hm. Maybe you konw of mdbook? :D Written in Rust, for generating the Rust documentation. Just one executable. You write your "site" in pure Markdown, and run "mdbook build". Rustic's site is built that way. You can also do pretty advanced things with front-end templates, injecting CSS and Javascript, etc, but you don't have to.
Yes, please do so, that would be interesting to know.
I'll see what I can do. I have a problem with VirtualBox and its guest extensions on Linux. I've wiped the VM's and I'm now comparing VirtualBox and VMWare Player. I have a feeling that VirtualBox is faster and has more options, but I haven't been able to determine this yet. But I'll be sure to run this test at some point.
Author of Rustic, an engine written in Rust.
Releases | Code | Docs | Progress | CCRL
Ras
Posts: 2487
Joined: Tue Aug 30, 2016 8:19 pm
Full name: Rasmus Althoff

Re: Ronald de Man is such a hero

Post by Ras »

mar wrote: Sat Mar 27, 2021 5:35 pmI'd expect the liker to remove lots of unused runtime functions.
Me too, that's what flto is supposed to do, but it seems not. Maybe it's because glibc is not designed to be minimal because it's a system library, and it covers a lot of edge cases and functionality that musl doesn't.
(I get the same binary when I compile it as .c or .cpp => probably missing some compiler switch, anyway it can't get worse than 120k)
I guess it's because MSCL's C support has always been anaemic and instead, it just treats it like C++.
yes, the point is that (at present time) .NET core, runtime and roslyn are all open-sourced, ergo no vendor lock-in is possible.
That's correct today. My point was about why it even exists.

It was only after MS had realised they were painting themselves into a corner that they decided to break out. The OSS release of .NET coincides with Windows Phone going downhills, and Windows had always been struggling in the server domain anyway. That's when MS grasped that walls work two ways - as lock-in as well as lock-out.

Since MS had failed to abuse the desktop monopoly for gaining traction in the phone market, they saw themselves trapped on the desktop, an island of waning relevance. Means, if things had gone the way MS had wanted them to go, .NET would be a vendor lock-in, and relying on market failure of the vendor is a risky kind of bet, in a way. Not a posteriori of course, but ex ante.
but I know for sure that C# supports structs and operator overloads, while Java doesnt.
"Embrace, extend, extinguish" is Microsoft's core strategy (besides abusing the desktop monopoly of course), so they had to add something even if only to prevent easy backporting to Java.

Remember, this is exactly what they had tried before with the Internet Explorer when they wanted to make the WWW a Windows-only thing. This company is pretty predictable. They're often incompetent, but not actually stupid, unfortunately. They are clever in a mischievous, scummy, and often even outright illegal way.
Rasmus Althoff
https://www.ct800.net
Dann Corbit
Posts: 12540
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Ronald de Man is such a hero

Post by Dann Corbit »

Here is the size of the SF from Abrok:

Code: Select all

 
Directory of C:\Users\dannc\chess\stockfish

03/27/2021  11:57 AM    <DIR>          .
03/27/2021  11:57 AM    <DIR>          ..
03/27/2021  09:18 AM             4,495 AUTHORS
03/27/2021  09:18 AM            35,821 Copying.txt
03/27/2021  09:18 AM               232 info.txt
03/27/2021  09:18 AM        22,245,376 stockfish_21032717_x64_avx2.exe
Note that it is 22 MB.
It is not the 800K I am fawning over (although that is very nice), it is the 22MB.
Now, you may say, but the SF has the nnue data binary pre-loaded and the Cfish version does not do that. So there is no real savings, since you need the nnue data to play chess.

But, in that one folder, I have three binaries, not one. So that gives 44 free megabytes for that single folder.
And I tend to ship these binaries to large collections of machines, and I use wi-fi internet that sometimes gets balky.
His binary saves me 22 MB multiplied by (binary count-1), multiplied by target machine count.
I have lots of machines.
This is the miracle to which I genuflect.
Not to mention that his binary is also faster and (for some reason) it seems a bit keener on finding checkmates.
If only it had hooks for my logging it would be perfect, and that is the only thing that keeps CFish from being my primary analysis engine.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
Fulvio
Posts: 395
Joined: Fri Aug 12, 2016 8:43 pm

Re: Ronald de Man is such a hero

Post by Fulvio »

Dann Corbit wrote: Sat Mar 27, 2021 8:05 pm His binary saves me 22 MB multiplied by (binary count-1), multiplied by target machine count.
There is a flag ( -DNNUE_EMBEDDING_OFF ) to compile stockfish without the nnue network
Fulvio
Posts: 395
Joined: Fri Aug 12, 2016 8:43 pm

Re: Ronald de Man is such a hero

Post by Fulvio »

mvanthoor wrote: Sat Mar 27, 2021 3:22 pm I would have to test it and see if a binary I build on Debian 10 will run on 8. At this time, I build my binaries on Debian 8 to be sure. If Rust manages to compile binaries as such that they run on Debian 8, even when running Debian 10, I can ditch the old VM.
You can use "objdump -T" to view the minimum required glibc version.
For example, if I compile stockfish on my machine:

Code: Select all

objdump -T stockfish 

stockfish:     file format elf64-x86-64

DYNAMIC SYMBOL TABLE:
0000000000000000  w   D  *UND*	0000000000000000              _ITM_addUserCommitAction
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 __strtof_l
0000000000000000  w   D  *UND*	0000000000000000              _ITM_memcpyRtWn
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 fileno
0000000000000000  w   DF *UND*	0000000000000000  GLIBC_2.2.5 pthread_getspecific
0000000000000000  w   DF *UND*	0000000000000000  GLIBC_2.3.2 pthread_cond_destroy
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 __strcoll_l
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 __nl_langinfo_l
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 dgettext
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 fseeko64
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 wmemcpy
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 memset
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 mbrtowc
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 wcslen
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 close
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 __duplocale
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 ioctl
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 abort
0000000000000000  w   DF *UND*	0000000000000000  GLIBC_2.2.5 pthread_setspecific
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.29  exp
...
Requires at least glibc_2.29 so it will not work on Debian 10:
https://packages.debian.org/buster/libc6
User avatar
lucasart
Posts: 3232
Joined: Mon May 31, 2010 1:29 pm
Full name: lucasart

Re: Ronald de Man is such a hero

Post by lucasart »

mvanthoor wrote: Sat Mar 27, 2021 2:21 pm
lucasart wrote: Sat Mar 27, 2021 9:58 am Sadly, it seems that no one cares anymore, with hello world programs compiling to several megabytes in new programming languages like Go, and even Rust...
Rust keeps it's debug symbols, even in a release compile. (I.e., you can make a release compile that runs at full speed, and still debug it.) You can make it smaller:

Code: Select all

Marcel@WORKSTATION MINGW64 /c/code/rustic
$ ls -lha ./target/release/rustic.exe
-rwxr-xr-x 2 Marcel None 2.4M Mar 27 14:13 ./target/release/rustic.exe

Marcel@WORKSTATION MINGW64 /c/code/rustic
$ strip -s ./target/release/rustic.exe

Marcel@WORKSTATION MINGW64 /c/code/rustic
$ ls -lha ./target/release/rustic.exe
-rwxr-xr-x 1 Marcel None 863K Mar 27 14:13 ./target/release/rustic.exe
In Rust, everything is linked statically. The philosophy is that, if you have an executable, it should run if the OS and CPU support it; no dependencies. (Except of course if you create DLL's and specifically load those.) The above executable contains the engine, but also everything from the standard library and third-party crates that the engine needs.
Are you sure that your binary does not depend on the C standard library ?
https://doc.rust-lang.org/edition-guide ... aries.html
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.