Escape from the Unix Cosmic Background Radiation

[sje]

Escape from the Unix Cosmic Background Radiation

Say that you've got your magnificent chess software fully debugged and it's time to start working on middle to low level speed optimizations. Many such optimizations might individually produce much less than one percent of an improvement, and a change of that magnitude can't be reliably measured if the host platform has background noise which affects a program's running time on the order of one percent or more.

What to do? One idea in to run the tests multiple times on multiple machines and hope that the averages will provide decisive information, one way or the other. Another idea is to test the program on a low end embedded host which has no real operating system, so no virtual memory, and no daemons running at unpredictable times with unpredictable loads.

Maybe the best practical approach is to test on a Unix system which is itself running at a single-user runlevel with only minimal and unavoidable background processing. It may be that no windowing subsystem would be available, only a character display like the terminals from the Old Days. Even then, there could still be some randomness caused by disk I/O.

[vittyvirus]

Escape from the Unix Cosmic Background Radiation

Say that you've got your magnificent chess software fully debugged and it's time to start working on middle to low level speed optimizations. Many such optimizations might individually produce much less than one percent of an improvement, and a change of that magnitude can't be reliably measured if the host platform has background noise which affects a program's running time on the order of one percent or more.

What to do? One idea in to run the tests multiple times on multiple machines and hope that the averages will provide decisive information, one way or the other. Another idea is to test the program on a low end embedded host which has no real operating system, so no virtual memory, and no daemons running at unpredictable times with unpredictable loads.

Maybe the best practical approach is to test on a Unix system which is itself running at a single-user runlevel with only minimal and unavoidable background processing. It may be that no windowing subsystem would be available, only a character display like the terminals from the Old Days. Even then, there could still be some randomness caused by disk I/O.

+1

[ZirconiumX]

Escape from the Unix Cosmic Background Radiation

Say that you've got your magnificent chess software fully debugged and it's time to start working on middle to low level speed optimizations. Many such optimizations might individually produce much less than one percent of an improvement, and a change of that magnitude can't be reliably measured if the host platform has background noise which affects a program's running time on the order of one percent or more.

What to do? One idea in to run the tests multiple times on multiple machines and hope that the averages will provide decisive information, one way or the other. Another idea is to test the program on a low end embedded host which has no real operating system, so no virtual memory, and no daemons running at unpredictable times with unpredictable loads.

Maybe the best practical approach is to test on a Unix system which is itself running at a single-user runlevel with only minimal and unavoidable background processing. It may be that no windowing subsystem would be available, only a character display like the terminals from the Old Days. Even then, there could still be some randomness caused by disk I/O.

+1

Certainly Windows would not help with your problem - there are way too many background daemons on a modern Windows system.

I'd recommend an Arch Linux base install, then removing things like networking to ensure you have as little background noise as possible. I do get the feeling that Unix background noise, like CBR is unavoidable (Linux/BSD has to do some processing too!), unless you use a different universe - but then there is no guarantee that your improvements will carry over.

Matthew:out

[vittyvirus]

Escape from the Unix Cosmic Background Radiation

Say that you've got your magnificent chess software fully debugged and it's time to start working on middle to low level speed optimizations. Many such optimizations might individually produce much less than one percent of an improvement, and a change of that magnitude can't be reliably measured if the host platform has background noise which affects a program's running time on the order of one percent or more.

What to do? One idea in to run the tests multiple times on multiple machines and hope that the averages will provide decisive information, one way or the other. Another idea is to test the program on a low end embedded host which has no real operating system, so no virtual memory, and no daemons running at unpredictable times with unpredictable loads.

Maybe the best practical approach is to test on a Unix system which is itself running at a single-user runlevel with only minimal and unavoidable background processing. It may be that no windowing subsystem would be available, only a character display like the terminals from the Old Days. Even then, there could still be some randomness caused by disk I/O.

+1

Certainly Windows would not help with your problem - there are way too many background daemons on a modern Windows system.

I'd recommend an Arch Linux base install, then removing things like networking to ensure you have as little background noise as possible. I do get the feeling that Unix background noise, like CBR is unavoidable (Linux/BSD has to do some processing too!), unless you use a different universe - but then there is no guarantee that your improvements will carry over.

Matthew:out

I use Linux VMs (mainly for learnng hacking and shell coding). What about a CC operating system?

[bob]

Escape from the Unix Cosmic Background Radiation

Say that you've got your magnificent chess software fully debugged and it's time to start working on middle to low level speed optimizations. Many such optimizations might individually produce much less than one percent of an improvement, and a change of that magnitude can't be reliably measured if the host platform has background noise which affects a program's running time on the order of one percent or more.

What to do? One idea in to run the tests multiple times on multiple machines and hope that the averages will provide decisive information, one way or the other. Another idea is to test the program on a low end embedded host which has no real operating system, so no virtual memory, and no daemons running at unpredictable times with unpredictable loads.

Maybe the best practical approach is to test on a Unix system which is itself running at a single-user runlevel with only minimal and unavoidable background processing. It may be that no windowing subsystem would be available, only a character display like the terminals from the Old Days. Even then, there could still be some randomness caused by disk I/O.

Look up "lightweight kernels" in the world of Linux. They work quite well at stabilizing what is basically an unstable environment. No virtual memory, no paging, no long list of daemon processes randomly waking up to cause time jitter, etc...

[syzygy]

A 1% speedup can easily be measured in the presence of background noise by simply running the engine long enough or repeating a short test often enough. The background noise will even out.

More problematic is compiler noise. A small change in one part of a function may by chance lead to a better (or worse) optimisation in another part of the same function (or even another function) which will probably disappear with the next unrelated change. Disabling virtual memory and daemons won't help here. Running the engine for a very long time will not help, either.

Yet another problem is different responses to optimisations by different processor versions (let alone different processor architectures). Disabling virtual memory and daemons won't help here, either. (Running on different hardware does.)

[sje]

Look up "lightweight kernels" in the world of Linux. They work quite well at stabilizing what is basically an unstable environment. No virtual memory, no paging, no long list of daemon processes randomly waking up to cause time jitter, etc...

I'm in the process of restoring a 1989 Macintosh SE/30 running the Mac OS of the day which allowed no background events other than responses to mouse actions, keystrokes, and diskette insertions. I've got an ANSI C compiler of the same vintage for the SE/30, and it shouldn't be too much trouble to port Oscar to the 8 MB RAM 16 MHz 32 bit 68030 machine for testing.