The upcoming Y2038 catastrophe

[AlvaroBegue]

Perhaps gettimeofday has been good enough for the purpose of measuring thinking times in crafty, but it is true that it is not the right tool for the job of measuring elapsed time between events in general.

I understand Mr. Hyatt is unlikely to change his mind, because it's not his M.O., but perhaps others are following this conversation and want to know who is right. For those people, here's a link: http://blog.habets.pp.se/2010/09/gettim ... asure-time

The discussion was not about "which is most accurate". It was about does ntp provide a monotonic clock, which it does unless someone interferes. If you want a really accurate measure, you can always access the CPU cycle counter.

NTP on a lan produces distributed clock accuracy of 1ms or less, which was the goal. It also does not cause the clock to step backward so long as no root user dinks around with the system time on the ntp server. That was what the discussion was about.

Why are you quoting "which is most accurate"? I never used those words.

About whether gettimeofday provides a monotonic clock, maybe you should read its Linux man page:

The time returned by gettimeofday() is affected by discontinuous
jumps in the system time (e.g., if the system administrator manually
changes the system time). If you need a monotonically increasing
clock, see clock_gettime(2).

[bob]

If you can read this, then it likely means that you can connect to the Internet. And that means that your machine can use nntp to get the current time. Therefore, unless your software is misconfigured, your hardware is horribly buggy, or you have an idiot for a system administrator, then once your Internet connected machine finishes booting its idea of the current time will never be incorrect by thousands of seconds, or even a single second.

Over several decades, many smart guys combined with many millions of taxpayer dollars have produced an incredibly accurate network time infrastructure. This system includes careful averaging of multiple independent cesium clocks distributed around the planet. The long term performance of the individual clocks is well known, and even better clocks are developed over the years. This is what drives nntp, and gettimeofday() combines that with its own long term observations of its host to produce an almost incredibly accurate value of the current time.

Is gettimeofday() fed by nntp as good as a local cycle counter for measuring intervals? It's actually far superior, because that local counter is from a single source time base that is unlikely to be corrected for thermal drift. Further, that local counter will produce interval measurements which are never compared with nor corrected by those produced by a counter on a different chunk of hardware. This means that a nodes per second frequency measurement can't be compared among different machines using only local counters because each machine will have its own idea as to the length of a second.

The original statement I made was that it is dangerous to have a root user that can adjust the clock, as there is nothing you can do to prevent him from jumping it forward, jumping it backward, etc. If you use ntpd, this doesn't happen with two exceptions caused by human stupidity, namely daylight savings time. But that's thankfully not how Unix manages time, although it is how it is often interpreted by user applications.

Of course a human can wreck anything a human can create. And yes, there are still some that don't use ntpd. I see an occasional email with a bogus header where it was sent 15 minutes after it was received, and such. But that is a choice.

If you have ever used a cluster, or a local network of multiple machines, plus "make", without ntp things can go badly wrong. With, it doesn't happen.

[bob]

If you can read this, then it likely means that you can connect to the Internet. And that means that your machine can use nntp to get the current time. Therefore, unless your software is misconfigured, your hardware is horribly buggy, or you have an idiot for a system administrator, then once your Internet connected machine finishes booting its idea of the current time will never be incorrect by thousands of seconds, or even a single second.

What happens with daylight savings?
I suppose I could just look it up, but I guess the smart way of dealing with that is having the clock use UT and only change the timezone for conversion to local time.

That is what we do. But there are still problems if you use software that ends up with local time. It's a stupid idea that ought to go away. Why it was extended even longer I have no idea, to me it seems backward. I'd prefer to have DST during the winter to stop the 5pm = dead dark short days.

[sje]

What happens with daylight savings?
I suppose I could just look it up, but I guess the smart way of dealing with that is having the clock use UT and only change the timezone for conversion to local time.

Unix has always used UTC exclusively.

Unix was inspired by Multics ("Unix" = "eunuchs" = "castrated Multics"), and Multics used UTC. Except that Multics allowed for only eight time zones either way from Greenwich, the reason the first Multics machine set up in Hawaii didn't work so well.

[syzygy]

If you can read this, then it likely means that you can connect to the Internet. And that means that your machine can use nntp to get the current time.

Network News Transfer Protocol
We're not on rec.games.chess.computer anymore...

Therefore, unless your software is misconfigured, your hardware is horribly buggy, or you have an idiot for a system administrator, then once your Internet connected machine finishes booting its idea of the current time will never be incorrect by thousands of seconds, or even a single second.

That's just silly. Internet connections have downtime. Internet was designed to cope with that. This is not a problem for NTP either, as it will recover. BUT it simply does not guarantee monotonicity.

Firstly, for monotonic updates you need OS support. POSIX-compliant systems support (small) monotonic updates, but not all systems are POSIX-compliant. NTP does not mandate support for monotonic updates.

Secondly, NTP explicitly allows the clock to be stepped forward or backward in time if the necessary correction exceeds a threshold. This threshold is typically 125ms. (The ntpd implementation on my machine can be forced to adjust slowly using the "-x" option, but as options go this is optional. It also may defeat the purpose of ntp which is to keep your clock in sync with other clocks.)

If your PC has an unreliable hardware clock and your internet connection has a period of downtime, then a backward step in time can easily happen.

And what should not need to be mentioned: if your application relies on NTP working correctly, then you rely on external servers over which you have no control.

For a chess engine this is all relatively unimportant, as nobody will be hurt if the engine loses on time.

[sje]

Monotonicity is not a guarantee of anything other than monotonicity. If scratch a tally mark on the bathroom wallpaper every time I take a leak, that's monotonic but it's not a reliable time keeper.

Yes, if your machine is not connected to the Internet, then indeed it can't take direct advantage of any Internet service. Amazing!

And yes, if your machine is made from crappy parts or is managed by a crappy administrator, then you can't expect it to produce time measurements which smell like roses.

And if someone thinks that the collective, independent hosts of the international time server network could be maliciously impaired without detection, then they are living in Fantasyland.

There have been so many straw men here that I could set up a scarecrow factory.

[bob]

If you can read this, then it likely means that you can connect to the Internet. And that means that your machine can use nntp to get the current time.

Network News Transfer Protocol
We're not on rec.games.chess.computer anymore...

Therefore, unless your software is misconfigured, your hardware is horribly buggy, or you have an idiot for a system administrator, then once your Internet connected machine finishes booting its idea of the current time will never be incorrect by thousands of seconds, or even a single second.

That's just silly. Internet connections have downtime. Internet was designed to cope with that. This is not a problem for NTP either, as it will recover. BUT it simply does not guarantee monotonicity.

Firstly, for monotonic updates you need OS support. POSIX-compliant systems support (small) monotonic updates, but not all systems are POSIX-compliant. NTP does not mandate support for monotonic updates.

Secondly, NTP explicitly allows the clock to be stepped forward or backward in time if the necessary correction exceeds a threshold. This threshold is typically 125ms. (The ntpd implementation on my machine can be forced to adjust slowly using the "-x" option, but as options go this is optional. It also may defeat the purpose of ntp which is to keep your clock in sync with other clocks.)

If your PC has an unreliable hardware clock and your internet connection has a period of downtime, then a backward step in time can easily happen.

And what should not need to be mentioned: if your application relies on NTP working correctly, then you rely on external servers over which you have no control.

For a chess engine this is all relatively unimportant, as nobody will be hurt if the engine loses on time.

Unless there is a hardware failure, or human intervention, time NEVER advances too fast on a computer. It ALWAYS slips backward due to missed RTC interrupts. But by "slipping backward" it does NOT "jump backward" it simply does not advance as quickly as it should. ntpd handles this quite nicely, always keeping the clock moving in the "right direction" as smoothly as possible.

If a human breaks something, anything can happen. But boot it up, let it run, and the clock will always be monotonic until some pesky human breaks something somewhere.

But how about reality. If a machine is suspended, or rebooted, or replaced, or has a power failure, it will come back up, and when it does, the time will be back to correct before the first user can log in. And from that point forward, time will work just as I expect.

The "dark ages" are gone. I watched someone lose a chess game due to daylight savings time shift, when a game at an ACM event (I think it was ACM but am not certain, it was so long ago) stepped over the 2am "fall back" boundary. I watched a program use wall clock and lose on time when a move started before 23:59 am and the clock reset to 0:00 am. gettimeofday() has always worked flawlessly and not been bothered by either of those. ntpd solves the different times on different nodes causing serious file timestamp issues.

Why you want to dream up all these situations where it won't work. As I initially conceded, "any human with root access is a risk here". He can certainly break any timing assumption one might have. But done correctly, ntpd eliminates ALL of that nonsense except when a human intervenes. Granting users root access is dangerous AND foolish. We don't do it here. We've never done it anywhere I have been. And no company I have ever consulted with did either. All understand the risks.

[bob]

Monotonicity is not a guarantee of anything other than monotonicity. If scratch a tally mark on the bathroom wallpaper every time I take a leak, that's monotonic but it's not a reliable time keeper.

Yes, if your machine is not connected to the Internet, then indeed it can't take direct advantage of any Internet service. Amazing!

And yes, if your machine is made from crappy parts or is managed by a crappy administrator, then you can't expect it to produce time measurements which smell like roses.

And if someone thinks that the collective, independent hosts of the international time server network could be maliciously impaired without detection, then they are living in Fantasyland.

There have been so many straw men here that I could set up a scarecrow factory.

I don't even understand where this argument came from. With regard to time measurement, it simply means that if you take two time samples, no matter how close to gather, t(t1) <= t(t2). Unless you let a human get involved, unix guarantees that. And adding ntpd to keep time synched across a cluster, LAN, WAN or the internet works just fine.

[bob]

Perhaps gettimeofday has been good enough for the purpose of measuring thinking times in crafty, but it is true that it is not the right tool for the job of measuring elapsed time between events in general.

I understand Mr. Hyatt is unlikely to change his mind, because it's not his M.O., but perhaps others are following this conversation and want to know who is right. For those people, here's a link: http://blog.habets.pp.se/2010/09/gettim ... asure-time

The discussion was not about "which is most accurate". It was about does ntp provide a monotonic clock, which it does unless someone interferes. If you want a really accurate measure, you can always access the CPU cycle counter.

NTP on a lan produces distributed clock accuracy of 1ms or less, which was the goal. It also does not cause the clock to step backward so long as no root user dinks around with the system time on the ntp server. That was what the discussion was about.

Why are you quoting "which is most accurate"? I never used those words.

About whether gettimeofday provides a monotonic clock, maybe you should read its Linux man page:

The time returned by gettimeofday() is affected by discontinuous
jumps in the system time (e.g., if the system administrator manually
changes the system time). If you need a monotonically increasing
clock, see clock_gettime(2).

Did you read what I wrote. The unix time is monotonic UNLESS either a human manually sets the time, or there is a hardware failure. It does NOT happen otherwise. Back to the beginning post I wrote here, "giving a user root access is a security risk." <period>.

If you don't do that, the clock will always be monotonic. If you give a user root access, NO clock can guarantee monoticity. It is easy to write a program that can access kernel mode when you are root. And then you can change anything you want, the date / time is just a small part of what is at risk then.

[bob]

Perhaps gettimeofday has been good enough for the purpose of measuring thinking times in crafty, but it is true that it is not the right tool for the job of measuring elapsed time between events in general.

I understand Mr. Hyatt is unlikely to change his mind, because it's not his M.O., but perhaps others are following this conversation and want to know who is right. For those people, here's a link: http://blog.habets.pp.se/2010/09/gettim ... asure-time

The discussion was not about "which is most accurate". It was about does ntp provide a monotonic clock, which it does unless someone interferes. If you want a really accurate measure, you can always access the CPU cycle counter.

NTP on a lan produces distributed clock accuracy of 1ms or less, which was the goal. It also does not cause the clock to step backward so long as no root user dinks around with the system time on the ntp server. That was what the discussion was about.

Why are you quoting "which is most accurate"? I never used those words.

About whether gettimeofday provides a monotonic clock, maybe you should read its Linux man page:

The time returned by gettimeofday() is affected by discontinuous
jumps in the system time (e.g., if the system administrator manually
changes the system time). If you need a monotonically increasing
clock, see clock_gettime(2).

Did you read what I wrote. The unix time is monotonic UNLESS either a human manually sets the time, or there is a hardware failure. It does NOT happen otherwise. Back to the beginning post I wrote here, "giving a user root access is a security risk." <period>.

If you don't do that, the clock will always be monotonic. If you give a user root access, NO clock can guarantee monoticity. It is easy to write a program that can access kernel mode when you are root. And then you can change anything you want, the date / time is just a small part of what is at risk then.

So no, I am "not likely to change my mind" when my POV is clearly correct here. If you have no human to intervene, time is perfectly monotonic unless there is a hardware failure. If the hardware fails, ANYTHING can happen. Whoop-de-do, who would have thought that?
But for a running system with no root users, it works just as intended.