strcpy() revisited

[bob]

I feel exactly the same as when I started.

So you missed the point. http://www.youtube.com/watch?v=qQGgaI-BcI4#t=00m39s

They waste time on EVERY strcpy() to check for overlap.

That's normal in libraries to detect bugs in the calling code. If you think this is the permanent state of the library for release code, I suggest you submit a bug report.

It IS "in the release code version of the library". It is in the ONLY version of the library. Many have complained.

WRONGLY I might add because they only catch strcpy(a, a+n), but NOT strcpy(a+n, a) which is the one more likely to break. ... SO the burn cpu cycles to get the string length of the source. Then they only check one of the two overlapping cases where they should have checked both, and then they abort if they detect the first, ignore the second and let it crash,

If the lib is doing something wrong, submit a bug report. Maybe you can improve it to detect all cases of UB.

It takes one more add and one more compare. How hard is that? I think it is doing something wrong by detecting the overlap in the first place.

Seems perfectly reasonable to me. Somewhere. Some alternate universe. Where insanity reigns supreme... ... when they COULD have fixed both with a simple call to memmove().

Yes, let's cover up bugs instead of expose them.... that is insanity.

So, to recap the position you must believe, "In order to compile more efficiently and be able to use tricky optimizations, it is perfectly OK to slow down strcpy() by a factor of 2."??? I wonder if those "wonderful optimizations" will offset that factor of 2? That is, I wonder if this is one of those "useful optimizations" that slows the code down rather than speeding it up?

I suppose that is OK. But I can't even disable it to gain the speed back. I guess we call that a "de-optimizer"???

[bob]

Maybe they didn't think enough people would run in the debugging mode? They might have wanted programmers to always have the info, and just underestimated how many compiled binaries and source programs were out there in the wild, that relied on the UB.

Those existing programs are not affected, as the change is introduced during translation time, not in the link-time instance of strpcy(). One already must have rebuilt the project for the checks to enter, and then XCode would show the stack trace and the offending line right in your face. There is a reason there is no outcry about it other than here.

Are you talking about the same strcpy() Abort? That's not done by the compiler, it is done inside the lib. You can find Apple developers commenting on this and they even posted the lib source code showing the overlap check. This fails for gcc or clang on Mavericks, not on mountain lion, not on other platforms using the same version of gcc.

BTW most users don't run applications under Xcode.

[wgarvin]

Some versions of GCC have intrinsic knowledge of the overlaps-are-UB restriction of strcpy. Therefore, you can't CHANGE that restriction in just the library function and expect it to work (well you can on this Apple target since it doesn't do the optim, but in general you can't). Anyway, redefining it after 25+ years is not a simple step, even though its backward-compatible with correct programs. A lot of libraries and compilers would have to be changed and it would be a decade before programmers could rely on the new semantics.

...If you just want broken programs to "possibly" work anyway, on Apple's platform only, then sure they could change their library function to do what you suggest. Thats a relatively worthless change though. It might keep an incorrect program from crashing, or it might not. Maybe the result will be what the programmer intended, but since UB was involved, maybe it won't. The change might help Apple users who are stuck with the broken software, so maybe that is worthwhile. It won't really benefit the correct programs at all unless they are targeting Apple only--on other platforms it would still be UB.

[bnemias]

I feel exactly the same as when I started.

So you missed the point. http://www.youtube.com/watch?v=qQGgaI-BcI4#t=00m39s

They waste time on EVERY strcpy() to check for overlap.

That's normal in libraries to detect bugs in the calling code. If you think this is the permanent state of the library for release code, I suggest you submit a bug report.

It IS "in the release code version of the library". It is in the ONLY version of the library. Many have complained.

Great. So why are you complaining here? Add your considerable knowledge to that thread where it might do some good.

WRONGLY I might add because they only catch strcpy(a, a+n), but NOT strcpy(a+n, a) which is the one more likely to break. ... SO the burn cpu cycles to get the string length of the source. Then they only check one of the two overlapping cases where they should have checked both, and then they abort if they detect the first, ignore the second and let it crash,

If the lib is doing something wrong, submit a bug report. Maybe you can improve it to detect all cases of UB.

It takes one more add and one more compare. How hard is that? I think it is doing something wrong by detecting the overlap in the first place.

If it's so easy, when you file a report, you can even include a patch.

Seems perfectly reasonable to me. Somewhere. Some alternate universe. Where insanity reigns supreme... ... when they COULD have fixed both with a simple call to memmove().

Yes, let's cover up bugs instead of expose them.... that is insanity.

So, to recap the position you must believe, "In order to compile more efficiently and be able to use tricky optimizations, it is perfectly OK to slow down strcpy() by a factor of 2."??? I wonder if those "wonderful optimizations" will offset that factor of 2? That is, I wonder if this is one of those "useful optimizations" that slows the code down rather than speeding it up?

I said no such thing. Quit the strawman. It's just simple writing solid code 101. To find bugs in your programs, you expose them instead of covering them up. Speed is irrelevant since you catch bugs in non-release code.

The irony is you talk about speed degradation and then want the libraries to cater to code that relies on UB. Those are contrary, and library users should not have to have a sluggish library because their peers refuse to excise such things from their code.

[hgm]

I don't really understand how they can program for years and years in a language like C without ever learning how it works.

That is quite natural. If something works universally in a certain way everywhere and always, it becomes a de-facto standard. The law can specify that I am not allowed to cross the street at every corner, but if everyone does it, the chances of being fined for it will fall close to zero even if the entire police force would be put to work on it, so everyone will keep doing it and ignore the law.

The de-facto standard was that strcpy() to lower address would work even if there was overlap. I think it is very unwise to break such a de-facto standard. There really must be a very compelling reason to do such a thing. It would have been much more productive overall to elevate the de-facto standard to real standard. Restricting the UB in the strcpy spec to low-to-high overlapped copying, and requiring that it would work for high-to-low copying irrespective of overlap, would not have broken anything. It would of course make it impossible to benefit from the possibility to optimize copying speed by using forms of 8-by-8 copying and the like. But that could have been made available by a new routine, strblockcpy(), and everyone would be happy.

[bnemias]

It takes one more add and one more compare. How hard is that? I think it is doing something wrong by detecting the overlap in the first place.

You do realize that to implement the solution you've been advocating, covering up the bug by invoking memmove(), that it is necessary to actually detect the overlap?

[mvk]

Those existing programs are not affected, as the change is introduced during translation time, not in the link-time instance of strpcy(). One already must have rebuilt the project for the checks to enter, and then XCode would show the stack trace and the offending line right in your face. There is a reason there is no outcry about it other than here.

Are you talking about the same strcpy() Abort? That's not done by the compiler, it is done inside the lib. You can find Apple developers commenting on this and they even posted the lib source code showing the overlap check. This fails for gcc or clang on Mavericks, not on mountain lion, not on other platforms using the same version of gcc.

The magic is in the header file on my system: <string.h> under certain conditions includes <secure/_string.h>, which then aliases strcpy to __builtin_strcpy_chk. Whenever I link to the lib-supplied instance of strcpy (for example by using -U_FORTIFY_SOURCE, or running an old compile of crafty should also work), then it doesn't perform the check.

This should not be new information, and for me the point where I start to repeat myself means that I will prune my side of the discussion tree.

[syzygy]

Not really, because you were responding to a post by Wylie that was definitely about my example.

You really think you know better what I was talking about than I do myself?

Ok, so you quoted Wylie's post but you were not responding to it at all. Makes sense! Of course I should have understood this, just like a compiler should always understand what Bob intends with his non-standard compliant code.

[mvk]

Of course I should have understood this, just like a compiler should always understand what Bob intends with his non-standard compliant code.

Rumor is that gcc 4.1.14, which will be out in a few months from now, does that when you compile with --std=bobc

[syzygy]

[No idea what they are talking about there. There is no size argument to strcpy(). strcpy(dest, src) is all there is.

Yes, that is a strange mistake. I suppose it should have read string (or source string) argument.

This is interesting:

Code: Select all

#include <stdio.h>
#include <string.h>

int main&#40;int argc, char **argv&#41;
&#123;
  char *a = argv&#91;1&#93;;
  char b&#91;256&#93;;
  strcpy&#40;b, a&#41;;
  strcpy&#40;b+1, b&#41;;
  printf&#40;"strlen&#40;"%s") = %d\n", b, strlen&#40;b&#41;);
  return 0;
&#125;

Code: Select all

$ gcc -O3 bla.c
$ ./a.out 12345
strlen("112345") = 5

How can that be?

Answer: the program learns the original length of the string b from the first strcpy() which it implements using stpcpy(). The second strcpy() is implemented using memcpy(). Since strcpy(b+1, b) cannot possibly involve overlapping regions, this memcpy() cannot possibly change the length of b.

Say what? memcpy() does NOT say it copies byte by byte. In fact, it has several versions, some of which copy 8-byte-chunks. That certainly will overlap in your example.

That does not matter one bit as memcpy() never involves overlapping regions, see the standard.

So the program does not have to recalculate strlen(b) but can output the result it found earlier.

You consider that acceptable behavior?

Of course. The string is guaranteed not to change by the standard, so the strlen() calculated earlier is still valid. I consider this an excellent use of optimisation opportunities.

Btw, lots of code nowadays uses the restricted keyword. The natural prototype of strcpy() is:

Code: Select all

char *strcpy(char *restrict, const char *restrict);

Clearly the areas do not overlap (or you can expect anything to happen). strcpy() is hardly special in this.