Cleaning up X server warnings(keithp.com)
keithp.com
Cleaning up X server warnings
http://keithp.com/blogs/xserver-warnings/
3 comments
Reading serialized data and then casting it into a structure type is ill-defined, very bad practice, and there's nothing in C that tells you it's a good or safe thing to be doing. So, please don't do that.
The exact layout of a structure in memory is up to the compiler, and can easily change with compiler options, even if the compiler itself and the target architecture and platform remain the same.
An externally-visible serialization format should of course be stable, and not depend on the compiler used to build the code, or the hardware platform which might have ideas of how various fields should be aligned (or even how to order the bytes in integers larger than 8 bits).
Serializing data is something you do, just blindly copying the run-time representation that you have to an external medium is not serialization. You should do it field by field, with a well-defined format chosen for each field.
The exact layout of a structure in memory is up to the compiler, and can easily change with compiler options, even if the compiler itself and the target architecture and platform remain the same.
An externally-visible serialization format should of course be stable, and not depend on the compiler used to build the code, or the hardware platform which might have ideas of how various fields should be aligned (or even how to order the bytes in integers larger than 8 bits).
Serializing data is something you do, just blindly copying the run-time representation that you have to an external medium is not serialization. You should do it field by field, with a well-defined format chosen for each field.
That's the theory, sure. But in practice, if you know all your targets support the right "#pragma pack" and "__attribute__((packed))" and other markers that you use - what's wrong with making use of them?
Also, if you're passing data between two processes (e.g. your code fork()d a child), you can be sure that the data format is going to match, no packing required. (still need to ensure alignment in memory though)
This is not necessarily true. Lots of systems support running 32-bit and 64-bit binaries simultaneously, and structs won't necessarily be portable between the two. Some versions of Mac OS X even supported running 32-bit PowerPC binaries (in an emulator, but supported directly by the OS) next to both 32-bit and 64-bit x86 binaries, so you could not only get size mismatches, but also endian mismatches.
This does not apply to simply forking, of course, but it does apply to the "two processes" case in general.
This does not apply to simply forking, of course, but it does apply to the "two processes" case in general.
Oh yes, I was a bit unclear but I meant two processes (from the same binary) communicating.
Without that condition, you've got all the original problems, as listed. You can also have incompatibilities between compilers, or even compiler versions - e.g. gcc has ABI changes between some versions.
Without that condition, you've got all the original problems, as listed. You can also have incompatibilities between compilers, or even compiler versions - e.g. gcc has ABI changes between some versions.
On the same machine, it's going to be rare
But the most common occurrence of this I can think are online games.
But the most common occurrence of this I can think are online games.
Depends on what system and context you're talking about. I'd assume it's rare on Linux where most things are built from source, but it's pretty common on e.g. Apple platforms where binary compatibility means 32-bit processes keep being supported for years, and they still need to IPC to various system services. I've seen one highly amusing bug in Apple's iOS frameworks where they didn't quite handle the 32->64 IPC case properly because of an implicit assumption that data types remained identical.
The fact that you can't know all of your future targets will support that is a problem.
Endian issues are also a big problem here. The need to byte-swap everything eliminates a lot of the convenience.
It's also way too easy to make breaking changes to the struct, since there's no standard way to mark a struct as being something that you serialize/deserialize, and normally you can change struct fields at will in C code.
Just take the extra five minutes to write code to translate between your struct and a stream of bytes. It'll be easier to understand, less risky, and more compatible.
Endian issues are also a big problem here. The need to byte-swap everything eliminates a lot of the convenience.
It's also way too easy to make breaking changes to the struct, since there's no standard way to mark a struct as being something that you serialize/deserialize, and normally you can change struct fields at will in C code.
Just take the extra five minutes to write code to translate between your struct and a stream of bytes. It'll be easier to understand, less risky, and more compatible.
"The fact that you can't know all of your future targets will support that is a problem."
I don't see this as much of a problem as endianness, but yeah, maybe, x86 made us accustomed. And you can use portable types, defined on headers (linux does that, like u8, u16, etc)
But sometimes you know your target won't change (for a long time)
"It's also way too easy to make breaking changes to the struct, since there's no standard way to mark a struct as being something that you serialize/deserialize, and normally you can change struct fields at will in C code."
In the same way you can break your software doing any change. You can mark it with a naming convention but the easiest way is seeing the "pack" directives on it. And unit tests
"Just take the extra five minutes to write code to translate between your struct and a stream of bytes. It'll be easier to understand, less risky, and more compatible. "
Well, it's not five minutes. And it makes your program slower (for the most convoluted data types). A simple example:
http://en.wikipedia.org/wiki/BMP_file_format
And yes, GIMP reads field by field https://git.gnome.org/browse/gimp/tree/plug-ins/file-bmp/bmp...
I don't see this as much of a problem as endianness, but yeah, maybe, x86 made us accustomed. And you can use portable types, defined on headers (linux does that, like u8, u16, etc)
But sometimes you know your target won't change (for a long time)
"It's also way too easy to make breaking changes to the struct, since there's no standard way to mark a struct as being something that you serialize/deserialize, and normally you can change struct fields at will in C code."
In the same way you can break your software doing any change. You can mark it with a naming convention but the easiest way is seeing the "pack" directives on it. And unit tests
"Just take the extra five minutes to write code to translate between your struct and a stream of bytes. It'll be easier to understand, less risky, and more compatible. "
Well, it's not five minutes. And it makes your program slower (for the most convoluted data types). A simple example:
http://en.wikipedia.org/wiki/BMP_file_format
And yes, GIMP reads field by field https://git.gnome.org/browse/gimp/tree/plug-ins/file-bmp/bmp...
Does it really make your program slower? That padding you have to eliminate for this technique is added for speed, after all. The initial read may be faster, but every access to the data is going to be slower.
I'm not quite sure what the BMP file format is supposed to be an example of, especially since file formats are separate from techniques used to read or write them.
I'm not quite sure what the BMP file format is supposed to be an example of, especially since file formats are separate from techniques used to read or write them.
Some years ago I had a discussion with a then-colleague who pointed out something interesting that stuck with me: the world has mostly settled on little endian. If you stay outside of certain niches you are unlikely to ever see a big endian CPU.
Time was you might have to run on a SPARC or PowerPC or whatever. But x86 and little-endian ARM have pretty much won for most people.
I don't think my friend was right that this means you can totally ignore the issues, but I have to admit he was right that from a strictly practical perspective it's not the huge deal it was in the 90s. I agree with you that you don't know how the future will break you, but my guess is the momentum of binary compatibility will keep it so for a while.
Time was you might have to run on a SPARC or PowerPC or whatever. But x86 and little-endian ARM have pretty much won for most people.
I don't think my friend was right that this means you can totally ignore the issues, but I have to admit he was right that from a strictly practical perspective it's not the huge deal it was in the 90s. I agree with you that you don't know how the future will break you, but my guess is the momentum of binary compatibility will keep it so for a while.
That's a good point, and he's largely right. On the other hand, I've been burned before (Macs were always going to be 68k until they went PPC, they were always going to be PPC until they went Intel, they were always going to be Intel until Apple made these tiny mini-Macs that were ARM) and I'm wary of making assumptions anymore.
This is usually my kind of instinct too, which is why I was attributing the other viewpoint to someone else. But it was interesting to understand where someone like that is coming from. You can spend a lot of time adding the right swaps etc. but in the end if you don't own hardware that works that way, aren't testing it regularly, from a certain perspective you may be wasting your time.
OTOH I recall that insightful article from Rob Pike about how the "right" way to do it in a testable fashion is to not think in terms of swapping at all, and just do shifts that are portable regardless of architecture. http://commandcenter.blogspot.com/2012/04/byte-order-fallacy...
(By the way, in your 68k -> PPC -> Intel -> ARM example, the endianness only changes once. This was actually part of my friend's original argument, that endianness changes are even more costly than the instruction set, and platform vendors would be unwise to change it. "Modern" CPUs support both endianness types, so those shipping ARM platforms are in effect consciously deciding to be the same as Intel.)
OTOH I recall that insightful article from Rob Pike about how the "right" way to do it in a testable fashion is to not think in terms of swapping at all, and just do shifts that are portable regardless of architecture. http://commandcenter.blogspot.com/2012/04/byte-order-fallacy...
(By the way, in your 68k -> PPC -> Intel -> ARM example, the endianness only changes once. This was actually part of my friend's original argument, that endianness changes are even more costly than the instruction set, and platform vendors would be unwise to change it. "Modern" CPUs support both endianness types, so those shipping ARM platforms are in effect consciously deciding to be the same as Intel.)
For what it's worth, I completely agree with Pike there. It makes a lot more sense to me, and refrains from completely falling afoul of the C standard.
> The fact that you can't know all of your future targets will support that is a problem.
To be honest I've even seen compilation issues coming up between ubuntu 10.x and 11.x. I don't worry about future targets that much. Until they're tested, I assume they will not work.
To be honest I've even seen compilation issues coming up between ubuntu 10.x and 11.x. I don't worry about future targets that much. Until they're tested, I assume they will not work.
> But in practice, if you know all your targets support the right "#pragma pack" and "__attribute__((packed))" and other markers that you use - what's wrong with making use of them?
You know that at this moment they all support the right __attribute__((packed)) et co.. Bonus points for endianness crap.
You know that at this moment they all support the right __attribute__((packed)) et co.. Bonus points for endianness crap.
All of that is true unless it is, say, cache data that you can afford to ditch if the format has changed, and you validate it after reading (say with a carefully structured "magic" value).
I thought that the issue is the const-ness of the variables, and nothing to do with casting void* ->sometype*
Essentially, they are making use of const pointers to ensure that the code doesn't change the data. (gcc would throw a warning if you did). BUT: the problem comes when you want to free() the data. A strict interpretation of C would be that you can't free() something that's const, because it clearly is altering the data. However, you have to free the data sometime or other...
A possible workaround is to write a freeconst() macro that does the cast to void* and calls free(), and wrap this macro in the 'temporarily disable this warning #pragma'. This way, you only need to turn off the warning in one place in your code.
Essentially, they are making use of const pointers to ensure that the code doesn't change the data. (gcc would throw a warning if you did). BUT: the problem comes when you want to free() the data. A strict interpretation of C would be that you can't free() something that's const, because it clearly is altering the data. However, you have to free the data sometime or other...
A possible workaround is to write a freeconst() macro that does the cast to void* and calls free(), and wrap this macro in the 'temporarily disable this warning #pragma'. This way, you only need to turn off the warning in one place in your code.
The BSD solution to this problem is __DECONST(), a macro which basically casts through `uintptr_t`.
#define __DECONST(type, var) ((type)(uintptr_t)(const void*)(var))On a related note, if you're using new and delete you can simply delete a pointer-to-const without jumping through any const_cast hoops, though I often wish I had a way of blocking that for APIs where the callee doesn't take ownership of a const Foo*.
Is that actually a standard though? ISTR some compilers complaining if you use delete with a const pointer (sorry, this is from ages ago, I cannot recall which compiler...)
You can make a fair argument that using delete with a const pointer is a type error, after all the data can hardly be considered const if it has just been eradicated...
You can make a fair argument that using delete with a const pointer is a type error, after all the data can hardly be considered const if it has just been eradicated...
The notion of warnings is not that it the behavior is broken, or even wrong, but rather dangerous. A good solution would be to include a pragma or such that that would allow programmers to say "I recognize this is dangerous, but it is correct".
Even better would be a pragma that allows you to say the above, and also point to a test/test_suit that verifies that the specific case is still working right and warns on "something has changed, test foo no longer holds, here's the associated warning as a hint".
Even better would be a pragma that allows you to say the above, and also point to a test/test_suit that verifies that the specific case is still working right and warns on "something has changed, test foo no longer holds, here's the associated warning as a hint".
Well, some warnings are valid, but yeah, the "I recognize this is dangerous, but it is correct" makes perfect sense.
I hate those GTK runtime errors. When I run a gtk app from a shell & find pages of warnings, I lose all faith in the app.
If you're not careful and not on x86-likes, then reading serialized data like that unaligned values might cause errors.
I always wondered why so many FOSS projects always generate warnings. I always assumed that the large mature projects like the X server did this because they were performing some crazy optimization that the compiler was too naive to know about. I never had to do this myself, so I thought that the pro's used their arcane knowledge to eek out an extra 0.1% performance out of something for the benefit of everyone. Turns out that the truth is so much less interesting.
Often, it will be because the projects are older than the warnings. And when the amount of warnings cross a threshold, they will not be fixed.
Yes, definitely this. If there are hundreds of warnings, there's no motivation to fix any of them. If there are none, hopefully no-one wants to add the code that creates one.
Perhaps they now need to compile the project with warnings-as-errors to enforce clean code?
Perhaps they now need to compile the project with warnings-as-errors to enforce clean code?
It always surprised me that GCC has no source-level way of controlling compiler warnings.
Many programmers (and gcc apparently) live with the assumption that compiler warnings must always be fixed. This is largely false. Compiler warnings are there to help with additional information, not to be something to fix. There are many cases where the compiler issues a warning which must be ignored.
Many compilers support pragmas to silence a specific warning class in a specific point in the code. IMHO it's a much better approach to silence just a single warning instance where you know the compiler is getting the wrong assumption than silencing a whole warning class like the article suggests.
But it seems that in GCC you have no choice. Instead, code written with/for gcc often silences the warning by inserting some dummy code. For instance, the "unitialized warning" is usually fixed by assigning the variable to itself. It's shitty to look at, and someone that doesn't know this trick might wonder why this is being done.
I've reported a feature request many years ago, and it was quickly closed...
Many programmers (and gcc apparently) live with the assumption that compiler warnings must always be fixed. This is largely false. Compiler warnings are there to help with additional information, not to be something to fix. There are many cases where the compiler issues a warning which must be ignored.
Many compilers support pragmas to silence a specific warning class in a specific point in the code. IMHO it's a much better approach to silence just a single warning instance where you know the compiler is getting the wrong assumption than silencing a whole warning class like the article suggests.
But it seems that in GCC you have no choice. Instead, code written with/for gcc often silences the warning by inserting some dummy code. For instance, the "unitialized warning" is usually fixed by assigning the variable to itself. It's shitty to look at, and someone that doesn't know this trick might wonder why this is being done.
I've reported a feature request many years ago, and it was quickly closed...
I thought thats exactly what these are for:
http://gcc.gnu.org/onlinedocs/gcc/Diagnostic-Pragmas.html
Also work for clang like so (replace clang with GCC for its version):
Also work for clang like so (replace clang with GCC for its version):
#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wsomeignoredwarning"
triggered by this code
#pragma clang diagnostic popSure, sometimes they warn about totally valid behaviors. The "all warnings must be fixed" dogma arises not out of the belief that all warnings are errors, but rather that the value of having compiler warnings catch bugs early is well worth the cost of changing code style to conform to the compiler's warnings suite.
gcc has this capability:
http://gcc.gnu.org/onlinedocs/gcc/Diagnostic-Pragmas.html
I recon you meant unused variables (as assinging an unitialized variable to itself would beat the purpose) you can use __attribute__((unused)) on it - normally #defined to a macro.
I recon you meant unused variables (as assinging an unitialized variable to itself would beat the purpose) you can use __attribute__((unused)) on it - normally #defined to a macro.
Interesting how the -Wcast-qual warnings are unfixable, so it shouldn't exist. Damned if you do, damned if you don't. A lot of warnings seem to fall on this.
Or warnings that would make sense on stricter languages but doesn't make sense on C exactly because of what you can do with C, like, for example, reading serialized data, then casting it to (MyStructure *)
And a little bit offtopic, but GTk have some unfixable warnings as well, something like "blah is not implemented in this platform" so, if I have to use this, but BLAH is not implemented, thanks for warning me, but shut up if I can't fix it