Why did the OpenSSL punycode vulnerability happen?(words.filippo.io)
words.filippo.io
Why did the OpenSSL punycode vulnerability happen?
https://words.filippo.io/dispatches/openssl-punycode/
14 comments
Even in C, you can write an abstraction for a growable buffer. The problem is, you have to make all the rest of the code work with it, rather than a char* or whatnot.
BoringSSL and libressl added CBS (bytestring) and CBB (bytebuilder) interfaces in 2015. Converting everything over is a chore, but that's exactly what those projects have been methodically doing over the years.
Unfortunately, I don't see those interfaces in OpenSSL 3.0, though maybe (hopefully) they're working toward something similar.
Unfortunately, I don't see those interfaces in OpenSSL 3.0, though maybe (hopefully) they're working toward something similar.
How many projects have to keep re-inventing those kind of data structures until WG14 takes security seriously?
Then you have the new problem of restricting its growth and that adds a new failure mode to deal with. The C idiom works fine if you don't have implementation bugs.
Null terminated string is probably the worst data structure that was invented. It is has caused numerous security problems. It takes worst performance characteristics from array and linked lists (slow resizing, slow indexing). It definitely does not work fine.
Resizing arrays is such an annoying problem... because memory address space is flat. If only we had some sort of resizeable, non-overlapping by design segments, kind of like we have virtual memory mappings... but that'd completely kill address arithmetic. Oh well.
Everything has that failure mode, in every language.
Writing into a zero length output buffer does not fail and consumes no memory resources.
Who cares? You still have to write the code that handles failure when the buffer length isn't zero.
Or you can use char * and realloc (or a callback) for it to grow, if you're not forced to use caller allocated buffers.
> Not involved in OpenSSL, but this is a fairly common pattern in a lot of C APIs.
A better approach IMHO, typically found in the Win32 API for example, is to take an extra argument which receives the buffer size needed to hold the whole output, in addition to the size of the buffer you pass.
This allows the caller to detect the condition and decide if it's an error or not if the required buffer size returned is greater than the size of the buffer, and potentially call it again with a sufficiently large buffer.
A better approach IMHO, typically found in the Win32 API for example, is to take an extra argument which receives the buffer size needed to hold the whole output, in addition to the size of the buffer you pass.
This allows the caller to detect the condition and decide if it's an error or not if the required buffer size returned is greater than the size of the buffer, and potentially call it again with a sufficiently large buffer.
> A better approach IMHO, typically found in the Win32 API for example, is to take an extra argument which receives the buffer size needed to hold the whole output, in addition to the size of the buffer you pass.
This is exactly what the function in question does and is what lead to the buffer overflow. Basically, this sort of API requires you to continue decoding (so you can calculate the length), but stop writing to the output buffer once the buffer is full. The function had a bug that lead it to keep writing to the buffer in some cases even after it was full.
This is exactly what the function in question does and is what lead to the buffer overflow. Basically, this sort of API requires you to continue decoding (so you can calculate the length), but stop writing to the output buffer once the buffer is full. The function had a bug that lead it to keep writing to the buffer in some cases even after it was full.
Isn't that roughly how heartbleed works? Trusting the provided length?
If the caller is an external actor, yes, their provided length should not be trusted. However, this is not always the case. The caller may be another part of the same program, trusted not to perform malicious actions to the same extent that the rest of the program is trusted.
Hearbleed was due to failure to sanitize external data[1], not sure how that's directly relevant to what I wrote.
I mean sure if the caller fails to sanitize and as a result passes a size that's bigger than the actual buffer, the callee doesn't have much in the way of detecting that. But that's a general C issue, nothing specific to what I wrote.
[1]: https://git.openssl.org/gitweb/?p=openssl.git;a=commitdiff;h...
I mean sure if the caller fails to sanitize and as a result passes a size that's bigger than the actual buffer, the callee doesn't have much in the way of detecting that. But that's a general C issue, nothing specific to what I wrote.
[1]: https://git.openssl.org/gitweb/?p=openssl.git;a=commitdiff;h...
I am by no means a rust developer and asking in ignorance, would this have been a problem had it been written in rust?
> would this have been a problem had it been written in rust?
The answer would be "it depends on whether you consider denial-of-service a problem". The key detail which makes all the difference is that, unless you're playing with raw pointers (which can only be dereferenced in "unsafe" blocks), the pointer to a buffer slice is always kept together with its length (in a "fat pointer"). Attempting to write through it past the buffer's bounds will result in a Rust panic, which usually aborts the whole process (there are ways to abort just one thread, or even to treat it similarly to a C++ exception, but a library cannot depend on them since the program might be compiled in the panic=abort mode). While that's obviously better than allowing for remote code execution, it still could be considered an issue.
Of course, that's assuming you want a similar API and are writing the code in a similar style, just replacing the direct pointer manipulation with slice manipulation. I don't know whether, in this particular case, more idiomatic Rust code would have avoided the issue. And, of course, the "growable container" approach would completely avoid it, but Rust is also used in places where memory allocation is not allowed, so having a non-allocating API still makes sense.
The answer would be "it depends on whether you consider denial-of-service a problem". The key detail which makes all the difference is that, unless you're playing with raw pointers (which can only be dereferenced in "unsafe" blocks), the pointer to a buffer slice is always kept together with its length (in a "fat pointer"). Attempting to write through it past the buffer's bounds will result in a Rust panic, which usually aborts the whole process (there are ways to abort just one thread, or even to treat it similarly to a C++ exception, but a library cannot depend on them since the program might be compiled in the panic=abort mode). While that's obviously better than allowing for remote code execution, it still could be considered an issue.
Of course, that's assuming you want a similar API and are writing the code in a similar style, just replacing the direct pointer manipulation with slice manipulation. I don't know whether, in this particular case, more idiomatic Rust code would have avoided the issue. And, of course, the "growable container" approach would completely avoid it, but Rust is also used in places where memory allocation is not allowed, so having a non-allocating API still makes sense.
It should be noted that you can simply use the fallible API's for slices in Rust, then you don't have a crash but can cleanly abort the operation and return an error.
True, but it doesn't make much sense to use the fallible slice APIs when you know they will never fail (the only way they could fail would be if the code has a bug). It would just complicate not only the code within the function, but also the API to the function, which also becomes a fallible API (and this propagates outward, until it meets a caller which already was fallible for other reasons). And complicating the code unnecessarily increases the chance of logic bugs.
It's actually preferrable in many of those "infallible" cases to panic rather than handling the error since it usually indicates that your application is in unknown territory and there is no "safe" way to handle it.
If you're writing cryptographic code, panicking for every issue is not good, that's a DoS vector that could be easily exploited. It's safer to handle the error in-band and let the application higher up decide if it should panic, error or continue.
IMO you should use the fallible slice API's in any safety-critical code, such a cryptographic code.
Yes, there is more code, but it does not become a lot more complex, if you need to you can unwrap to explicitly panic. You should still insert asserts to catch issues. But if there is an issue, such as running out of memory or anything else, you can handle it more appropriately than producing a Denial-of-Service issue immediately, which is definitely not good.
Yes, there is more code, but it does not become a lot more complex, if you need to you can unwrap to explicitly panic. You should still insert asserts to catch issues. But if there is an issue, such as running out of memory or anything else, you can handle it more appropriately than producing a Denial-of-Service issue immediately, which is definitely not good.
Rust would take the growable container approach here, so it wouldn't be vulnerable in the same way. There's nothing in C that prevents you from doing this either, but as the parent post mentions it's common style to do this (see also things like snprintf).
> There's nothing in C that prevents you from doing this either
Yet these errors consistently happen in C projects of various sizes.
Yet these errors consistently happen in C projects of various sizes.
I am a Rust evangelist as much as the next person, but this is really a case of C developers preferring caller reallocation over callee reallocation, which as the root comment points out is fraught with danger. If the C implementation here used callee reallocation, while you do still have to be careful, the risk of this kind of error is greatly reduced (but at the cost of having to use dynamic memory, which might not be appropriate in all cases).
Yes, Rust would eliminate this error, but you can still do it "safer" in C (but you have to give up certain things to do it that way).
Yes, Rust would eliminate this error, but you can still do it "safer" in C (but you have to give up certain things to do it that way).
Because all the important and used stuff that gets looked at is written in C.
At the moment rust is used for some very marginal and unimportant leaf projects.
At the moment rust is used for some very marginal and unimportant leaf projects.
Might be some survivorship bias.
In no other mainstream language other than C (and maybe non-idiomatic C++) would this have ever been a problem.
That is fair, though, I'd guess that no other mainstream language except for modern C++ and Rust would fit the bill of OpenSSL. Both of which are embarrassingly recent.
It is not a failure of C that we couldn't improve on it for so many decades.
It is not a failure of C that we couldn't improve on it for so many decades.
Yeah, most folks in C++ would just use an std::vector, right? I guess if you wanted to use the modern abstraction of std::array, it could be a problem, since using `operator[]` on a non-existent element produces UB.
It is an exercise in navel gazing until an organization steps up to fund: 1) a linkable clib harness that exports Rust functions to replace OpenSSL functions. and 2) pays to get the resulting harness and underlying code FIPS certified so people can use it.
Rust code would probably return an error on output buffer overflow instead of siltently discarding data. Something like this: https://docs.rs/base16ct/latest/base16ct/mixed/fn.decode.htm...
Probably not. It's definitely possible to write this function exactly as it is in openssl with this issue, nothing about Rust's safety claims prevent it. However, Rust doesn't have the same issues that lead to this kind of pattern being common like they are in C. Result<> types being so common, better handling of dynamic allocations, and at least for now a generally (it can't last forever) more security conscious community mean that other patterns will be chosen unstead. I'm not sure any language can actually prevent it from being possible outright, but you possibly pair it with some kind of formal proof system like using Coq and TLA+ might let you prove thag an issue like this doesn't exist in a specific implementation. However it's incredibly hard to do this.
Not in this case as far as I can tell.
So the blog mentions that in certain cases you have to decode the punycode from one field to compare it to the value in another field.
Would it have been safer to encode the data in the other field to punycode and compare the encoded values?
That way a hacker can’t mess with your decoder (where bugs like to lie). But at the other end you risk that your encoder has an issue. I do t know how to judge if those are equal risks or not.
Thoughts?
Would it have been safer to encode the data in the other field to punycode and compare the encoded values?
That way a hacker can’t mess with your decoder (where bugs like to lie). But at the other end you risk that your encoder has an issue. I do t know how to judge if those are equal risks or not.
Thoughts?
Encoding and decoding are both transforms on attacker supplied input.
I don't think its helpful to armchair quarterbacking other peoples mistakes that ended up in a vulnerability when the overall code quality is good. There are a dozen ways they could have done it differently, which may or may not have resulted in different exotic bugs.
Punycode was the footgun here. Not the language, or the implementation, or the code. They were forced to do something stupid and complex and dangerous deep within the bowels of a critical library.
I don't think its helpful to armchair quarterbacking other peoples mistakes that ended up in a vulnerability when the overall code quality is good. There are a dozen ways they could have done it differently, which may or may not have resulted in different exotic bugs.
Punycode was the footgun here. Not the language, or the implementation, or the code. They were forced to do something stupid and complex and dangerous deep within the bowels of a critical library.
If you want to avoid risks both in the encoder and decoder, you might have success with a function that takes an encoded strict and a decoded string and compares them directly, encoding/decoding (whatever works better) on the fly. This avoids the allocation altogether, at the cost of having another less-composable piece of code that must be maintained.
I'm not sure if that is the way to go. An allocation per se isn't hard to get right, especially in this case where its scope is well-defined and very limited. What happened here is more the fact that while a single piece of code is easy to get right, a whole codebase isn't. So the only value added by my proposed solution above is that it works in an environment where you can't allocate (e.g. memory-constrained systems where you need to know the required amount of memory statically).
Bottom line: I don't think you can really solve this problem on a technical level.
edit: typo
I'm not sure if that is the way to go. An allocation per se isn't hard to get right, especially in this case where its scope is well-defined and very limited. What happened here is more the fact that while a single piece of code is easy to get right, a whole codebase isn't. So the only value added by my proposed solution above is that it works in an environment where you can't allocate (e.g. memory-constrained systems where you need to know the required amount of memory statically).
Bottom line: I don't think you can really solve this problem on a technical level.
edit: typo
It feels like issues like those are more common in parsers, this specific kind of software.
But why?
Why is parsing so hard? or is it just in low lvl languages? or maybe languages with poor string primitives?
I've written parsers in high level languages and it didnt felt dangerous or insanely hard
But why?
Why is parsing so hard? or is it just in low lvl languages? or maybe languages with poor string primitives?
I've written parsers in high level languages and it didnt felt dangerous or insanely hard
Poor string primitives is the answer, I think. Or more generally, C’s tendency to require a buffer of sufficient size to be passed for output, and the complexity involved with:
Callers also have to make sure the buffer size they’re passing to the parser is actually accurate… you don’t want to malloc 100 bytes and tell the parser the buffer is 200. They also have to make sure that if they say the buffer is 100 bytes, that the string actually terminates with a \0 by the end of it, or it gets even more confusing.
It’s overall a complicated problem, and arises specifically because of the tendency of C library functions to avoid allocations… because the ownership model of idiomatic C is that callers should be the ones that allocate (because they probably know best how to allocate and when to free, etc.)
- What size to pick for the buffer
- Making sure the buffer is of the size you think it is
- What to do when the buffer isn’t big enough
- How to know for sure that the buffer is big enough
Callers also have to make sure the buffer size they’re passing to the parser is actually accurate… you don’t want to malloc 100 bytes and tell the parser the buffer is 200. They also have to make sure that if they say the buffer is 100 bytes, that the string actually terminates with a \0 by the end of it, or it gets even more confusing.
It’s overall a complicated problem, and arises specifically because of the tendency of C library functions to avoid allocations… because the ownership model of idiomatic C is that callers should be the ones that allocate (because they probably know best how to allocate and when to free, etc.)
It's not hard to use a "vector" library or even just a small set of macros that ensure memory safety when you are manipulating slices of strings or buffers.
Unfortunately it's very tempting to play fast and loose with raw pointers, with ad hoc validation logic all over the place.
OTOH, maybe SSL maintainers consider punycode parsing performance more important than security.
Unfortunately it's very tempting to play fast and loose with raw pointers, with ad hoc validation logic all over the place.
OTOH, maybe SSL maintainers consider punycode parsing performance more important than security.
Parsing isn’t all that hard, it’s about working with the right tools and level of abstractions. Messing around in C with null-terminated strings is just hilariously error prone.
I think it's easy to hand write a parser that can get into a weird state. Some formats are also easier to write a parser for safely vs others.
For example, if you have a Tag Length Value data format that says "5ABCDE" maybe that means "the next 5 bytes are a string, then there's something else afterwards or it's the end", you don't want to allocate a buffer of 5 values and just keep writing into it without checking that you're reading valid values.
Doing this sort of thing efficiently may also mean that you're tempted to remove something like a bounds check. After all, if you already know that the value is 5 bytes, why check on every access? People really want parsers to be fast. Ideally in the above example I wouldn't even need to copy those bytes out, I could just reference those values, which now leads to potential lifetime issues.
Further, in C, buffers don't have lengths attached to them. In every other language you typically don't work with null terminated strings. So now you have to manage sizes of things throughout your parser.
One way to view this is that the programmer's mental model of the parsing machine can easily drift from the implementation of the parsing machine, leading to vulnerabilities.
Basically it ends up being very easy to accidentally end up with out of bounds reads/writes + there's pressure to be fast + formats can be very complex.
That's my view on it at least.
For example, if you have a Tag Length Value data format that says "5ABCDE" maybe that means "the next 5 bytes are a string, then there's something else afterwards or it's the end", you don't want to allocate a buffer of 5 values and just keep writing into it without checking that you're reading valid values.
Doing this sort of thing efficiently may also mean that you're tempted to remove something like a bounds check. After all, if you already know that the value is 5 bytes, why check on every access? People really want parsers to be fast. Ideally in the above example I wouldn't even need to copy those bytes out, I could just reference those values, which now leads to potential lifetime issues.
Further, in C, buffers don't have lengths attached to them. In every other language you typically don't work with null terminated strings. So now you have to manage sizes of things throughout your parser.
One way to view this is that the programmer's mental model of the parsing machine can easily drift from the implementation of the parsing machine, leading to vulnerabilities.
Basically it ends up being very easy to accidentally end up with out of bounds reads/writes + there's pressure to be fast + formats can be very complex.
That's my view on it at least.
>Doing this sort of thing efficiently may also mean that you're tempted to remove something like a bounds check. After all, if you already know that the value is 5 bytes, why check on every access? People really want parsers to be fast.
In what kind of software bounds check have this significant perf. penalty?
The only people I've heard talking about such a stuff were firmware devs.
>Further, in C, buffers don't have lengths attached to them. In every other language you typically don't work with null terminated strings. So now you have to manage sizes of things throughout your parser.
Cannot C have some wrapper over those poor strings
that leads to better safety? dev's experience, etc, etc?
In what kind of software bounds check have this significant perf. penalty?
The only people I've heard talking about such a stuff were firmware devs.
>Further, in C, buffers don't have lengths attached to them. In every other language you typically don't work with null terminated strings. So now you have to manage sizes of things throughout your parser.
Cannot C have some wrapper over those poor strings
that leads to better safety? dev's experience, etc, etc?
> In what kind of software bounds check have this significant perf. penalty?
All kinds. But a lot of it is just that parsers are extremely easy to benchmark and benchmarks promote optimization.
> Cannot C have some wrapper over those poor strings
Sure, but there's nothing native.
All kinds. But a lot of it is just that parsers are extremely easy to benchmark and benchmarks promote optimization.
> Cannot C have some wrapper over those poor strings
Sure, but there's nothing native.
>Sure, but there's nothing native.
But why? strings are used by all programmers everyday
I struggle to understand why you wouldn't want to make them state of the art, or decent at least.
But why? strings are used by all programmers everyday
I struggle to understand why you wouldn't want to make them state of the art, or decent at least.
> > Sure, but there's nothing native.
> But why? strings are used by all programmers everyday
Different kinds of strings can have extremely different performance profiles.
A statically allocated string of ASCII (single byte) characters.
A dynamically allocated string of unicode (multibyte) characters where the length of allocation is not known ahead of time is very different. C requires the developer to know the differences and knows how to deal with them treat them.
> But why? strings are used by all programmers everyday
Different kinds of strings can have extremely different performance profiles.
A statically allocated string of ASCII (single byte) characters.
A dynamically allocated string of unicode (multibyte) characters where the length of allocation is not known ahead of time is very different. C requires the developer to know the differences and knows how to deal with them treat them.
C is an extremely conservative language. You don't even get booleans unless you're in C99 and import it.
You do get them without #include <stdbool.h>, you'll just have to contend with ugly spellings like _Bool and _True.
bool, true, and false as native keywords are on track to show up in C23. https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2393.pdf
Because WG14 doesn't care about security, for them C is basically a portable macro assembler with added conveniences.
Your questions are less interrelated than you might think.
Why is general parsing hard? It's sense-making from free form input. We have so many different languages and syntaxes and there are different algorithmic approaches needed to parse them depending on the format and language, frequently made as dense as possible for various more or less (usually more) misguided "optimization" reasons.
Why is parsing in low level languages hard? This is more about incidental features of low level languages. C just happens to be exceptionally unsuited and unsafe for the tasks involved in parsing. From a safety and security POV C is chainsaw juggling, but C for parsing untrusted data is chainsaw juggling while running through a mine field.
Why is general parsing hard? It's sense-making from free form input. We have so many different languages and syntaxes and there are different algorithmic approaches needed to parse them depending on the format and language, frequently made as dense as possible for various more or less (usually more) misguided "optimization" reasons.
Why is parsing in low level languages hard? This is more about incidental features of low level languages. C just happens to be exceptionally unsuited and unsafe for the tasks involved in parsing. From a safety and security POV C is chainsaw juggling, but C for parsing untrusted data is chainsaw juggling while running through a mine field.
I'll give you a tautological and useless answer. I hope you don't mind.
A parser is a kind of interpreter, where code is executed based on external input. When user input controls how code is executed you have opened the doors of hell: it's hard to guarantee that all of the possible executions are safe.
But why?
Parsers executes safe code fragments in order based on the input
How safe operations result in unsafe results?
I'm not talking about side channels here.
Parsers executes safe code fragments in order based on the input
How safe operations result in unsafe results?
I'm not talking about side channels here.
Well, I see two ways it might be: the operations are not really safe in an absolute sense or the operations composition does not preserve safety.
Or you could say that an operation can be safe in a context but ops+context does not compose as well as we would like to guarantee the preservation of safety
Probably low level + safe + performant == hard
But why? where does the complexity come from
There are naturally lots of edge cases when you parse a format, because you have to constrain the combination of all the different fields.
Some formats are simple and the fields don't interact with each other at all, some are complex and the format changes depending on other values.
Parsing is hard because you have to handle all the possible inputs someone could throw at you, and depending on the format that can leave hundreds of very rare edge case no reasonable human would normally think of.
This is also why fuzzing is so effective on parser, fuzzers are great at throwing many different combinations at the wall until they find a new interesting edge case, and jumping off from there to see if they can mutate it into more.
Some formats are simple and the fields don't interact with each other at all, some are complex and the format changes depending on other values.
Parsing is hard because you have to handle all the possible inputs someone could throw at you, and depending on the format that can leave hundreds of very rare edge case no reasonable human would normally think of.
This is also why fuzzing is so effective on parser, fuzzers are great at throwing many different combinations at the wall until they find a new interesting edge case, and jumping off from there to see if they can mutate it into more.
You are of course correct. This also ties into my sibling comment - the cleverness in clever low-level parsing code is based on assumptions that may be wrong for some part of the huge input space.
The first level of complexity comes from the format. A bit array is super easy to parse (in C, and assuming you take care of endianness). JSON is more complicated; YAML is more complicated than JSON; XML is more complicated than YAML; X.509 is more complicated than XML (I think, anyway). The more complex the data format, the more complex the parsing; the more complex; the more opportunity for bugs.
The second level of complexity comes from variability. A bit array doesn't vary, every bit is in the same place. A string varies. Anything that can vary causes complexity; more varying, more complexity. This applies to the data format and the data.
The third level of complexity comes from features. Every feature is a new thing that has to be parsed and then affects some code somewhere, the result of which affects more parsing and code. The more features and options there are, the more complexity.
"Why does it seem easier in high-level languages?" High-level languages have slowly had their bugs stripped out, and give you features that are rarer in low-level languages. You literally aren't writing the same routines in high-level languages because you don't need to. If you had to do all the same things, you'd have the same bugs. And a lot of newbies simply are lucky and don't personally run into the bugs that are already there.
The second level of complexity comes from variability. A bit array doesn't vary, every bit is in the same place. A string varies. Anything that can vary causes complexity; more varying, more complexity. This applies to the data format and the data.
The third level of complexity comes from features. Every feature is a new thing that has to be parsed and then affects some code somewhere, the result of which affects more parsing and code. The more features and options there are, the more complexity.
"Why does it seem easier in high-level languages?" High-level languages have slowly had their bugs stripped out, and give you features that are rarer in low-level languages. You literally aren't writing the same routines in high-level languages because you don't need to. If you had to do all the same things, you'd have the same bugs. And a lot of newbies simply are lucky and don't personally run into the bugs that are already there.
>"Why does it seem easier in high-level languages?" High-level languages have slowly had their bugs stripped out, and give you features that are rarer in low-level languages. You literally aren't writing the same routines in high-level languages because you don't need to. If you had to do all the same things, you'd have the same bugs. And a lot of newbies simply are lucky and don't personally run into the bugs that are already there.
Which bugs precisely are you talking about?
Sane string implementation, so instead of performing some shenanigans with buffers to concat two strings, I can just "a" + "b"?
>If you had to do all the same things, you'd have the same bugs.
Why in lower level languages people cannot write some handy abstractions which will result in better security and dev. experience?
Which bugs precisely are you talking about?
Sane string implementation, so instead of performing some shenanigans with buffers to concat two strings, I can just "a" + "b"?
>If you had to do all the same things, you'd have the same bugs.
Why in lower level languages people cannot write some handy abstractions which will result in better security and dev. experience?
>Sane string implementation, so instead of performing some shenanigans with buffers to concat two strings, I can just "a" + "b"?
Because system language users care about what happens when you do that. Where is it being allocated? What happens to the original strings? And a lot of other questions that relate to memory management. These languages are faster in part because of the control you get over allocations.
In C++ one could use a vector<char> to build a temporary string (or use stringstream for a fancier interface), but that also means that whenever something is appended to it, it must check the container capacity to handle possible reallocations, etc. Very similar to how most high level languages handle strings. But that comes at a cost.
A common C pattern is to simply receive a pointer to a block of memory to use for the string, write to that and null terminate it. If the buffer size is insufficient, it will stop writing but continue parsing so it can return the size of the required buffer so the user can allocate that and call the function again. Most libraries avoid allocating stuff on behalf of the user. On the common case (buffer size is enough), it'll be much faster as no heap memory needs be allocated, moved around, etc.
Because system language users care about what happens when you do that. Where is it being allocated? What happens to the original strings? And a lot of other questions that relate to memory management. These languages are faster in part because of the control you get over allocations.
In C++ one could use a vector<char> to build a temporary string (or use stringstream for a fancier interface), but that also means that whenever something is appended to it, it must check the container capacity to handle possible reallocations, etc. Very similar to how most high level languages handle strings. But that comes at a cost.
A common C pattern is to simply receive a pointer to a block of memory to use for the string, write to that and null terminate it. If the buffer size is insufficient, it will stop writing but continue parsing so it can return the size of the required buffer so the user can allocate that and call the function again. Most libraries avoid allocating stuff on behalf of the user. On the common case (buffer size is enough), it'll be much faster as no heap memory needs be allocated, moved around, etc.
What is a string? A series of bytes? Characters? What locale is being used? What are you assigning it to? What's dealing with memory? What happens when you leave current scope? What if the two strings aren't the same type, or one or both of them contains "binary" (and what is binary)? What are you doing with the string? Does your language have a bunch of fancy features like calculating length, taking slices, copying? Are you going to read character by character, or use a regex? Are you going to implement a state machine, linked list, hash, sorting algorithm, objects/classes? For any of the functions you'll be passing this data to, will they ever expect some specific type, much less encoding, specific character set, or length? Do you need to use "safe" functions "safely"? What do you do about 3rd party libraries, external applications, network sockets? Are your operations thread-safe and concurrent?
Higher-level languages, being, you know, higher-level, have a barrage of subtly difficult and uninteresting crap taken care of for you. Sure, you could "write some handy abstractions which will result in better security and dev. experience". And you would end up with.... a high-level language. But it would be slow, fat, and there'd be a dozen things you just couldn't do. Kernels and crypto pretty much need to be low-level.
The real reason low-level languages don't get any easier to program in is standards bodies. A bunch of chuckleheads argue over the dumbest things and it takes two decades to get some marginally better feature built into the language, or some braindeadness removed. There's only so much that function abstractions can do before you lose the low-level benefits of speed, size and control. (and to be fair, good compilers aren't exactly easy to write)
Higher-level languages, being, you know, higher-level, have a barrage of subtly difficult and uninteresting crap taken care of for you. Sure, you could "write some handy abstractions which will result in better security and dev. experience". And you would end up with.... a high-level language. But it would be slow, fat, and there'd be a dozen things you just couldn't do. Kernels and crypto pretty much need to be low-level.
The real reason low-level languages don't get any easier to program in is standards bodies. A bunch of chuckleheads argue over the dumbest things and it takes two decades to get some marginally better feature built into the language, or some braindeadness removed. There's only so much that function abstractions can do before you lose the low-level benefits of speed, size and control. (and to be fair, good compilers aren't exactly easy to write)
As the handy abstraction would have a significant performance overhead, a low-level language would not want to use it everywhere or even as a default - especially in the third-party libraries you'll be using everywhere.
The handy abstraction is not a win-win, it's a tradeoff of better security and dev. experience versus performance and control - and a key point is that people who have explicitly chosen a low-level language likely have done so exactly because they want this tradeoff to be more towards performance or control. If someone really wants to get better security and dev. experience at the cost of performance, then why not just use a high-level language instead of combining the worst of both worlds where you have to do the work in a low-level language (even if partially mitigated by these handy abstractions) but still pay the performance overhead that a high-level language would have?
The handy abstraction is not a win-win, it's a tradeoff of better security and dev. experience versus performance and control - and a key point is that people who have explicitly chosen a low-level language likely have done so exactly because they want this tradeoff to be more towards performance or control. If someone really wants to get better security and dev. experience at the cost of performance, then why not just use a high-level language instead of combining the worst of both worlds where you have to do the work in a low-level language (even if partially mitigated by these handy abstractions) but still pay the performance overhead that a high-level language would have?
> Why in lower level languages people cannot write some handy abstractions which will result in better security and dev. experience?
Because those programs are sloooooooooooooooooooooooooooooow
Because those programs are sloooooooooooooooooooooooooooooow
parsers and serializers have one thing they often do: read (and write) to a (usually manually allocated) byte array. And the content of that byte array is often under attacker control.
C has terrible support for things dealing with byte arrays. They must be manually allocated, and accesses must be checked to be in-bound manually.
Lots of critical software have parsers written in C. This combination leads to CVEs like this one.
FWIW, a bug such as this one (which ends up with an invalid array access) could happen in any language, and would end up with a panic in Rust, or a NullPointerException in Java, etc... The thing that makes this especially dangerous is that, because C is low-level and unchecked, this can also lead to Remote Code Execution instead of a simple Denial Of Service/crash.
C has terrible support for things dealing with byte arrays. They must be manually allocated, and accesses must be checked to be in-bound manually.
Lots of critical software have parsers written in C. This combination leads to CVEs like this one.
FWIW, a bug such as this one (which ends up with an invalid array access) could happen in any language, and would end up with a panic in Rust, or a NullPointerException in Java, etc... The thing that makes this especially dangerous is that, because C is low-level and unchecked, this can also lead to Remote Code Execution instead of a simple Denial Of Service/crash.
> or a NullPointerException in Java
Just nitpicking, but the exception for an invalid array access in Java would be IndexOutOfBoundsException (or one of its subclasses), not NullPointerException.
Just nitpicking, but the exception for an invalid array access in Java would be IndexOutOfBoundsException (or one of its subclasses), not NullPointerException.
Parsing code without any kind of framework or high level helpers is, for lack of a better word, fiddly. C strings are an awful tool to do it because memory safety depends on getting a lot of buffer size / string length calculations right - there are plenty of opportunities to make mistakes. The fiddliness also induces developers to make changes in existing code in the form of local clever tricks instead of adapting the code to cleanly implement new requirements. It's only a small change and the tests (hopefully there are any) pass, job done! But maybe it violates a non-obvious assumption for another clever trick somewhere else, etc...
I think the biggest difference is bounds checking. If you write off the end of your array in Java, you get an exception. In C, you get a CVE. Other things like manual memory management and fiddly string types don't help, but simple bounds checks would catch so many things.
tl;dr: Using C/C++ and being human => memory safety problems.
Also, as we learned back when Heartbleed was discovered, the OpenSSL code is not in good shape. It "suffers from maintenance", as one clever wag said about legacy code. There's a reason LibreSSL forked the code. More distributions need to switch away from OpenSSL.
And before anyone pipes up, I'm not claiming LibreSSL does not and will not ever haver vulnerabilities. I'm saying that ripping stuff like punycode out of the library reduces the attack surface. https://isc.sans.edu/diary/rss/29208
And before anyone pipes up, I'm not claiming LibreSSL does not and will not ever haver vulnerabilities. I'm saying that ripping stuff like punycode out of the library reduces the attack surface. https://isc.sans.edu/diary/rss/29208
>Also, as we learned back when Heartbleed was discovered, the OpenSSL code is not in good shape. It "suffers from maintenance", as one clever wag said about legacy code. There's a reason LibreSSL forked the code. More distributions need to switch away from OpenSSL.
Anyone who's ever worked with the OpenSSL API or looked at its code can tell you that it's a steaming pile of crap. It's no surprise that this vulnerability was discovered. Honestly, OpenSSL should just be banned because it's so horrible, and there are better alternatives available.
Anyone who's ever worked with the OpenSSL API or looked at its code can tell you that it's a steaming pile of crap. It's no surprise that this vulnerability was discovered. Honestly, OpenSSL should just be banned because it's so horrible, and there are better alternatives available.
I just started making openssl -Werror safe. Oh my, what did I get into.
Halfway through it's about 125 changed files, > 1000 changes. look at the WIP commit. The API is insane. 50% of args are unused. All the structs and vtables updates are uninitialized, ie missing methods.
https://github.com/rurban/openssl/commits/Werror
Halfway through it's about 125 changed files, > 1000 changes. look at the WIP commit. The API is insane. 50% of args are unused. All the structs and vtables updates are uninitialized, ie missing methods.
https://github.com/rurban/openssl/commits/Werror
One of the (possibly first?) things the LibreSSL people did after forking OpenSSL was to enable -Wall, -Werror, -Wextra, -Wuninitialized on the code[1]. Many years ago we'd look at compiler (and linter) warnings with a skeptical eye, but these days, they really mean something. That alone smoked out a lot of lurking problems.
1 https://en.wikipedia.org/wiki/LibreSSL#Proactive_measures
1 https://en.wikipedia.org/wiki/LibreSSL#Proactive_measures
Ongoing related thread:
The latest OpenSSL vulns were added fairly recently - https://news.ycombinator.com/item?id=33437158 - Nov 2022 (56 comments)
The latest OpenSSL vulns were added fairly recently - https://news.ycombinator.com/item?id=33437158 - Nov 2022 (56 comments)
> As curl author Daniel Stenberg said, "I've never even considered to decode punycode. Why does something like OpenSSL need to decode this?"
> [...]
> Internationalization is not the issue, internationalization is the job.
I want to cheer at this.
Internationalization is hard, really hard, especially in a computing world so defined by its english-language dominance. The little amount of effort demonstrated in defining and testing this feature demonstrates that.
For all I admire Stenberg's work and focus on quality, that was a rather poor take from him.
> [...]
> Internationalization is not the issue, internationalization is the job.
I want to cheer at this.
Internationalization is hard, really hard, especially in a computing world so defined by its english-language dominance. The little amount of effort demonstrated in defining and testing this feature demonstrates that.
For all I admire Stenberg's work and focus on quality, that was a rather poor take from him.
Oh I don't think Daniel was asking why we're doing i18n. My remark was not at him, at all. I was just pre-empting a possible made up objection. He's in fact correct to wonder why punycode decoding ended up in OpenSSL, as the rest of that section explores. The point being that OpenSSL could do its job and still support i18n domains and emails without having to ever decode punycode if only the spec had made different tradeoffs.
Exactly this. I had the very image of this discussion with Python folks for DNSname SANs back when Python was learning not to just rely on CN.
Python folks felt like the correct thing was: 1) Implement a Punycode decoder. 2) Decode hostnames and certificate information to get Unicode 3) Compare the Unicode Strings. They were fretting about how complicated it would be to arrange all this, the months of work needed and the need to bring in teams who understood about i18n issues...
And I was like No, don't do any of that, take the bytes and compare the bytes. If the bytes aren't identical that's not a match, you are done. It seemed to take a while for it to sink in that this is correct and simpler and thus better.
Python folks felt like the correct thing was: 1) Implement a Punycode decoder. 2) Decode hostnames and certificate information to get Unicode 3) Compare the Unicode Strings. They were fretting about how complicated it would be to arrange all this, the months of work needed and the need to bring in teams who understood about i18n issues...
And I was like No, don't do any of that, take the bytes and compare the bytes. If the bytes aren't identical that's not a match, you are done. It seemed to take a while for it to sink in that this is correct and simpler and thus better.
Yes, it should be possible to compare labels for equality octet-wise. There should be no normalization considerations in this case because every A-label in a certificate should already be normalized.
ah. Thanks for the clarification!
[deleted]
I decided to review SBOMs from about 3,800 popular mobile apps to see if any included vulnerable versions of OpenSSL v3.0.x. No mobile apps did (not surprised) but what did surprise me was 98% of the OpenSSL versions included in these apps were vulnerable to older CVEs. About 16% of the apps included OpenSSL, mostly as a transitive dependency.
I posted additional details in this blog+video: https://www.andrewhoog.com/post/how-to-detect-openssl-v3-and...
I posted additional details in this blog+video: https://www.andrewhoog.com/post/how-to-detect-openssl-v3-and...
> No mobile apps did (not surprised) but what did surprise me was 98% of the OpenSSL versions included in these apps were vulnerable to older CVEs.
Just the openssl version is not enough, since it could be patched to fix vulnerabilities without increasing the version (this is very common on Linux distributions, which often apply security patches instead of migrating to a new version; for instance, Fedora released a patched 3.0.5 instead of going to 3.0.7).
And using an older openssl version does not necessarily mean it's using vulnerable code; according to your blog post, the most common use is SQLCipher, which from a quick look at its README.md seems to use openssl only for the encryption algorithms. Unless the vulnerability was on the basic algorithms used (AES, HMAC, etc), it won't affect this usage.
Just the openssl version is not enough, since it could be patched to fix vulnerabilities without increasing the version (this is very common on Linux distributions, which often apply security patches instead of migrating to a new version; for instance, Fedora released a patched 3.0.5 instead of going to 3.0.7).
And using an older openssl version does not necessarily mean it's using vulnerable code; according to your blog post, the most common use is SQLCipher, which from a quick look at its README.md seems to use openssl only for the encryption algorithms. Unless the vulnerability was on the basic algorithms used (AES, HMAC, etc), it won't affect this usage.
Great points.
Static binary analysis looks for the version string but doesn’t currently do deeper analysis of reversed code to see if it’s patched. Could go either way.
And determining if the code is triggered and exploitable is quite challenging. Dynamic analysis can help here, provided you have the coverage.
More generally tho, istm that there will be instances when the version is unpatched and there is some exploitable vector (even if it’s just crashing the app). My hope is to raise awareness for developers (and security) about 1) transitive dependencies and 2) some really old OpenSSL versions in very popular mobile apps. I don’t believe most folks think about this and awareness can lead to shipping safer apps.
Static binary analysis looks for the version string but doesn’t currently do deeper analysis of reversed code to see if it’s patched. Could go either way.
And determining if the code is triggered and exploitable is quite challenging. Dynamic analysis can help here, provided you have the coverage.
More generally tho, istm that there will be instances when the version is unpatched and there is some exploitable vector (even if it’s just crashing the app). My hope is to raise awareness for developers (and security) about 1) transitive dependencies and 2) some really old OpenSSL versions in very popular mobile apps. I don’t believe most folks think about this and awareness can lead to shipping safer apps.
Why would a TLS library even parse punycode? The sole reason we still use this hack is so that core infrastructure does not have to change when we use internationalized domains.
If TLS libraries, DNS servers, HTTP servers, ... needed to be patched anyways for the use of IDNs, then why did we not do it properly and just use UTF-8?
No matter how many vulnerabilities we've introduced into software, "Š" in my name still cannot be represented in domain names and is instead written with american letters xn--pga.
I'm sorry for poorly articulating my thoughts here.
If TLS libraries, DNS servers, HTTP servers, ... needed to be patched anyways for the use of IDNs, then why did we not do it properly and just use UTF-8?
No matter how many vulnerabilities we've introduced into software, "Š" in my name still cannot be represented in domain names and is instead written with american letters xn--pga.
I'm sorry for poorly articulating my thoughts here.
The idea is that back when IDNA was created sending just-UTF-8 in places that had historically been ASCII-only would break software. The scale of the breakage was probably unknown. The real question is: would a flag day have been needed to just use UTF-8 where previously only ASCII was allowed. I think the answer to that is "no". The punycode thing certainly avoided the question altogether, which is why it was done, but there's no question that had we bitten the bullet then and just gone with just-use-UTF-8 life today would be easier.
(There's also Unicode normalization issues here. One of the problems with just-use-UTF-8 is that dumb servers would only match on octet-wise label equality, but with Unicode you want normalization-insensitive matching. Now, at the time the notion of normalization-insensitive matching hadn't been invented... Requiring something like IDNA made it possible to slip in a normalization requirement.)
(There's also Unicode normalization issues here. One of the problems with just-use-UTF-8 is that dumb servers would only match on octet-wise label equality, but with Unicode you want normalization-insensitive matching. Now, at the time the notion of normalization-insensitive matching hadn't been invented... Requiring something like IDNA made it possible to slip in a normalization requirement.)
> Why was this code even necessary
> The answer is: an explicit IETF design choice, that made punycode decoding part of X.509 verification, without even a line of acknowledgement in the Security Considerations.
> The answer is: an explicit IETF design choice, that made punycode decoding part of X.509 verification, without even a line of acknowledgement in the Security Considerations.
About the classification: I would add that the people choosing the severity are a bit alone because of secrecy, so they also can’t really ask too much for advice.
That’s probably where the NSA could be useful because of their big number of competent and sworn to secrecy employees, but nobody can trust that the zero day sent for an opinion will not be used to fuck with a foreign country on day one.
That’s probably where the NSA could be useful because of their big number of competent and sworn to secrecy employees, but nobody can trust that the zero day sent for an opinion will not be used to fuck with a foreign country on day one.
[deleted]
So, punycode I do think was silly. We should just have used UTF-8 in DNS and have left it at that.
Using UTF-8 would not have required a flag day. It would have required upgrading some servers in order to be able to have non-ASCII domainnames, but it wouldn't have broken anyone not using non-ASCII domainnames.
Using UTF-8 would not have required a flag day. It would have required upgrading some servers in order to be able to have non-ASCII domainnames, but it wouldn't have broken anyone not using non-ASCII domainnames.
So it would have broken the whole internet for anyone using a UTF-8 domain name and this system would be adopted by no one. It’s like IPv6, using IPv6 doesn’t break anything for anyone not using it. It’s just that if you use ipv6, you can’t talk to half of the internet including this site.
It wouldn't have broken the whole internet though. It's hyperbolic to suggest that it would have.
Yes, it would, because at the start no one would be supporting it so no one would be able to connect. So it would have broken the entire internets ability to connect to you. And remember, as a user, you don’t just use one DNS server that you need to upgrade, you use multiple and they all need to support it and a typical user is not in a position to upgrade it.
If you don’t believe that, believe this: if it were as simple as you claim, just add in UTF8, the world wouldn’t have settled on punycode. They didn’t, because it is not actually that simple. It’s so hard, it’s actually simpler to settle for punycode.
If you don’t believe that, believe this: if it were as simple as you claim, just add in UTF8, the world wouldn’t have settled on punycode. They didn’t, because it is not actually that simple. It’s so hard, it’s actually simpler to settle for punycode.
[deleted]
> I did not have the heart to figure out why it works like this. Maybe there's a good reason to do progressive parsing. Maybe it's an artifact of how C makes you do memory management or of OpenSSL's C style. Anyway.
Not involved in OpenSSL, but this is a fairly common pattern in a lot of C APIs. You want to decode some data, but you're not sure how big the output is going to be ahead of time. You could write a separate function to calculate the length, but that function has to do most of the work of actual decoding to figure out that length. A lot of times the output is small enough it can fit in some conservatively sized buffer so you can save a fair bit of work by having a (potentially stack-allocated) buffer of some fixed size and then allocating a precisely sized buffer on the heap if it turns out to not be big enough. Further, having a separate length function means you typically end up with two similar but separate decode implementations which has its own problems.
In most other languages, you have some sort of growable container in the standard library so you just avoid the problem entirely at the expense of having less control over memory allocations.