Having studied this for many hours now... and also having read part of that dissertation... and knowing something about block ciphers...
I don't think it is probable that any peer will want to know the order of the "ping" replies. A heartbeat is not used to diagnose anything. Any heartbeat response will suffice to keep the connection alive.
The need for a payload/sequence number is presented as a "truth" of networking but it seems to be invalid here.
A 32 bit sequence number would suffice, except for the fact that DTLS also uses this packet for path MTU discovery. Strangely, an IP packet cannot exceed the ethernet MTU of 1500 bytes anyway and at present I have no idea how this DTLS MTU discovery is supposed to work. Supposedly, the only way to check for MTU limits on routers is to set a bit in the IP header that says "do not fragment" and then wait for possible ICMP replies. This should work for UDP as well. But perhaps you can also discover UDP fragmentation at the receiving end since UDP might not automatically recombine the fragments. Then, any protocol on top of UDP would be able to do this as well. It was introduced in DTLS so that any applications using DTLS wouldn't need to do it themselves.
Whatever the case may be, and however useful or stupid the design, these messages could not exceed 1500 bytes. So why 64k? Probably just because a byte is max 255 which is too low, a word (2 bytes) is max 65535 which is enough. I don't know what features UDP/DTLS have for reassembling packets, but apparently they thought it easier to make $FFFF the max value than to impose a better limit on this number.
Also the padding is required just in case some peer sends simple sequence numbers as payload. A 32-bit sequence number + 3 bytes of header would be 7 bytes. Adding a 16 byte padding of random data would scramble the plaintext so the plaintext of the encryption (all of this is encrypted) could not be guessed.
In the unlikely case that some implementation uses 13 bytes of payload with 16 bytes of padding, and the block cipher uses 128-bit blocks, which is 16 bytes, then this whole security measure is voided since the block cipher simply encrypts bocks of 128 bits in serial.
All in all it doesn't sound like it's very well thought out.
Can you explain why it is neccesary or functional that "one side of the protocol can send large messages, and the counterparty can reply with smaller ones, relying on the fact that the message includes discardable padding." ??
If the payload has variable size to facilitate MTU discovery, then there is some logic to sending messages of varying sizes. An MTU can never exceed the 1500 of ethernet though, so this is weird stuff. What good can a payload of 64k really do? And MTU discovery doesn't involve any echoing by the receiving party. Either packets get blocked (at the IP layer) and error messages get returned (by the router blocking them) OR the packages arrive fully (or get reassembled) so this Heartbeat code won't know about any issues anyway. And the "echo" proves nothing. I really don't get this, what is the point...?
It is wholly inappropriate to start blaming these authors, or to even suggest they are malicious (which is even worse).
They are the ones the technical computer community that operates at this level of network implementation have "chosen" to do this work (or they've chosen themselves, which is practically the same thing) which is an honour to them.
If anything goes wrong here, it is that culture at large, the way the RFCs get debated and approved, etcetera. There is no one to blame here in isolation.
Although I will agree with you that it is remarkable that this person forgot to write the check on payload_length after stating that so explicitly in the document.
I have always detested C (also C++) because it's so unreadable... the snippets of code you cite are just so dense ie. a function like n2s() gives pretty much no indication of what it does to a casual reader. Just reading the RFC (it is pretty much written in a C style) gives me the creeps.
The RFC doesn't mention why there has to be a payload, why the payload has to be random size, why they are doing an echo of this payload, why there has to be a padding after the payload. If this data is just a regular C struct like the RFC makes it out to be (I didn't know you could have a struct with a variable size, but apparently the fields are really pointers or it's just a mental model and not a real struct).
Apparently the purpose of the payload is path MTU discovery. Something that is supposed to happen at the IP layer, but I don't know enough about datagram packets. I guess an application may want to know about the MTU as well...
I'm not here to point fingers, I'm just saying C is a nightmare to me and a reason for me to never be involved with system programming or something like drafting RFC's ;-).
But if one can argue that C is a bad choice for writing this stuff, then that is not an isolated thing. "C" is also the language of the RFCs. "C" is also the mindset of the people doing that writing. After all, the language you speak determines how you think. It introduces concepts that become part of your mental models. I could give many examples, but that's not really the point.
And it's about style and what you give attention to. To me, that RFC is a real bad document. It starts to explain requirements to exceptional scenario's (like when the payload is too big) before even having introduced and explained the main concepts and the how and why's.
So while you may argue that this is a C problem and not a protocol problem, it is really all related.
And you may also say, in response to someone blaming these coders, that blame is inappropriate (and it is) because these are volunteers and they are donating their free time to something to find valuable, the whole distribution and burden of responsibility is, naturally, also part of the culture and how people self-organize and so on.
As someone else explained (https://news.ycombinator.com/item?id=7558394) the protocol is real bad but it is the result of more or less political limitations around submitting RFCs for approval. There is no reason for the payload in TLS (but apparently there is in DTLS) but my point is simply this:
If you are doing inelegant design this will spill over into inelegant implementation. And you're bound to end up with flaws.
Rather than trying to isolate the fault here or there, I would say this is a much larger cultural thing to become aware of.
I don't think it is probable that any peer will want to know the order of the "ping" replies. A heartbeat is not used to diagnose anything. Any heartbeat response will suffice to keep the connection alive.
The need for a payload/sequence number is presented as a "truth" of networking but it seems to be invalid here.
A 32 bit sequence number would suffice, except for the fact that DTLS also uses this packet for path MTU discovery. Strangely, an IP packet cannot exceed the ethernet MTU of 1500 bytes anyway and at present I have no idea how this DTLS MTU discovery is supposed to work. Supposedly, the only way to check for MTU limits on routers is to set a bit in the IP header that says "do not fragment" and then wait for possible ICMP replies. This should work for UDP as well. But perhaps you can also discover UDP fragmentation at the receiving end since UDP might not automatically recombine the fragments. Then, any protocol on top of UDP would be able to do this as well. It was introduced in DTLS so that any applications using DTLS wouldn't need to do it themselves.
Whatever the case may be, and however useful or stupid the design, these messages could not exceed 1500 bytes. So why 64k? Probably just because a byte is max 255 which is too low, a word (2 bytes) is max 65535 which is enough. I don't know what features UDP/DTLS have for reassembling packets, but apparently they thought it easier to make $FFFF the max value than to impose a better limit on this number.
Also the padding is required just in case some peer sends simple sequence numbers as payload. A 32-bit sequence number + 3 bytes of header would be 7 bytes. Adding a 16 byte padding of random data would scramble the plaintext so the plaintext of the encryption (all of this is encrypted) could not be guessed.
In the unlikely case that some implementation uses 13 bytes of payload with 16 bytes of padding, and the block cipher uses 128-bit blocks, which is 16 bytes, then this whole security measure is voided since the block cipher simply encrypts bocks of 128 bits in serial.
All in all it doesn't sound like it's very well thought out.