If a NUL parameter is passed to utf8toutf16 and friends, they'll return UTF8_ERR_INVALID_DATA.
However, you are correct about codepoint_read's behavior. I actually have a test for passing NUL to the function, but the value of decoded isn't checked:
TEST(CodepointRead, InvalidData)
{
const char* i = nullptr;
size_t il = 0;
unicode_t o;
EXPECT_EQ(0, codepoint_read(i, il, &o));
}
It's definitely a bug, but relatively harmless. The function is only used internally and the calling sites guard against that kind of attack.
Nevertheless, I thank you for the extra set of eyeballs. I have been looking at these functions far too long, you tend to gloss over obvious problems after a while.
I'll be sure to get a fix in for the next release. :)
You can find all case folding codepoints on the Unicode Consortium website: ftp://ftp.unicode.org/Public/UNIDATA/CaseFolding.txt and ftp://ftp.unicode.org/Public/UNIDATA/SpecialCasing.txt.
As you get down the list you'll notice what a pain in the ass the special cases are. There's a special case for the final sigma in a Greek word:
03A3; 03C2; 03A3; 03A3; Final_Sigma; # GREEK CAPITAL LETTER SIGMA
You must remove the dot from "i" when upper or titlecasing... but only in Lithuanian:
No, currently Turkish, Greek, Lithuanian and Azeri case mappings are not always grammatically correct. This is because utf8rewind currently does not take the system locale into account when case mapping. Fixing these issues is planned for a future release.
The library currently doesn't employ any kind of fuzzing. I rely on multiple tests for every kind of input. I've found that is a pretty reliable way to test for "unknown unknowns", even if it's extremely time-consuming.
Adding testcase fuzzing is definitely something to consider though, because it would most likely have found the very issues this release fixes.
If a NUL parameter is passed to utf8toutf16 and friends, they'll return UTF8_ERR_INVALID_DATA.
However, you are correct about codepoint_read's behavior. I actually have a test for passing NUL to the function, but the value of decoded isn't checked:
It's definitely a bug, but relatively harmless. The function is only used internally and the calling sites guard against that kind of attack.
Nevertheless, I thank you for the extra set of eyeballs. I have been looking at these functions far too long, you tend to gloss over obvious problems after a while.
I'll be sure to get a fix in for the next release. :)