What can’t the internet handle in 2022? Apostrophes(wsj.com)
wsj.com
What can’t the internet handle in 2022? Apostrophes
https://www.wsj.com/articles/internet-mangles-names-accents-web-forms-11664462695
19 comments
https://archive.ph/K4gV2
Once again, programmers forget alphabets other than the Latin one exist. And by Latin, I mean the one containing only A through Z, with no “fancies”.
Then there’s the issue of character encodings, which the article does a decent job explaining! Nitpick though: they claims it’s a UTF-8/ASCII thing, but it’s actually a UTF-8/Windows-1252 issue.
In related news, even GNU coreutils fails to support UTF-8 properly despite a claim of support for multibyte character sets: https://catgirl.ai/log/cut-c-harmful/
Then there’s the issue of character encodings, which the article does a decent job explaining! Nitpick though: they claims it’s a UTF-8/ASCII thing, but it’s actually a UTF-8/Windows-1252 issue.
In related news, even GNU coreutils fails to support UTF-8 properly despite a claim of support for multibyte character sets: https://catgirl.ai/log/cut-c-harmful/
Most of us realize that other encodings and alphabets exist, it's just that supporting them is a nightmare. Do you have any idea how many un-typeable, invisible Unicode control characters there are? Did you know you can display the same Unicode text using different byte combinations?
How on Earth are devs supposed to sanitize this??
How on Earth are devs supposed to sanitize this??
> How on Earth are devs supposed to sanitize this??
By using the widely-available free tools which are near universal by now? This hasn't been technically _hard_ since around the turn of the century[1] but developers, especially Americans, have been lazy and tried to avoid doing the work needed to get beyond C-style string handling or, in the case of the apostrophes, using parameterized queries.
1. Having build a site which supported English, both simplified and traditional Chinese, Thai, and Vietnamese (including what passed for webfonts back then) in the 90s, I know how bad it used to be but also that was 3 decades ago.
By using the widely-available free tools which are near universal by now? This hasn't been technically _hard_ since around the turn of the century[1] but developers, especially Americans, have been lazy and tried to avoid doing the work needed to get beyond C-style string handling or, in the case of the apostrophes, using parameterized queries.
1. Having build a site which supported English, both simplified and traditional Chinese, Thai, and Vietnamese (including what passed for webfonts back then) in the 90s, I know how bad it used to be but also that was 3 decades ago.
Unicode themselves provide algorithms on handling them. There are libraries that implement these algorithms. Use them; Don't roll your own.
The libraries are not good. Support for the libraries in runtimes varies wildly and has drastic performance implications: unicode has variable width characters, so sometimes character array indexes are no longer O(1)!
Unicode is not a good system. Yes, we can struggle with it but it will always be a struggle because of the nature of its design.
A major omission is a way to indicate the encoding of a particular byte array. You open a file on disk and read its characters -- what encoding is it in? What encoding is used for the various buffers embedded in dozens of historic file formats for audio, video, archives, etc?
Developers are forced to guess and hack. The results are about as expected, given the quality of the underlying interfaces.
Unicode is not a good system. Yes, we can struggle with it but it will always be a struggle because of the nature of its design.
A major omission is a way to indicate the encoding of a particular byte array. You open a file on disk and read its characters -- what encoding is it in? What encoding is used for the various buffers embedded in dozens of historic file formats for audio, video, archives, etc?
Developers are forced to guess and hack. The results are about as expected, given the quality of the underlying interfaces.
Why would you index into a "character array"?
Also, what is a "character" to you?
Also, what is a "character" to you?
> Why would you index into a "character array"?
Obviously, because you need to examine characters. Here are a couple examples:
* You're implementing any software that deals with rfc6531 compliant email. You'll need to parse strings as unicode to understand where to route mail, for example to understand 你好@example.com
* You're implementing a database, or any software with a search function which indexes unicode fields. You'll have to understand the logical character structure of the data, or your searches won't match properly.
> Also, what is a "character" to you?
Characters are defined by the unicode specification. In UTF8 they can be represented by multi-byte sequences. Using the example of 你好@example.com, the @ is the seventh byte position, but the third character position. You cannot implement software that handles internationalized emails without parsing and understanding strings like this.
Obviously, because you need to examine characters. Here are a couple examples:
* You're implementing any software that deals with rfc6531 compliant email. You'll need to parse strings as unicode to understand where to route mail, for example to understand 你好@example.com
* You're implementing a database, or any software with a search function which indexes unicode fields. You'll have to understand the logical character structure of the data, or your searches won't match properly.
> Also, what is a "character" to you?
Characters are defined by the unicode specification. In UTF8 they can be represented by multi-byte sequences. Using the example of 你好@example.com, the @ is the seventh byte position, but the third character position. You cannot implement software that handles internationalized emails without parsing and understanding strings like this.
That's not how you'd parse email. In fact, the bit before @ is encoded differently from the bit after @.
You're much more likely to iterate over elements in a string than index. You might iterate over code points, abstract characters, glyphs, grapheme clusters, etc. You might do it before or after a certain kind of normalisation. Since there are many ways to split a string (depending on what you're doing), it won't be possible to make all of them have O(1) indexing.
You're much more likely to iterate over elements in a string than index. You might iterate over code points, abstract characters, glyphs, grapheme clusters, etc. You might do it before or after a certain kind of normalisation. Since there are many ways to split a string (depending on what you're doing), it won't be possible to make all of them have O(1) indexing.
What do you mean "that's not how you'd parse email?" I didn't describe an algorithm.
Why don't you explain how you would parse the above email address without understanding the unicode encoding?
Why don't you explain how you would parse the above email address without understanding the unicode encoding?
If you're asking how to get everything after the '@' symbol, you can use knowledge of UTF-8 (assuming your byte array is UTF-8) to just say `FindIndexOfByte(utf8Buffer, (byte)'@') + 1`. But treat everything else as an opaque blob. The URL portion of that email address, as far as you should be concerned, it just the bytes after the '@' symbol. If you want to verify if it's a valid URL, you should use a URL library, not some hack that assumes a TLD is 3+ characters. However, even that naive method of a byte search is wrong. According to RFC 2822, '@' symbol is actually allowed on the left hand side if escaped. You should be using an email parsing library that implements the spec properly.
Naming the smallest unit of information in C a 'char' was a terrible mistake made decades ago. A character has never been a single byte in many countries. Shift-JIS, for example, can be one or two bytes, and it's two and a half decades old. Even in UTF-16 (used by Java and C#), a 'char' isn't actually a character, but a codepoint. One that could even be an unpaired surrogate.
Really, you should never be indexing into the backing byte/char array unless you know what you're doing. If you think you need to, you should be asking yourself, "what is it I really need to do?" Because you, as a programmer dealing with internationalization, must not assume English/Western ideas about language. For example, in abugidas[0], a single word can be a whole sentence. For a more common language, Japanese typically doesn't use spaces because they're not needed when writing kanji. But that's not foolproof because hiragana and katakana aren't for words, but sounds. How do you do a word count in those languages? You don't.
[0]: https://en.wikipedia.org/wiki/Abugida
Naming the smallest unit of information in C a 'char' was a terrible mistake made decades ago. A character has never been a single byte in many countries. Shift-JIS, for example, can be one or two bytes, and it's two and a half decades old. Even in UTF-16 (used by Java and C#), a 'char' isn't actually a character, but a codepoint. One that could even be an unpaired surrogate.
Really, you should never be indexing into the backing byte/char array unless you know what you're doing. If you think you need to, you should be asking yourself, "what is it I really need to do?" Because you, as a programmer dealing with internationalization, must not assume English/Western ideas about language. For example, in abugidas[0], a single word can be a whole sentence. For a more common language, Japanese typically doesn't use spaces because they're not needed when writing kanji. But that's not foolproof because hiragana and katakana aren't for words, but sounds. How do you do a word count in those languages? You don't.
[0]: https://en.wikipedia.org/wiki/Abugida
> You should be using an email parsing library that implements the spec properly.
> you should use a URL library, not some hack
I think what you're somehow missing is that I am specifically offering authoring an email parsing library (or URLs, or whatever) as an example of where things break down.
Your answer is essentially "let someone else program it, because it's complicated and fraught with peril" -- but my entire point is that it's complicated and fraught with peril.
You are agreeing with me, and your post is a fine example of why the interfaces are bad and error prone.
> you should use a URL library, not some hack
I think what you're somehow missing is that I am specifically offering authoring an email parsing library (or URLs, or whatever) as an example of where things break down.
Your answer is essentially "let someone else program it, because it's complicated and fraught with peril" -- but my entire point is that it's complicated and fraught with peril.
You are agreeing with me, and your post is a fine example of why the interfaces are bad and error prone.
> Your answer is essentially "let someone else program it, because it's complicated and fraught with peril" -- but my entire point is that it's complicated and fraught with peril.
Are you asking how one would write code for the libraries you should use? In general, there already is one: ICU.[0] But if you had to write your own, as I said above, Unicode provides algorithms[1] that one might want to implement, in addition to a massive (over 1000 page) spec[2] detailing how to handle the various languages. For example, if you need to write the code to handle bidirectional text, there's UAX/TR #9[3]. Want to implement case changing? See Chapter 3.[4]
There's also the Unicode Character Database[5] that contains information on every code point. Your library/program would need to hold a copy of this whole thing.
[0]: https://icu.unicode.org/
[1]: https://unicode.org/reports/
[2]: https://www.unicode.org/versions/Unicode15.0.0/
[3]: https://unicode.org/reports/tr9/
[4]: https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf
[5]: https://www.unicode.org/reports/tr44/tr44-30.html
Are you asking how one would write code for the libraries you should use? In general, there already is one: ICU.[0] But if you had to write your own, as I said above, Unicode provides algorithms[1] that one might want to implement, in addition to a massive (over 1000 page) spec[2] detailing how to handle the various languages. For example, if you need to write the code to handle bidirectional text, there's UAX/TR #9[3]. Want to implement case changing? See Chapter 3.[4]
There's also the Unicode Character Database[5] that contains information on every code point. Your library/program would need to hold a copy of this whole thing.
[0]: https://icu.unicode.org/
[1]: https://unicode.org/reports/
[2]: https://www.unicode.org/versions/Unicode15.0.0/
[3]: https://unicode.org/reports/tr9/
[4]: https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf
[5]: https://www.unicode.org/reports/tr44/tr44-30.html
No. Let's do a quick recap:
You said that people ought never address text by characters or bytes.
I'm pointing out that this is nonsense and I offered several examples.
You responded saying basically that those tasks are complex and someone else should do it.
In your followup response, you recommend using ICU which ... has an interface for iterating by character. Which is all I had mentioned to begin with. Sigh.
You said that people ought never address text by characters or bytes.
I'm pointing out that this is nonsense and I offered several examples.
You responded saying basically that those tasks are complex and someone else should do it.
In your followup response, you recommend using ICU which ... has an interface for iterating by character. Which is all I had mentioned to begin with. Sigh.
Why the hell do you need to route the email unless you’re working on an email server? Send the email and have the user validate it. If they validate it, it’s a valid email. Don’t be clever, be smart.
Even if you’re working on an email server, you shouldn’t be touching that code unless it’s broken or you’re writing an implementation from scratch. In that case, treat it as an opaque blob.
Even if you’re working on an email server, you shouldn’t be touching that code unless it’s broken or you’re writing an implementation from scratch. In that case, treat it as an opaque blob.
> unless you’re working on an email server?
We are talking about email software, inclusive of servers, clients, plugins, etc. It's literally the example I'm offering (but there are neigh infinite others)
> Send the email and have the user validate it. I
But I haven't proposed a problem of validating an email. Besides, even if the problem is validating an email this is not necessarily a workable answer.
> Even if you’re working on an email server, you shouldn’t be touching that code unless it’s broken
A lot of the arguments supporting unicode seem to boil down to acknowledging that all these things are broken, yet disregarding them because the problems exist at a more technical level than most of HN commenters are familiar with.
You might not write this kind of software for a living, but some of us do and we know that these interfaces are not good.
We are talking about email software, inclusive of servers, clients, plugins, etc. It's literally the example I'm offering (but there are neigh infinite others)
> Send the email and have the user validate it. I
But I haven't proposed a problem of validating an email. Besides, even if the problem is validating an email this is not necessarily a workable answer.
> Even if you’re working on an email server, you shouldn’t be touching that code unless it’s broken
A lot of the arguments supporting unicode seem to boil down to acknowledging that all these things are broken, yet disregarding them because the problems exist at a more technical level than most of HN commenters are familiar with.
You might not write this kind of software for a living, but some of us do and we know that these interfaces are not good.
> Characters are defined by the unicode specification.
Unicode's glossary offers four definitions for this word, which ought to be a strong hint that this is not a technical term but instead a vague idea people have which doesn't align very well to the technical problem.
Unicode's glossary offers four definitions for this word, which ought to be a strong hint that this is not a technical term but instead a vague idea people have which doesn't align very well to the technical problem.
As I said initially: The interface (including the underlying unicode specification) is not good.
This is precisely my point. Internationalization is building on a foundation of sand.
This is precisely my point. Internationalization is building on a foundation of sand.
> what encoding is it in?
UTF-8. Unless you've got clear directions that it needs to be something else, the correct choice is always UTF-8. Stop emitting things in other encodings, "Doctor it hurts when I stab myself with this knife". Yes it does, so don't.
UTF-8. Unless you've got clear directions that it needs to be something else, the correct choice is always UTF-8. Stop emitting things in other encodings, "Doctor it hurts when I stab myself with this knife". Yes it does, so don't.
That's a great guess, but my point is that it's a guess. It will sometimes be wrong, and when the guess is wrong you will have articles like the above written complaining about the fallout.
These types of errors are guaranteed specifically because of the policy of guessing and loose heuristics you're describing.
These types of errors are guaranteed specifically because of the policy of guessing and loose heuristics you're describing.
Unicode fixes it. It's why legacy encodings are unworkable.
Unicode absolutely does not fix the issue of not knowing the encoding of a legacy byte buffer with text in it.
> unicode has variable width characters, so sometimes character array indexes are no longer O(1)!
> Unicode is not a good system.
Unicode is excellent. Your gripe is with human languages and their scripts. It wasn't the Unicode consortium who decided that some scripts have oddities like single glyphs that can encode anything between a single sound and a whole sentence, or oddities like different 'cases' where glyphs change form just because they happen to be at the beginning of a sentence or proper noun
> Unicode is not a good system.
Unicode is excellent. Your gripe is with human languages and their scripts. It wasn't the Unicode consortium who decided that some scripts have oddities like single glyphs that can encode anything between a single sound and a whole sentence, or oddities like different 'cases' where glyphs change form just because they happen to be at the beginning of a sentence or proper noun
I definitely agree. The article even mentions someone with a diaeresis in their name, which is distinct from an umlaut. Good luck sorting that mess out.
That said, you could clearly whitelist a much more reasonable range of characters than just the Latin alphabet. Hypens, apostrophes, common accented letters etc. should probably all be allowed. There are standard algorithms for canonicalising the unicode representation of them.
That said, you could clearly whitelist a much more reasonable range of characters than just the Latin alphabet. Hypens, apostrophes, common accented letters etc. should probably all be allowed. There are standard algorithms for canonicalising the unicode representation of them.
This comment is taking on flak, but Wirth &co took a similar stance, even as Zürchers.
>Did you know you can display the same Unicode text using different byte combinations?
Even ASCII has upper and lower case letters. Languages are hard.
Even ASCII has upper and lower case letters. Languages are hard.
It's inert text, query parameters will sanitize it for you.
It's only inert if you're not transforming it (for example, calling toLower on all strings passed in) or doing equality checks. Many applications use these and it is a minefield in Unicode.
But what's the semantic purpose of these operations? Linguistically these are very strange things to do to arbitrary text, and yet because in ASCII it was easy, programmers got into the habit of just doing it all over the place without thinking about what (if anything) it meant.
For transform operations like "toLower" often it turns out that to the extent it means anything it's only relevant to ASCII and so an operation which only transforms ASCII is appropriate anyway.
For transform operations like "toLower" often it turns out that to the extent it means anything it's only relevant to ASCII and so an operation which only transforms ASCII is appropriate anyway.
> But what's the semantic purpose of these operations?
The list is infinite, but let's use a very simple one which is the focus of this article: Names of things.
You name a file on a case insensitive filesystem. Should the case insensitivity only apply to certain encodings, or all of them? You'll need to do a case transformation and string comparison to make this work.
You create an email address, per rfc6531. You'll need to deal with case flattening, string comparisons, and lexical parsing.
These are not strange operations. These are not only common they are obviously required operations, because names identify things and names (eg emails) are often structured and must be parsed and understood.
Can you imagine the fallout if software developers said "sorry, you can only search for ASCII names because we think indexing unicode strings is a very strange thing to do?"
The list is infinite, but let's use a very simple one which is the focus of this article: Names of things.
You name a file on a case insensitive filesystem. Should the case insensitivity only apply to certain encodings, or all of them? You'll need to do a case transformation and string comparison to make this work.
You create an email address, per rfc6531. You'll need to deal with case flattening, string comparisons, and lexical parsing.
These are not strange operations. These are not only common they are obviously required operations, because names identify things and names (eg emails) are often structured and must be parsed and understood.
Can you imagine the fallout if software developers said "sorry, you can only search for ASCII names because we think indexing unicode strings is a very strange thing to do?"
> You'll need to do a case transformation and string comparison to make this work.
Better to say you need a canonicalisation table and you will need to canonicalise the text. NTFS filesystems actually carry their canonicalisation table with them, so if you happen to be using a USB drive with an NTFS filesystem where Z and Q are just different versions of the same hypothetical "Q-Z" character, filenames ZZ150 and QQ150 are indistinguishable on that USB drive, even though your Windows PC has no trouble distinguishing Z from Q.
Notice that unlike "case transformation" this doesn't pretend we're doing something with larger semantic value, and indeed the actual names on disk aren't promised to be Unicode text under NTFS, they're just arbitrary 16-bit values which probably make sense as UCS-2 or these days UTF-16 when rendered, the tables just convert some values into others.
Better to say you need a canonicalisation table and you will need to canonicalise the text. NTFS filesystems actually carry their canonicalisation table with them, so if you happen to be using a USB drive with an NTFS filesystem where Z and Q are just different versions of the same hypothetical "Q-Z" character, filenames ZZ150 and QQ150 are indistinguishable on that USB drive, even though your Windows PC has no trouble distinguishing Z from Q.
Notice that unlike "case transformation" this doesn't pretend we're doing something with larger semantic value, and indeed the actual names on disk aren't promised to be Unicode text under NTFS, they're just arbitrary 16-bit values which probably make sense as UCS-2 or these days UTF-16 when rendered, the tables just convert some values into others.
Do you understand that filesystem drivers are yet another example of places where software developers need to do these types of operations which you're dismissing as "very strange things?"
How do you propose that I, a person who among other things develops filesystems, implement a canonicalization table without doing these "very strange things?"
Furthermore, filesystem operations are not limited to calls into the filesystem driver. Programs often need to deal with paths outside of filesystem calls: The names of files need to be displayed on screens. They need to be indexed in databases and searched for. They need to be encapsulated within other protocols and data formats and often need to be parsed and understood.
You are essentially agreeing with me that these are normal, necessary things. Your position appears to be "someone else does that type of thing for me," which is not a very compelling argument.
How do you propose that I, a person who among other things develops filesystems, implement a canonicalization table without doing these "very strange things?"
Furthermore, filesystem operations are not limited to calls into the filesystem driver. Programs often need to deal with paths outside of filesystem calls: The names of files need to be displayed on screens. They need to be indexed in databases and searched for. They need to be encapsulated within other protocols and data formats and often need to be parsed and understood.
You are essentially agreeing with me that these are normal, necessary things. Your position appears to be "someone else does that type of thing for me," which is not a very compelling argument.
Case insensitive file systems might as well be a historic accident, because lower case letters didn't always exist and then it was kept for backward compatibility.
Case insensitive functions are common. Filesystems are just one of many places where this kind of function is expected.
It's just a particularly common canonicalization function. Canonicalization is a common need across all languages as we've already clearly established.
It's just a particularly common canonicalization function. Canonicalization is a common need across all languages as we've already clearly established.
I think you shouldn't try to reproduce the filesystem name comparison algorithm. Check if file exists or generate numeric name.
I can't find where rfc6531 requires case flattening. The word "case" is used there in the meaning of "situation".
I can't find where rfc6531 requires case flattening. The word "case" is used there in the meaning of "situation".
Case sensitivity is addressed in other RFCs relating to email and DNS.
> Check if file exists or generate numeric name.
Nonsense. You may not be in control of the names, and checking a file with stat(2) may not address the problem at hand (and: is many orders of magnitude different in cost as well).
> Check if file exists or generate numeric name.
Nonsense. You may not be in control of the names, and checking a file with stat(2) may not address the problem at hand (and: is many orders of magnitude different in cost as well).
I imagine those applications have a better idea what sanitation they need. Is it even the same for all of them? And toLower is what they do now, but how do you predict the algorithm won't change in future? When you sanitize proactively for future algorithms, it's just data loss.
> but it’s actually a UTF-8/Windows-1252 issue.
Or possibly ISO-8859-1, which is, helpfully, slightly different.
Or possibly ISO-8859-1, which is, helpfully, slightly different.
True. But, in this case, I'm betting it's Windows-1252. The character in Ch'ien's name is most likely `U+2099 RIGHT SINGLE QUOTATION MARK`. In UTF-8, that's `E2 80 99`. ISO-8859-1 doesn't have 80 or 99 defined, but Windows-1252 would decode it to ’ (what's on the card).
I want URLs that handle spaces. Like I could enter www.coca cola.com. [and not www.coca-cola.com. nor www.cocacola.com.) and it would work. "Simple space". Even the Latin/Roman alpha had space(s).
The Romans wrotewithoutspaces as in "continuous script" [1], however they·did·use·the·interpunct for delineation on inscriptions. In order to have spaces in domain names we would need to use another sign, such as • [2], to separate words and avoid ambiguity.
[1] https://en.wikipedia.org/wiki/Scriptio_continua
[2] The bullet • (U+2022) alt code is 7, or Opt + 8 on Mac. The interpunct · (U+00B7) alt code is 250, or Shift + Opt + 9 on Mac. Rather inconvenient for the regular user.
[1] https://en.wikipedia.org/wiki/Scriptio_continua
[2] The bullet • (U+2022) alt code is 7, or Opt + 8 on Mac. The interpunct · (U+00B7) alt code is 250, or Shift + Opt + 9 on Mac. Rather inconvenient for the regular user.
Is that ever ambiguous in Latin, or is there enough variation in declensions and conjugations (and enough of them) that the word endings are always clear?
Quote from Wikipedia: 'For example, a phrase written in scriptio continua as collectamexiliopubem may be interpreted as collectam ex Ilio pubem, meaning "a people gathered from Troy", or collectam exilio pubem, "a people gathered for exile". Thus, readers had to be much more cognisant of the context to which the text referred.' As in everything, context rules over content.
www.google.com [a ton of spaces that HN won’t let me type] jknotgoogle.com
Domain registration already has a lot of rules around length, what characters can be where in the domain name (ex. no hyphens as the very first/last characters), ... I'm sure it'd be trivial to have a "no-repeating-spaces" rule or similar.
google.com [spaces] pwned.evilcorp.com it is
I just want a single contiguous space character so I don't have to suppress the space or substitute with an alternate character --also URL boxes can make it conspicuously obvious for better UI. And while they're at it, allow & (ampersand).
Maybe the location bar should accept the space but strip it? That would work for input if not reading the URL. Maybe only allowing single consecutive spaces to avoid the address spoofing example. I could go for that.
I would love & natively though. For my wedding (bride&groom.com for example) I looked into faking it with punycode but nothing was close enough, and anything other than an ampersand would have been too hard to input.
I would love & natively though. For my wedding (bride&groom.com for example) I looked into faking it with punycode but nothing was close enough, and anything other than an ampersand would have been too hard to input.
good think hyperlinks are typically underlined so there will be a large training line at the end
Unless someone turns off the underlining with a stylesheet.
Google search urls are typically long enough that they are shortened by ... At least on mobile
You can have spaces in URLs: https://matthewminer.name/spaced%20name
You just can't have them in domain names.
You just can't have them in domain names.
Sure, I meant the whole URL, including the domain portion as in my example.
Space is perfectly legal in URLs, but not in a domain name so your example won’t work for that reason.
> Like I could enter www.coca cola.com. [and not www.coca-cola.com
Aside from all the comments about what's wrong with this on a technical level, this is a poor example since the product name is actually "Coca-Cola" and not "Coca Cola".
Aside from all the comments about what's wrong with this on a technical level, this is a poor example since the product name is actually "Coca-Cola" and not "Coca Cola".
It only takes a quick glance at sites like Stack Overflow to see that things like character encodings, the difference between the external representation of a string and its actual content, different quoting rules in different contexts (CSV versus XML versus JSON versus ...) is a complete mystery to many people trying to work as developers.
In print, code is often represented in a monospace font, materials intended for beginners sometimes indicate stuff that you are actually supposed to type with little keycap symbols, etc. Have there been any programming languages (with associated IDE:s) where strings were represented by typography (or colour, or whatever) rather than by surrounding them with some sort of quotation marks?
In print, code is often represented in a monospace font, materials intended for beginners sometimes indicate stuff that you are actually supposed to type with little keycap symbols, etc. Have there been any programming languages (with associated IDE:s) where strings were represented by typography (or colour, or whatever) rather than by surrounding them with some sort of quotation marks?
You can't just blame the devs here. Having worked with software that had to be able to render every language under the sun, I'm more familiar than I want to be with encodings, fonts, rendering etc, but language support in eg. Python is a train wreck and even getting "special" characters in and out of HTML can be a real minefield.
At some point we have to at least gently blame the overall developer community. We've had actual decades to figure this out. Joel's The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)[1] manifesto was written almost 20 years ago!
99% of programmers shouldn't be treating text as anything but opaque blobs of data that, when sent to a rendering function, become readable by users. Don't try to take text blobs apart and make assumptions about their binary structure. We should not be looking under the hood and trying to divine things like how many letters are in them (what's a letter?) or trying to manually transform them in some way (case, splitting, concatenation, input sanitization and so on). Use tried and true libraries for these. Same for when you need to transform it into some particular encoding. The only safe question to ask of text is "how much memory does this thing use?"
It is truly a minefield, but it's a minefield that developers are expected to understand and navigate. "Text in different encodings are scary outliers that we shouldn't support" is pee-wee league level software engineering.
1: https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
99% of programmers shouldn't be treating text as anything but opaque blobs of data that, when sent to a rendering function, become readable by users. Don't try to take text blobs apart and make assumptions about their binary structure. We should not be looking under the hood and trying to divine things like how many letters are in them (what's a letter?) or trying to manually transform them in some way (case, splitting, concatenation, input sanitization and so on). Use tried and true libraries for these. Same for when you need to transform it into some particular encoding. The only safe question to ask of text is "how much memory does this thing use?"
It is truly a minefield, but it's a minefield that developers are expected to understand and navigate. "Text in different encodings are scary outliers that we shouldn't support" is pee-wee league level software engineering.
1: https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
This is like if civil engineers were allowed to build bridges without any formal education.
I'm not even talking about a full 4-year course, the majority of stupid moves in programming can be taught away in a course that's a few weeks of learning, especially once you've had a bunch of experience already.
I'm not even talking about a full 4-year course, the majority of stupid moves in programming can be taught away in a course that's a few weeks of learning, especially once you've had a bunch of experience already.
Thankfully most code isn't mission critical and bugs aren't life threatening. The things that are seem to be held to a higher standard.
As someone who codes a fair bit at work without having a CS degree, is there any such course online you could recommend?
Like a coursera course titled "How not to be dumb at coding"out something of the sorts.
Like a coursera course titled "How not to be dumb at coding"out something of the sorts.
The set of search results for "falsehoods programmers believe about" could each be a single lesson in "How not to be dumb at coding 101"
https://duckduckgo.com/?q=falsehoods+programmers+believe+abo...
https://duckduckgo.com/?q=falsehoods+programmers+believe+abo...
I have an Irish O' surname, and I've been to too many conferences where it's been changed to O%apos; on my nametag, and in once case a registration failed because there is a second unicode apostrophe that some text editors use which means my name won't match what's in their database. Not to mention many sites consider it an illegal character and drop the apostrophe all together.
It's infuriating, I've seriously considered changing my name because of it.
It's infuriating, I've seriously considered changing my name because of it.
Yep... post arrives to "O'Rourke", websites tell me "your name is invalid", checkouts tell me my details are incorrect so I can't pay. And not just small sites - I've seen it on banks, large retailers, companies who should know better.
Sometimes it's due to a legacy back-end system dating back to the 1990's or earlier, so they have to strip out "bad" characters before sending the order or contact form through.
Sometimes it's due to a legacy back-end system dating back to the 1990's or earlier, so they have to strip out "bad" characters before sending the order or contact form through.
This wouldn't be nearly as much of an issue if the validation were consistent...
It's like those infuriating bank / other websites that truncate passwords on login forms but not on the password change form.
It's like those infuriating bank / other websites that truncate passwords on login forms but not on the password change form.
My name has a dash “-“ due to the way my foreign birth certificate is written. I’m sure it’s not as annoying as an apostrophe, but goodness is it frustrating sometimes. For example, some hotels use lastname+room number for the wifi password. I never know if my last name has the dash or not. Sometimes it doesn’t work at all, so I’m not sure if the dash is being replaced by another character. Then there’s the problem of forms that don’t allow dashes in the last name field. Doesn’t sound like a problem until I’m filling out a form that’s meant to be legally binding in some way.
aproppos "Hello, I'm Mr. Null. My Name Makes Me Invisible to Computers"
https://www.wired.com/2015/11/null/
On the plus side, you have a strong privacy guarantee :)
On the plus side, you have a strong privacy guarantee :)
It could be also, like X Æ A-12, or any of the following real names of real persons:
Ben Dover, Hashtag Follow, Destinee Hooker, Khumalo Tiny Dick, Guy Nutter
Guilherme Crabogiale Fuck, Jurassic Park, Beautiful Existence, Tokyo Sexwale, Mike Litoris, Batman Bin Suparman, Jesus Condom
Mrs. Rape, Kash Register, Donald Duck, Robert Fagot, Tatiana Vagina, Hitler Mussolini.
Ben Dover, Hashtag Follow, Destinee Hooker, Khumalo Tiny Dick, Guy Nutter
Guilherme Crabogiale Fuck, Jurassic Park, Beautiful Existence, Tokyo Sexwale, Mike Litoris, Batman Bin Suparman, Jesus Condom
Mrs. Rape, Kash Register, Donald Duck, Robert Fagot, Tatiana Vagina, Hitler Mussolini.
That's so weird to me. It means that somewhere someone is matching the string "null" and converting it into an actual null value.
What is generally happening in these cases is that data is getting stringified, with `null` becoming the string "null" (which is a common output for string formatting a null in various languages).
This output is is then later re-parsed, and to try to restore null inputs back to being null, the code matches on the string "null", ignoring the fact that that could have been a valid string itself.
This output is is then later re-parsed, and to try to restore null inputs back to being null, the code matches on the string "null", ignoring the fact that that could have been a valid string itself.
It's pretty believable to me that _some_ Java deserialization library from the early oughts totally does this as a default behavior.
Not all data structures are nullable or null reference is undesirable. In this case you can use the null object pattern https://en.wikipedia.org/wiki/Null_object_pattern#Descriptio...
I suspect some batch processing job with a CSV dump is the culprit behind a lot of bad treatment of strings.
As I understand it, in the original Irish there's no apostophe and the apostrophe was added by English speakers because of a mistaken idea that it's short for "of the", as in "five o'clock". Of course, a lot of names are also spelt differently in Irish:
O'Rourke = Ó Ruairc
O'Reilly = Ó Raghallaigh
O'Donovan = Ó Donnabháin
So you could get rid of the apostrophe and at the same time really spice your name up, perhaps.
O'Rourke = Ó Ruairc
O'Reilly = Ó Raghallaigh
O'Donovan = Ó Donnabháin
So you could get rid of the apostrophe and at the same time really spice your name up, perhaps.
They'll get a middle initial of "O" probably due to that space character, and similar issues in matching their last name.
It's frankly infuriating how names with apostrophes were considered "invalid" even in systems designed for the US population (including a great number of Irish immigrants - ah but blame them for having "funny names" - that are even full ASCII)
My first name contains an accented e, and what the computer will spit out is a crapshoot: é, e, é, é or just a plain blank square. It makes me want to buy an internationalised domain name out of spite.
Internationalised domain names will display fine because they are punycode that browsers and other clients happen to render as internationalised domains.
E-mail is even worse than web applications at this because even common and popular mail clients can't do UTF-8 in email addresses right (https://www.google.com/search?q="SMTPUTF8"+site%3Adovecot.or...).
E-mail is even worse than web applications at this because even common and popular mail clients can't do UTF-8 in email addresses right (https://www.google.com/search?q="SMTPUTF8"+site%3Adovecot.or...).
In my experience Punycode support in browsers varies greatly depending on the phishing[1] mitigations in place, making them quite unreliable in practice unfortunately.
I'm pleasantly surprised about SMTPUTF8 being a thing though. Yeah nobody uses it, but I thought email had zero UTF-8 support so it's at least better than nothing, I guess.
[1] https://en.wikipedia.org/wiki/IDN_homograph_attack#Homograph...
I'm pleasantly surprised about SMTPUTF8 being a thing though. Yeah nobody uses it, but I thought email had zero UTF-8 support so it's at least better than nothing, I guess.
[1] https://en.wikipedia.org/wiki/IDN_homograph_attack#Homograph...
SMTPUTF8 is about unicode email addresses. I think, UTF-8 messages have wide support.
Sorry, I was indeed talking about Unicode email addresses but I worded it poorly. Thanks for the precision!
There's also a famous Dutch physicist, Gerard 't Hooft, whose family name is "'t Hooft", including the initial(!!) apostrophe. Imagine that.
Imagine if even strongly typed languages had different string types for various encodings like that and enforced them. Then such bugs would be as rare as, say, a date appearing as a huge integer.
To a degree, Rust does this. The standard string type is always guaranteed to be valid UTF-8. Then there's a string type for the OS native format and a string type for what constitutes a valid path on the file system. Of course, arrays of bytes also exist and so does a type for a null-terminated "CString". Converting between them requires runtime checks (unless you invoke unsafe code and promise you know better than the compiler).
It's cumbersome at times, but a perfectly reasonable tradeoff in my opinion. Some libraries add other string types when they're needed to interact with a system. For example, the windows crate has wide strings.
It's cumbersome at times, but a perfectly reasonable tradeoff in my opinion. Some libraries add other string types when they're needed to interact with a system. For example, the windows crate has wide strings.
This is a good idea I believe.
While I agree it would be convenient to have it in the standard library of more languages, it can be implemented in strongly typed languages, using something like Newtype in haskell. In Typescript, for example, it could be:
Same could be done to represent a distance in Km or miles:
type SanitizedString = string
Same could be done to represent a distance in Km or miles:
type DistanceKm = number
type DistanceMiles = number
There was a research C compiler, maybe related to the meta-compilation effort that became Coverity, that did something like this. It let you attach arbitrary attributes to a variable and then enforce invariants regarding those attributes. The one that caught my eye was tagging checked vs. unchecked user input, so that using such input before checking (or sanitizing) would become a type error. It's a shame that the idea didn't catch on more.
When Python did that to separate raw bytes and proper Unicode, people pointed this out as a reason they don't want to use the language any more.
Sounds like Python wasn't right for those people in the first place. Abstracting away the internal data format is typical of Python.
The title doesn’t do this justice. As it goes on to say it’s not just apostrophes it’s practically everything outside the incredibly limited and non-inclusive range of 26 “ascii” letters.
It’s bad enough that apparently no one can spell my name correctly but when my partner and family with letters such as ö or ł can almost never type their correct name it irritates me greatly.
A few months ago at work I had to actually kick off at product managers and QA that these letters are not “special characters” in the same way ? or & are. In the end I had to refuse to implement the regex in the form validation and just said that it was done and that was the end of it. So incredibly dumb that peoples names are classified as “special characters” by some arbitrary set of rules people subscribe to.
It’s bad enough that apparently no one can spell my name correctly but when my partner and family with letters such as ö or ł can almost never type their correct name it irritates me greatly.
A few months ago at work I had to actually kick off at product managers and QA that these letters are not “special characters” in the same way ? or & are. In the end I had to refuse to implement the regex in the form validation and just said that it was done and that was the end of it. So incredibly dumb that peoples names are classified as “special characters” by some arbitrary set of rules people subscribe to.
Copy pasting code shared on slack, why do the apostrophe characters get converted into a different character that looks like an apostrophe?
This was going to be my point (roughly).
It's like when you travel and you start to judge gas stations by their bathroom. Anyone who has traveled a lot knows exactly what I mean.
copy/paste worked more reliably 20 years ago than it does today. So much so that I've started judging software based upon their ability to reliably copy/paste. Seems stupid, right? Just like judging gas stations by their bathroom seems stupid to people who rarely go into them because they don't travel.
But it flat out amazes me how bad copy/paste is today.
I once spent _hours_ trying to figure out why a PHP script was shitting all over itself. It turns out I copy/pasted the email text from Monday.com and it copied non-printing characters that caused PHP to fall over.
Thing is, at that point, I was an old hand at that stupid crap, it didn't take me that long to find it. I learned years ago not to trust copy/paste.
It's like when you travel and you start to judge gas stations by their bathroom. Anyone who has traveled a lot knows exactly what I mean.
copy/paste worked more reliably 20 years ago than it does today. So much so that I've started judging software based upon their ability to reliably copy/paste. Seems stupid, right? Just like judging gas stations by their bathroom seems stupid to people who rarely go into them because they don't travel.
But it flat out amazes me how bad copy/paste is today.
I once spent _hours_ trying to figure out why a PHP script was shitting all over itself. It turns out I copy/pasted the email text from Monday.com and it copied non-printing characters that caused PHP to fall over.
Thing is, at that point, I was an old hand at that stupid crap, it didn't take me that long to find it. I learned years ago not to trust copy/paste.
If you're on windows Ctrl+Shift+V is your friend.
Often it’s smart quotes or similar being stupid.
Curling smart quotes correctly is a deceptively hard problem. I implemented a basic natural language parser that does a decent job. However, straight quotes are ambiguous. Compounding the problem is that curled apostrophes and right single quotes are indistinguishable, which is an oversight in Unicode, IMO, because they are semantically different.
Here's my swat at the puzzle:
https://whitemagicsoftware.com/keenquotes/
When exporting Markdown documents to HTML or PDF, my text editor uses KeenQuotes to curl straight quotes to their most likely candidate:
https://github.com/DaveJarvis/keenwrite#download
Here's my swat at the puzzle:
https://whitemagicsoftware.com/keenquotes/
When exporting Markdown documents to HTML or PDF, my text editor uses KeenQuotes to curl straight quotes to their most likely candidate:
https://github.com/DaveJarvis/keenwrite#download
While I don't blame you or anything, this only works for English conventions. Dutch quotation marks are different, at least if you try to do smart quotes correctly.
Expected output: „Goedemorgen,” zei hij.
Returned output: “Goedemorgen,” zei hij.
And that's just language to take care of! Smart quotes for computers are hell and there should be a library to do all this so everyone doesn't need to reinvent the wheel.
Expected output: „Goedemorgen,” zei hij.
Returned output: “Goedemorgen,” zei hij.
And that's just language to take care of! Smart quotes for computers are hell and there should be a library to do all this so everyone doesn't need to reinvent the wheel.
Makes it even harder that some languages use straigth quotes as the gold standard – so whether to curve them or not is language dependent.
Which languages prefer straight quotes? Wikipedia doesn't mention any: https://en.wikipedia.org/wiki/Quotation_mark
Looks like hair splitting to me, they are all interchangeable. Apparently I used Dutch an Danish variants and nobody ever batted an eye. Cursive quotes look like just cursive version of straight quotes, you can't possibly differentiate them.
Quotation marks arguably depend on the country more than on the language, so Switzerland quotes like «this» in French, German and Italian, while the Germans, at least, don't quote like that.
It seems that Sweden and Finland use the same character for opening and closing quotes, like ”this”, which is similar to using straight quotes.
It seems that Sweden and Finland use the same character for opening and closing quotes, like ”this”, which is similar to using straight quotes.
I was going to say "Finnish" (that's my native language), but to my surprise, after checking it, we use "”" after all. (We use that both as the opening and closing quotation mark.)
OK, I'll whack the hornets' nest here: Base 10 Arithmetic should be the standard, with others being available only when wise programmers explicitly choose it because they have a good reason.
Every tangle I've ever had with IEEE 754 would have been unnecessary if our computers and languages used the same numbers that humans do! (And we have way more than enough compute power now to make Base10 arithmetic the default - I'll cut the folks in the '60s some slack due to resource limits, but why are we still dealing with this brokenness in the 21st century?)
Every tangle I've ever had with IEEE 754 would have been unnecessary if our computers and languages used the same numbers that humans do! (And we have way more than enough compute power now to make Base10 arithmetic the default - I'll cut the folks in the '60s some slack due to resource limits, but why are we still dealing with this brokenness in the 21st century?)
"IEEE754" is ubiquitous because it was the first non-platform-specific solution to a fundamentally hard engineering problem (how to approximate the entire real number line with a finite number of bits), and because it is supported in hardware (on the CPU itself), so that the representation of the number fits within the same registers as used for pointers, integers, and everything else, which makes it fast.
If you want base-10 encoding of non-integer values, and base-10 arithmetic on them, you're probably going to need a software library like [1], which implements the decimal math that is also in IEEE754, just not part of common hardware. It also may be that you actually want infinite-precision arithmetic, which definitely requires a library, because the representation of a number becomes its own dynamic data structure (like with a string).
Can you say more about what exactly is the "brokenness" that you're running up against?
[1] https://www.intel.com/content/www/us/en/developer/articles/t...
If you want base-10 encoding of non-integer values, and base-10 arithmetic on them, you're probably going to need a software library like [1], which implements the decimal math that is also in IEEE754, just not part of common hardware. It also may be that you actually want infinite-precision arithmetic, which definitely requires a library, because the representation of a number becomes its own dynamic data structure (like with a string).
Can you say more about what exactly is the "brokenness" that you're running up against?
[1] https://www.intel.com/content/www/us/en/developer/articles/t...
I disagree: instead humans should drop base-10 entirely everywhere and use base-2 or base-16 instead.
IEEE 754 does define decimal types in addition to binary ones.
One of the nice little details of email address syntax is that ' is not a special character, so niall.o'[email protected] is just fine.
One of the vexing things about MS Outlook and Exchange is that they insist on using a non-standard syntax in which ' is used as a quote, so they make an appalling mess of mail headers.
One of the vexing things about MS Outlook and Exchange is that they insist on using a non-standard syntax in which ' is used as a quote, so they make an appalling mess of mail headers.
It annoys me that apostrophe and English closing single quotation mark are the same character in Unicode. I'm tempted to use ' for apostophe so that the quotation marks (‘...’) match up properly.
> It annoys me that apostrophe and English closing single quotation mark are the same character in Unicode.
They're not. You should be more than just tempted to use ' (U+0027 APOSTROPHE) for apostrophe, because it is. Closing single quotation mark is ’ U+2019 RIGHT SINGLE QUOTATION MARK.
They're not. You should be more than just tempted to use ' (U+0027 APOSTROPHE) for apostrophe, because it is. Closing single quotation mark is ’ U+2019 RIGHT SINGLE QUOTATION MARK.
> When text is set, U+2019 right single quotation mark is preferred as apostrophe, but only U+0027 is present on most keyboards. Software commonly offers a facility for automatically converting the U+0027 apostrophe to a contextually selected curly quotation glyph.
https://www.unicode.org/versions/Unicode14.0.0/ch06.pdf
Typographically, U+2019 is the appropriate glyph for quotation marks in English and in some other western languages.
https://www.unicode.org/versions/Unicode14.0.0/ch06.pdf
Typographically, U+2019 is the appropriate glyph for quotation marks in English and in some other western languages.
That doesn't reflect actual practice any more than the previous Unicode recommendation to use U+02BC MODIFIER LETTER APOSTROPHE (which at least had some semantic justification in that quote marks are not part of words). Yes, typographically an apostrophe looks much like a right single quote (although the position may be different); in the current century fonts handle that.
> in the current century fonts handle that
Except that they can’t, because (due to the limitations of ASCII), U+0027 had an overloaded meaning, seeing that the character had to cover both opening and closing quotes (as well as being used as a prime tick). Which is why pretty much every font I’ve seen has a straight glyph at U+0027, whereas the glyph at U+2019 is the one you’d usually want for an apostrophe. You can also observe that word processors follow the Unicode standard in this regard. E.g., after typing “they're” into Libre Office, once you copy-paste it, you’ll see that the ASCII single quote has turned into U+2019.
Except that they can’t, because (due to the limitations of ASCII), U+0027 had an overloaded meaning, seeing that the character had to cover both opening and closing quotes (as well as being used as a prime tick). Which is why pretty much every font I’ve seen has a straight glyph at U+0027, whereas the glyph at U+2019 is the one you’d usually want for an apostrophe. You can also observe that word processors follow the Unicode standard in this regard. E.g., after typing “they're” into Libre Office, once you copy-paste it, you’ll see that the ASCII single quote has turned into U+2019.
There are dedicated opening/closing quotation mark characters too, U+2018 and U+2019: ‘’
Is there a good dataset that can be used to test such things? I'm thinking of a database of all kinds of names, addresses and emails that could be found in the wild.
For fuzzing, the Big List of Naughty Strings is a nice starting point: https://github.com/minimaxir/big-list-of-naughty-strings
I think we need a Big List of Nice Strings that your program must handle correctly.
It would be forgiven for rejecting an evil string, but names with apostrophes must be handled.
It would be forgiven for rejecting an evil string, but names with apostrophes must be handled.
I don't get it. There are no strings that are good to reject (not even the empty string). You gain nothing but bugs, and people that you turn their lives miserable.
I usually write a test with 'Россия&, '€& [shrug emoji in there dropped by HN idiocy], or the like as the input. If it handles that correct, then it'll probably be fine for everything else too.
One Person once sued their Bank to correct their Name, under the GDPR right oft correction. The Bank initially refused, because they considered their legacy EDCDIC system unreasonable. The Bank lost the case.
https://gdprhub.eu/index.php?title=Court_of_Appeal_of_Brusse...
https://gdprhub.eu/index.php?title=Court_of_Appeal_of_Brusse...
* Cries in Portuguese *
If you don't trust developers to use prepared statements, banning input strings with apostrophes can be a blunt way to avoid SQL injection attacks.
Robert'); DROP TABLE Students;-- https://xkcd.com/327/
Robert'); DROP TABLE Students;-- https://xkcd.com/327/
Ah, the old “make things easy for the computer” philosophy, much less work than diligently sanitizing input.
But it's not easy for the computer, since it needs to recompile the statement every single time. It's just easy for the programmer, and even that is questionable.
Damn you Bobby Tables!!! (fist hanging in the air)
Not trusting your developers to use prepared statements is a big problem beyond the sqli
A big problem that many, many organizations have, unfortunately.
I _just_ rejected a PR yesterday from a contractor that did the initial checkin of a new project with no .gitignore file and all the build artifacts (everything under obj and bin directories).
Why in the world am I having to protect the project from such things? I understand ignorance, but the person putting together the base of the project should know what the hell is going on.
Why in the world am I having to protect the project from such things? I understand ignorance, but the person putting together the base of the project should know what the hell is going on.
Oof. And I was just lamenting repos littered with .DS_Store files...
The US Federal government (and I imagine most/all states) only being able to handle names comprised of unaccented ASCII characters certainly doesn't help anything.
Further evidence that O'Brien must suffer.
Gartner top lead quadrant solution cannot handle dot., minus - and the likes, fun stuff too
Tyranny of the ASCII and EBCDIC.
user_named(4)
With very few exceptions (Gab, et al), the Internet cannot handle Free Speech in 2022. Especially speech that contravenes the official government narratives - that gets brutally shut down. This will not end well.