So you think you know what a number is(chris.improbable.org)
chris.improbable.org
So you think you know what a number is
http://chris.improbable.org/2014/8/25/adventures-in-unicode-digits/
9 comments
Speaking of tools having trouble recognizing things, one of the more common "bug reports" I get is that the Tarsnap website is disclosing user email addresses -- specifically, "[email protected]" -- and paths -- specifically, the paths /home/auser and /home/anotheruser.
>>> int("۲۶۷۹")
2679
Indeed. Seems to be a thing of python 3.
$ ruby -v
ruby 2.2.6p396 (2016-11-15 revision 56800) [x86_64-linux-gnu]
$ ruby -e "puts \"۲۶۷۹\".to_i"
0
$ python2.7 -c "print(int(\"۲۶۷۹\"))"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '\xdb\xb2\xdb\xb6\xdb\xb7\xdb\xb9'
$ python3.5 -c "print(int(\"۲۶۷۹\"))"
2679
$ node -v
v6.9.1
$ node -p "parseInt(\"۲۶۷۹\")"
NaNIt works in Python 2, you just have to ensure that your string is unicode. I can't get it to work directly from the command line for some reason, but if you open an interactive Python session it works:
Python 2.7.12 (default, Dec 14 2016, 13:32:53)
[GCC 4.9.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print(int(u'۲۶۷۹'))
2679Hmm, interesting. What's up with the terminal here? The shell is doing something to the text.
$ python2.7 -c "print(int(u'۲۶۷۹'))"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '\xdb\xb2\xdb\xb6\xdb\xb7\xdb\xb9'
$ python2.7
Python 2.7.13 (default, Jan 03 2017, 17:41:54) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print(int(u'۲۶۷۹'))
2679
$ python2.7 -c "print(list(u'۲۶۷۹'))"
[u'\xdb', u'\xb2', u'\xdb', u'\xb6', u'\xdb', u'\xb7', u'\xdb', u'\xb9']
$ python2.7
Python 2.7.13 (default, Jan 03 2017, 17:41:54) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print(list(u'۲۶۷۹'))
[u'\u06f2', u'\u06f6', u'\u06f7', u'\u06f9']DB B2, etc. are the UTF-8 encodings of U+06F2, etc. So Python is seeing mojibake: U+00DB (Û), U+00B2 (²), etc. which are not digits. Well, one of them kinda is, but it’s No (“Number, other”), not Nd (“Number, decimal digit”).
Yeah I get that, but why is that happening?
I’d guess because CPython is assuming the input to -c is ISO-8859-1 (Latin-1) when it decodes it using Py_DecodeLocale():
main()
…
setlocale(LC_ALL, "")
…
argv_copy[i] = Py_DecodeLocale(argv[i], NULL)
…
mbstowcs() or mbrtowc()
…
setlocale(LC_ALL, oldloc)
…
Py_Main(argc, argv_copy)
LANG="en_US.iso8859-1" python2.7
That's what I thought as well when I first found it – once the initial surprise passed, it seems nice to have something just work for anyone using another script. The Unicode consortium has put an enormous amount of work into building that database and it's nice to reuse that to make something friendlier for humans.
[deleted]
Note that Python has str.isdecimal, str.isdigit and str.isnumeric.
isdecimal isdigit isnumeric
12345 True True True
១2߃໔5 True True True
①²³🄅₅ False True True
⑩⒓ False False True
Five False False False
Don't forget to special case negative numbers though.
int("一万三千二百六十九")
Traceback (most recent call last):
File "python", line 1, in <module>
ValueError: invalid literal for int() with base 10: '一万三千二百六十九'
int("一三二六九")
Traceback (most recent call last):
File "python", line 1, in <module>
ValueError: invalid literal for int() with base 10: '一三二六九'
That actually came up in the comments (http://chris.improbable.org/2014/8/25/adventures-in-unicode-...)
Apparently the Unicode consortium chose to exclude those characters because they weren't encoded in a contiguous sequence so they're defined as Numeric_Type=Digit rather than Numeric_Type=Decimal. They've apparently realized that this is not helpful but chose to apply the updated policy only to new ranges.
Apparently the Unicode consortium chose to exclude those characters because they weren't encoded in a contiguous sequence so they're defined as Numeric_Type=Digit rather than Numeric_Type=Decimal. They've apparently realized that this is not helpful but chose to apply the updated policy only to new ranges.
Speaking of Japanese, I previously wrote some pretty basic code for translating arabic to/from Japanese numerals:
https://github.com/navarr/JapaneseNumerals/blob/master/Japan...
https://github.com/navarr/JapaneseNumerals/blob/master/Japan...
But those are the names of numbers, like "one", and not numerals. "one two three" would be the English equivalent.
At least AFAIK all the non-CJKV scripts (except perhaps Mongolian?) use right-to-left decimal characters with 0. So you should be able to "transliterate" (transnumerate) ۲۶۷۹ to 2979 with a simple look up table and no R-L confusion. In fact ۲9۷9 should be the same.
It's neat how that link lets you run Python interactively in the browser, with https://repl.it/site/languages/python3 . Normally that is just a JavaScript thing.
This applies in Java as well, for instance Integer.parseInt and Character.isDigit. Between Java and Python, you can get all sorts of numeric characters through a sizable percentage of web applications.
I haven't yet found a security vulnerability based in this, but I keep checking. :-)
I haven't yet found a security vulnerability based in this, but I keep checking. :-)
Also .Net: https://blogs.msdn.microsoft.com/oldnewthing/20040309-00/?p=...
… and, yes, that was definitely something I tried to find when I first noticed it.
… and, yes, that was definitely something I tried to find when I first noticed it.
A partition of the rationals into sets U and V such that every u in U is less than every v in V?
Expecting to see some foundational mathematical logic content, was disappointed.
Too bad, it's your old friend Unicode.
No worries, I'm sure the next version of Unicode will have a single glyph that provides a complete introduction to the topic.
We need to extend Unicode to account for quantum computing and qubits. It needs to evolve into Qunicode. Text with this feature baked in could be called Quneiform. It should be stored on light-reflecting crystals that are manufactured using kilns.
You should join the Unicode Consortium; I'm pretty sure you'd fit right in. :)
[deleted]
It's turtles all the way!
I had an encoding problem once, so I used Unicode. Now I have ۲۶۷۹ problems.
Should perhaps be called "So you think you know what a numeral is". Still interesting though.
I was expecting GEB references.
https://en.m.wikipedia.org/wiki/Gödel,_Escher,_Bach
https://en.m.wikipedia.org/wiki/Gödel,_Escher,_Bach
Having actually done a little number theoretical stuff back in the day, I see what you mean, but prosaically this was astonishing enough for me to make the link worth posting:
Decimal characters include digit characters, and all characters that can be used to form decimal-radix numbers, e.g. U+0660, ARABIC-INDIC DIGIT ZERO.
Rigid designators vs definite descriptions for numerals, anyone?
Decimal characters include digit characters, and all characters that can be used to form decimal-radix numbers, e.g. U+0660, ARABIC-INDIC DIGIT ZERO.
Rigid designators vs definite descriptions for numerals, anyone?
This is a pretty good article that could satisfy the headline: https://johncarlosbaez.wordpress.com/2016/06/29/large-counta...
Terry, is that you? B^>