Dealing with encodings in Ruby is a pain in the ass, let us help you(github.com)
github.com
Dealing with encodings in Ruby is a pain in the ass, let us help you
https://github.com/brianmario/charlock_holmes
2 comments
I hardly use ruby anymore and if I use it I run 1.8.
Are encodings in 1.9 really still that bad -- several years after its initial release.
Are encodings in 1.9 really still that bad -- several years after its initial release.
I don't really like the framing of the title for this project. Text is no longer treated as a dumb chain of bytes in 1.9, but that's not really what this project addresses. A good article with information about how strings have changed in 1.9 can be found here:
http://blog.grayproductions.net/articles/ruby_19s_string
The problem is, when an outside service or file is provided to you, it doesn't always come with information about how the text is encoded. With Ruby 1.9, you have the option of specifying the character encoding for a given string, but if the string is from an outside source, it's often a guessing game.
http://blog.grayproductions.net/articles/ruby_19s_string
The problem is, when an outside service or file is provided to you, it doesn't always come with information about how the text is encoded. With Ruby 1.9, you have the option of specifying the character encoding for a given string, but if the string is from an outside source, it's often a guessing game.
You can use this library in 1.8 or 1.9. It's especially nice in 1.9 because you can use it to tag strings with the correct encoding based on it's content, which has been a huge source of frustration in 1.9 up to this point.
Really though, the pain in the ass has little to do with 1.9 - encodings are just a pain because they aren't universally supported. So we'll often times get handed bytes with no encoding context so we have to assume or use detection libraries like this.
One argument is "well just use UTF-8 for everything" or "just pass everything around as binary then it'll be fine" - but unfortunately we don't live in a world where either of those solutions will work for everyone.
Really though, the pain in the ass has little to do with 1.9 - encodings are just a pain because they aren't universally supported. So we'll often times get handed bytes with no encoding context so we have to assume or use detection libraries like this.
One argument is "well just use UTF-8 for everything" or "just pass everything around as binary then it'll be fine" - but unfortunately we don't live in a world where either of those solutions will work for everyone.
It's a pity that there's no ruby native way of adding another encoding in Ruby 1.9, it's all down on the C level. Recently I needed to work with the GSM-7 encoding, and while I was tempted to try to add a patch to ruby for including it, I figured it was too marginal to warrant acceptance of the patch.