No, this encoding uses a bitwise encoding for each float, so they are fixed at 10 characters each and encoded with constant speed (except possibly for subnormal numbers, which I've heard are slower in some environments).
That's a definite possibility. We went with base-94 because it avoids UTF-8 overhead in situations where the browser will treat your data as a string (e.g. no typed-array support). We also wanted copy/paste to work, and for Flotsam data to be embeddable into JSON with minimal expansion.
That's not why I did it. I felt bad because my family and friends know me as being unreasonable, idealistic, etc. But HN doesn't know anything about me and might assume I'm an otherwise reasonable person. So I wanted to make sure I wasn't badmouthing Google here. They really are an awesome company and the point was never to make them look bad. More just to explain about where the incompatibility was.
Hang on, you're being a little unfair about this. I believe in what I said. I think FP is a great step forwards and I think it's going to win out. I'm biased because I think this.
But Google is worth billions of dollars and is succeeding wildly at innovating and changing the world, and I'm a lone developer. I can't very well stand here and tell Google, "hey you billionaire company, you're doing it all wrong; I've got the magic bullet right here."
Part of the dichotomy is whether to define success as financial wealth or as some kind of more abstract software quality. And I think that's a legitimate variable; if Google is wealthy but has lower-quality software for it, then they and I just have different priorities.
I stand for my opinion by investing all of my spare time into FP language development and research, but until I see a billion-dollar company using it, I'm not going to call anyone a loser for choosing Java.
I'm really glad you wrote this, and I'm now linking to it from my post. Definitely agree that you should try Google for yourself. It didn't work out long-term for me, but hopefully it's obvious that that isn't saying a lot.
I originally wrote it for that reason: it's sort of a mind-bending exercise and I wanted to see if it could be done. (The first attempt was actually in Bash, but that didn't work at all.) Since refining the implementation a bit, I've found these scripts to be unexpectedly useful. I mainly use them to encapsulate things like build rules, though now they have an inheritance hierarchy and some standard library functions that I reuse. Because these scripts are self-contained, I can upgrade library code without breaking old projects.
The coolest thing about these files, I think, is that they support inheritance from one another and are fairly smart about it. (This feature didn't exist at first, and as a result they were impossible to manage.) If you locally modify one to differ from one of its parents, it makes a note of this and preserves your changes to the modified attributes during upgrades. I use this on the Caterwaul page to provide custom HTML -- most scripts load some Javascript that shows you the attribute table.
I think it's an imperfect solution, though it might still be solid. It depends on your reduction strategy (basically breaking associativity). If you do a fold-left, then you'll accumulate such high relevance that you'll quickly start discarding every word in the right-hand data set. On the other hand, if you binary-split the folding process you're more likely to be OK.
I used a binary split and due to the sparseness of the data (and the uninsightfulness of my algorithm) didn't run into too many spurious drop cases. But the implementation I posted is very basic and shows my lack of background in NLP-related pursuits -- I'm lucky it did anything useful at all :)
The basic mechanism is to call f.toString() on some function, then construct a new one the way you want it using either eval() (for non-IE browsers), or new Function(code) (for all browsers). Caterwaul leverages this by implementing a JS parser, defining a macro syntax, and letting you determine how the code gets rewritten. (So you can do things like let[x = 5] in x + 1, which is valid JS to begin with but nobody would ever use it :) )
Very true. It's also one of the only languages out there to let you inspect functions' source code, which means you can implement full-blown Lisp macros in it using eval(). Given its popularity and the runtime performance of V8, I think this makes it the ultimate language for the next decade.
I have a preliminary implementation of this, geared for map/reduce-style parallelism in Perl, up at http://github.com/spencertipping/metaoptimize-challenge (in the fcm directory). It may be a start to solving step 3 -- using reduce-by-two on each step and dropping low-relevance words (I think this is valid, though I'm not 100% certain).