I'm not sure what resolution these images are, but if it's very high-res, it may make sense to make intermediate-sized versions first. I expect that would make recreating thumbnails (i.e. in a different size - or multiple sizes) much quicker.
Also, converting the (hi-res) images to JPEG before further processing may make the data somewhat easier to deal with. 25-30MB per image * 2 million would be 50-60 terabytes of data? Plus, how often would someone really benefit from having access to a 25-30MB TIFF vs a much smaller JPEG version in the same resolution? That alone could be reason to start with JPEG compression first.
It's not about performance, it's about <del>parallelism</del> concurrency.
If you don't care for Node's ability to handle multiple requests <del>in parallel</del> concurrently, with a single process with only one thread, than you might not have a need for Node at all. Why not use Ruby or Python?
If you really want to force Node into being a dumb scripting engine, you could use the various sync functions that Node.js provides. Problem is, they block the entire thread. But if you want to, you can. And I believe with some clever hacks, you can use the blocking nature of (for example) the fs sync functions to make other calls (including http requests) synchronous as well. But unless you use Node to build something that's not a server (people build command line tools with it as well), I don't think you want to.
By using synchronous functions, everything Node is built for will go down the drain. Everything done will be done in sequence, and during the time the sequence of operations has not finished, the Node runtime will be unavailable to serve any requests.
I suggest you look into promises as a somewhat saner way to deal with asynchronicity than callbacks, although you can pretty far with just the Async library. The future might be in Generators, a new feature of EcmaScript 6. This is available in V8. You can use this in Node 0.11 (unstable) if you use the --harmony flag. See here for a nice writeup: http://blogs.atlassian.com/2013/11/harmony-generators-and-pr...
The point I was trying to make is that I see this graph as doing something comparable to saying you're (by definition) born in "the year zero, at 00:00:00".
Does anyone else have trouble understanding (or "intuitively reading") the graph in the f.lux beta preferences? I discovered that's a kind of "ego-centric" graph. I mean ego-centric just like there once where earth-centric (and later) helio-centric models of the universe.
Because the graph is totally ego-centric, the graph starts when you wake up. I just can't wrap my head around that. In my mind, I wake up at a specific clock time, and the universe is configured in a certain way at this particular moment. In particular, the sun has a certain position in the sky. (interestingly, I use an earth-centric model in this regard).
What's (relatively) constant for me is how the sun moves through the sky (this depends on where you live on earth, plus time of year). Obviously, it's beyond my powers to change the time of year. I could change where I live on earth, but I'm not doing that very often. What's directly controlled by me is when I wake and go to bed... Why can't I change these positions on an otherwise static "map"?
I don't want to express the current year as relative to my life either. I.e. three periods: "the time I hadn't been born yet", "the time that I live", "the time beyond when I died". It's rather insane. Yes, we use Jesus date of birth as a reference point now, you could say that it's bad and we should count from a different epoch or so, but at least things are not expressed relative to my life.
This is a nice example of a "commitment device", "a means with which to lock yourself into a course of action that you might not otherwise choose but that produces a desired result". https://en.wikipedia.org/wiki/Commitment_device
"Attendees [...] learned a technique called "pre-hindsight" that uses emotional cues to create more foolproof plans. It works like this: Imagine that six months have passed, and you haven't achieved the body of your dreams. How surprised are you? The less surprised you are, the less likely it is you will succeed at your goal. Then think in detail about each reason you wouldn't be surprised if June comes and the number on the scale hadn't budged. Each reason—whether "I don't have time" or "I don't like running in the mornings"—is a possible cause of failure. Using the surprise level to anticipate these is crucial to creating a plan to address each weak point."
I like this technique a lot. Thinking about how surprised you'd be if you do not achieve your objective.
I think the technique has similarities with the exercise that Steven Covey writes about in "The Seven Habits" where you must think about what you want other people to say about you at your funeral. It engages you in a special kind of way, perhaps helping you to uncover knowledge that you did not know you held.
"This is annoying at the level of Arrays, but gets more difficult with more complex types, and function interactions. The recent brouhaha around the Promise/A+ highlights one such example: It is difficult to return a Promise of a Promise as a value from onFulfilled because then duck-wraps the return value as described in the Promise Resolution Procedure"
Could anyone share a pointer to this brouhaha around the Promise/A+ [spec, I assume]?
It seems the blog had a kind of "invisible" filter on the comments, which made it appear there were zero comments, or one (your own) if you had posted one. Now a whole pile of comments have become visible (manually approved I think). Funny to see all the reactions. Most commenters were probably not aware of one another.
If I ever would be doing such a thing, it would be more of a lesson in how to use node.js properly (even for non-typical tasks like sorting) then a performance comparison between node.js and .Net. Even then, I don't think it would be very interesting for people who like to read about node.js. It would be non-news to them, and for an atypical use-case.
I actually have no interest at all in getting a .net stack running on my mac.
The misleading information is what bothers me, so I hope to see it corrected. It's especially harmful because his blog will be most likely read by people who naturally favor .net. It's not even a tech blog per se. Then objective information is all the more important. (EDIT: Actually it does seem to a tech blog, which surprises me a bit, given the superficiality of his analysis)
You can see at the end of his posts, how he diverts from the main subject and goes on about (I'm sure to be) wonderful technologies and possibilities that the .net stack offers.
Thinking this through some more, using sort with a comparator function would be unnecessary slow. Given the objective of the algoritm (return a median), it's much better to convert the array of strings to an array of floats (or ints, whatever he wants) first.
I posted this comment:
"It's obvious that you're new to Node.js.
First of all, you should be aware that Async.js is a mere flow-control library. It does not offload work to separate threads, and neither is it able to parallelize work. Internally, it mostly does bean-counting (but very helpfully so).
As you can see in the source https://github.com/caolan/async/blob/master/lib/async.js#L35... async.sortBy simply uses Array.sort for the actual sorting. The only reason you'd want to use Async.sortBy is if the values that the array of "keys" is not known beforehand (and needed to be loaded through io - asynchronously). This is clearly exemplified in the documentation. https://github.com/caolan/async#sortBy
The implication of this that your call to async.sortby can be replaced by a call to array.sort. This will remove two unnecessary runs of async.map, inflicting a potentially huge performance penalty.
You do need to pass array.sort a comparator function, otherwise it will sort lexicographically (see https://developer.mozilla.org/en-US/docs/JavaScript/Referenc... ). That said, I'm not sure what the actual contents of your input file is. In your .Net example, you do not seem to bother to convert the array of strings to an array of ints (or floats). I think that .Net sort will sort an array of strings lexicographically as well. Furthermore, in the node.js example, you seem to be content with returning the resulting median as an int, not as float. Do the input "decimals" in the input file represent ints or floats? Do they all have exactly the same amount of decimals? Are both the Node.js and .Net algorithms doing the same thing? I think not.
Finally, we get to Array.sort. Array.sort blocks. Depending on the multi-threading efficiency of the underlying algorithm of Array.sort (which I don't have insight in), the code may not be able to use all available system resources. Keep in mind that Node.js is single-threaded. I practically don't know anything about .Net, but I assume it will magically start new processes and or threads if the runtime deems this beneficial. For Node.js, you may want to try the Cluster api, http://nodejs.org/api/cluster.html . You could try seeing if performance increases by adding one or more extra server processes.
I can't comment about the quality of the .net code since I don't have any experience with it.
I think it would be fair (and very educative to others) if you'd rerun the benchmarks with
1. Async.sortBy replaced with array.sort
2. with both .Net and Node.js algorithms fully doing the same thing (i.e. let them both sort either floats, ints, or strings), and
3. at least one extra server process for Node.
I think most interesting would be if you'd made the changes step-by-step, and run the benchmarks at each step.
My guess is that step 1 would give the biggest difference.
Depending on how you decide to resolve the differences in the two algorithms, performance of your .Net code may be slightly affected. It could potentially be speed up in fact, if somehow it's able to sort ints (or floats) faster than strings. The actual job of sorting probably overshadows it all though."
Looks cool!
How do you gauge the difficulty of adding support for CoffeeScript to this?
Re the indiegogo campaign: I think you should consider adding an extra supporter tier between 25 USD and 1000 USD. The 200 USD tier of CoffeeScript Redux http://www.kickstarter.com/projects/michaelficarra/make-a-be... brought in over 20% of the total amount pledged.
Also, converting the (hi-res) images to JPEG before further processing may make the data somewhat easier to deal with. 25-30MB per image * 2 million would be 50-60 terabytes of data? Plus, how often would someone really benefit from having access to a 25-30MB TIFF vs a much smaller JPEG version in the same resolution? That alone could be reason to start with JPEG compression first.