How we used Node.js to build real-time award flight search(blog.milewise.com)
blog.milewise.com
How we used Node.js to build real-time award flight search
http://blog.milewise.com/post/8792373531/how-we-used-node
12 comments
There are deferreds in node as well.
They aren't part of the core API like they are in Twisted, is this correct?
No. Node is a lower-level API than Twisted.
The "fibers" utility is quite popular. There are also several different promise libraries, and a few code-translation approaches that simulate coroutines.
They all build on top of the very simple EventEmitter and continuation-passing-style ("callback") style that the node-core API uses.
As a JavaScript luddite, I prefer passing functions around and listening to events :)
The "fibers" utility is quite popular. There are also several different promise libraries, and a few code-translation approaches that simulate coroutines.
They all build on top of the very simple EventEmitter and continuation-passing-style ("callback") style that the node-core API uses.
As a JavaScript luddite, I prefer passing functions around and listening to events :)
So does milewise scrape the airline sites for award travel? Does it also integrate with ITA or similar?
I ask because (1) the airline websites tend to be _really_ slow and (2) they tend not to find the cheapest/best flights - an ITA search can often beat them.
I ask because (1) the airline websites tend to be _really_ slow and (2) they tend not to find the cheapest/best flights - an ITA search can often beat them.
We actually do both, and then spent a few cycles de-duping etc.
1) Part of our reason for using Node+Socket.IO is to deal with the problem of having a bunch of different sources returning data over a wide spread of time.
2) You'd be surprised about the airline sites. While they don't necessarily have the widest variety of good fares, airline will often reserve their cheapest fares for their own sites (called Last Seat Availability).
1) Part of our reason for using Node+Socket.IO is to deal with the problem of having a bunch of different sources returning data over a wide spread of time.
2) You'd be surprised about the airline sites. While they don't necessarily have the widest variety of good fares, airline will often reserve their cheapest fares for their own sites (called Last Seat Availability).
Thanks ... looks like it's time to sign up for MileWise!
Not related to Node, but your homepage is really well designed. Attractive, simple and clean. Nice.
Sacha Greif did our homepage. (http://www.sachagreif.com/)
He's awesome, super talented and incredibly fun to work with.
He's awesome, super talented and incredibly fun to work with.
Could you describe the difficulties you encountered with Tornado and asynchronous Python code?
Don't want to turn this into a Node.js/Python flame war (we love Python as well here at MileWise), but regarding asynchronous Python code:
It's more just an annoyance of the language. Specifically, anonymous functions are must easier in JavaScript, so it was just easier to write read and write maintainable code.
In Python, you are limited to only passing in simple lambda functions (although deferreds do get around this issue somewhat). It's not a show stopper but when combined with all the other advantages of Node.js, it just made the decision to switch a lot easier.
We also ran into a lot of issues related to poor garbage collection in Python and were using up a lot of memory. This was not related to dangling references but rather Python's permanent allocation of Integer and Float objects.
Finally, we just saw better performance out of V8 than Python (although we never tried anything else besides vanilla Python 2.6).
For our specific problem, Node.js worked well. It's not always the right solution. We will write a more detailed blog post soon with specifics of how we used Node.
It's more just an annoyance of the language. Specifically, anonymous functions are must easier in JavaScript, so it was just easier to write read and write maintainable code.
In Python, you are limited to only passing in simple lambda functions (although deferreds do get around this issue somewhat). It's not a show stopper but when combined with all the other advantages of Node.js, it just made the decision to switch a lot easier.
We also ran into a lot of issues related to poor garbage collection in Python and were using up a lot of memory. This was not related to dangling references but rather Python's permanent allocation of Integer and Float objects.
Finally, we just saw better performance out of V8 than Python (although we never tried anything else besides vanilla Python 2.6).
For our specific problem, Node.js worked well. It's not always the right solution. We will write a more detailed blog post soon with specifics of how we used Node.
>> It's more just an annoyance of the language. Specifically, anonymous functions are must easier in JavaScript, so it was just easier to write read and write maintainable code.
That's pretty specious. You can easily write a function defined inside a function that you can pass by just naming it. "Anonymous" functions are just syntactic sugar. Furthermore, writing a new python named function is easier (fewer chars) to write than a javascript anonymous function, so the "sugar" gained is moot.
Finally, as you build more and bigger systems, you realize that those multi-line anonymous functions you had you actually want to be named (for documentation) and tested. So you'll end up de-anonymizing the complex (multi-line) ones anyway.
As for speed, why not compare a jit (v8) to a jit (pypy) ?
That's pretty specious. You can easily write a function defined inside a function that you can pass by just naming it. "Anonymous" functions are just syntactic sugar. Furthermore, writing a new python named function is easier (fewer chars) to write than a javascript anonymous function, so the "sugar" gained is moot.
Finally, as you build more and bigger systems, you realize that those multi-line anonymous functions you had you actually want to be named (for documentation) and tested. So you'll end up de-anonymizing the complex (multi-line) ones anyway.
As for speed, why not compare a jit (v8) to a jit (pypy) ?
This is getting ridiculous. These guys were busy launching a product, not trying to write a balanced comparison of the pros and cons of every evented framework out there.
Node worked better for them, what's the problem?
Node worked better for them, what's the problem?
Writing functions inline is a huge win, not just because of the number of characters you have to type.
It creates less noise and busy work in your code.
Explicitly specifying functions like having to pre-declare all your variables at the top of your scope.
It creates less noise and busy work in your code.
Explicitly specifying functions like having to pre-declare all your variables at the top of your scope.
I don't see how it's specious to prefer javascript syntax over python syntax for asynchronous code. It's an aesthetic consideration and so it's less about the fact that you can have the same sorts of functionality and structures in both environments and more about which style people prefer.
I really don't understand why whenever somebody switches to node from python people pounce on them like this, as if people aren't allowed to find a particular syntax annoying?
I really don't understand why whenever somebody switches to node from python people pounce on them like this, as if people aren't allowed to find a particular syntax annoying?
While it doesn't help you for the other issues you mentioned, I have found defer.namedCallbacks to make Twisted code much simpler to read.
This is a killer product and very useful for people who travel often. Way to go! Also, thanks for sharing your code. You all seem like a class act.
One comment: I typed in an email/password and right away it is asking me for my airline credentials but it isn't explaining what it is that the site wants to do with them or if it will store my info and collect information from now on. I'm curious about the site, but I thought twice about entering my info and skipped it. Just saying you might want to explain what it is you are doing to make people more comfortable.
EDIT: This information does seem to be covered in the FAQ, but you don't see the link to the FAQ anywhere when you are trying to capture user accounts on initial signup: https://secured.milewise.com/faq
EDIT: This information does seem to be covered in the FAQ, but you don't see the link to the FAQ anywhere when you are trying to capture user accounts on initial signup: https://secured.milewise.com/faq
Great feedback - we weren't sure whether at this point people would have a clear idea from the homepage of what we were doing w/ the info and they'd just want to get on with things, or if they'd want more contextual explanation.
This is definitely an area for metrics and user testing, but probably better for us to err on the side of caution =)
This is definitely an area for metrics and user testing, but probably better for us to err on the side of caution =)
I mean, I'd like to think you were going to provide me some great service.. but then that means I could register a domain name and put up a slick looking homepage and start collecting passwords :)
Just a side note, but do you use some other collection API such as Yodlee or are you maintaining your own data scrapers for all the airline sites?
Just a side note, but do you use some other collection API such as Yodlee or are you maintaining your own data scrapers for all the airline sites?
Aw shucks you've uncovered our master plan.
We have our own account scrapers augmented with a partner API from AwardWallet for the more esoteric programs (CVS Rewards anyone?)
We have our own account scrapers augmented with a partner API from AwardWallet for the more esoteric programs (CVS Rewards anyone?)
Interesting... I'd like to talk to you more offline. If you want to chat, shoot me an email, andrew at ishiboo dot com.
Great use case for Node.js.
Great use case for Twisted, Erlang, Ocaml/Lwt, Lua, Python/Gevent too.
Great point! I bet that now thanks to your razor wit, dcaylor will think twice before saying something nice about node without also saying something nice about every other programming platform.
Node gets a lot of attention despite not doing much of anything new, sometimes it can be helpful to point this out. I was not rude or insulting about it, unlike others.
I laughed at both your initial response and IsaacSchlueter's comment. Valid points from both.
It would have been a little odd for me to say "Nice use case for Python" considering the original post was about Node, and specifically said they tried writing it in Python but "writing asynchronous code in Python is like using your keys to open a beer bottle." That also made me laugh.
In fact, the original article specifically talked about why Node.js was a good choice for this use case. I agree. Not that you couldn't do it in other ways. Erlang would also be a good choice here, I'm sure. There is always more than one way to do most anything. Still, and in spite of the fact that Node might be getting too much hype, it is an excellent tool for this type of async need.
It would have been a little odd for me to say "Nice use case for Python" considering the original post was about Node, and specifically said they tried writing it in Python but "writing asynchronous code in Python is like using your keys to open a beer bottle." That also made me laugh.
In fact, the original article specifically talked about why Node.js was a good choice for this use case. I agree. Not that you couldn't do it in other ways. Erlang would also be a good choice here, I'm sure. There is always more than one way to do most anything. Still, and in spite of the fact that Node might be getting too much hype, it is an excellent tool for this type of async need.
Well, I hope you learned your lesson.
> Node gets a lot of attention despite
> not doing much of anything new
If anything, node's success so far has been a result of combining established technologies in ways that make it easy for more people to write reasonably fast programs that use the internet in fun and interesting ways.
Novelty isn't really what it's all about. If anything, I look at novelty as a hazard. Most ideas are bad, and it's hard to tell when they're new. If you can take some existing good ideas, and make them more fun, that tends to be a more successful strategy (or at least a more enjoyable one.)
Nicely done! Just, the "How" in the title is misleading... ;)
I have to sign up to search? Is this in production yet?
Can you go into more detail about how js on front and back helped? What percent of your codebase is actually shared? What type of functions are they? Many people claim that it's a big win while others call shenanigans but I haven't heard empirical data from anyone.
hey nmeyer im interested in building some similar stuff could you email me at [email protected] so we can talk more. Thanks.
Awesome. Emailed!
and as a former ITA Software employee, i can scoff and say they aren't actually doing anything resembling flight search. they're just scraping websites and APIs. node.js might be fine for that, but so is anything else. at that level it's more a matter of preference than capability.