Walkthrough of a MongoDB Map/Reduce Job(blog.fiesta.cc)
blog.fiesta.cc
Walkthrough of a MongoDB Map/Reduce Job
http://blog.fiesta.cc/post/10980328832/walkthrough-a-mongodb-map-reduce-job
2 comments
Hey Mike, do you think you could talk about how you test those js functions? Do you actually hardcode them as strings in python, or do you load them from a separate js file that you can develop / test separately. I've always been a little sketched out by the requirement that those MR jobs be written in js, would be curious to hear how you do it.
That's a great question, and to be honest it's a bit of a pain point so I probably should've talked about it in the post.
When developing a new M/R job from scratch I start by mirroring (at least part of) the data to a local database. Then I can iterate locally on the M/R using print() and printjson() to debug the map() and reduce() functions - those will print directly to the database log.
I tend to just embed the map() & reduce() functions as Python strings like you see in the post. I'm confident that there are better ways to handle this, though. One approach that can be interesting is to do development from the shell, that way you can write and debug the map() & reduce() in an actual JS environment. Once you're happy with them you can just drop them in as strings with the rest of your application code. Would love to hear how other people are approaching this stuff, too.
All of that said, I expect that the tooling here will improve over time.
When developing a new M/R job from scratch I start by mirroring (at least part of) the data to a local database. Then I can iterate locally on the M/R using print() and printjson() to debug the map() and reduce() functions - those will print directly to the database log.
I tend to just embed the map() & reduce() functions as Python strings like you see in the post. I'm confident that there are better ways to handle this, though. One approach that can be interesting is to do development from the shell, that way you can write and debug the map() & reduce() in an actual JS environment. Once you're happy with them you can just drop them in as strings with the rest of your application code. Would love to hear how other people are approaching this stuff, too.
All of that said, I expect that the tooling here will improve over time.