Brubeck: A new Python web framework running on mongrel2(brubeck.io)
brubeck.io
Brubeck: A new Python web framework running on mongrel2
http://brubeck.io/
52 comments
Amazing! I thought you should know that your second cousin inspired my love of jazz as a child, and Paul Desmond convinced me to pick up the saxophone after hearing Take Five. Thus, jazzychad was born.
The world keeps getting smaller and smaller :)
The world keeps getting smaller and smaller :)
I grew up in the town where he lives (Wilton, CT) and was lucky enough to experience his son Chris playing at our school frequently. Dave himself played a few times too.
I first heard take 5 when I was 11 and I've been a fan of odd time signatures ever since.
I first heard take 5 when I was 11 and I've been a fan of odd time signatures ever since.
I did a double-take too, and wondered if this was your creation!
OK, now this is interesting. Like twisted or Node.js enabled a new class of web applications, this is offering a lot more than just another Rails/Django clone. I'm happy that there are so many frameworks to choose from based on your personal preferences, but I'm really happy to see a project trying something completely different.
This is the first time j2labs crossed my radar… is this part of a larger project? Anyone in the know want to give the backstory on how this came about?
This is the first time j2labs crossed my radar… is this part of a larger project? Anyone in the know want to give the backstory on how this came about?
J2 Labs is the name of my consulting company (and twitter account @j2labs).
Brubeck + DictShield are both projects that have risen from building API's for startups. I aim to make Brubeck as useful as possible for building web projects a quick and easy process, without sacrificing easy scaling.
My thinking here is that a solid model should help developers build their idea fast and make it through the early days of startups without breaking their backs.
I don't particularly like working with Tornado because it offers little support for the plethora of Python drivers that are blocking only. On top of that, the callback model, while powerful, can lead to some seriously confusing code.
I believe people will find the combination of Mongrel2, eventlet and a database agnostic modeling system (DictShield) very flexible.
Brubeck + DictShield are both projects that have risen from building API's for startups. I aim to make Brubeck as useful as possible for building web projects a quick and easy process, without sacrificing easy scaling.
My thinking here is that a solid model should help developers build their idea fast and make it through the early days of startups without breaking their backs.
I don't particularly like working with Tornado because it offers little support for the plethora of Python drivers that are blocking only. On top of that, the callback model, while powerful, can lead to some seriously confusing code.
I believe people will find the combination of Mongrel2, eventlet and a database agnostic modeling system (DictShield) very flexible.
"...Tornado because it offers little support for the plethora of Python drivers that are blocking only."
Could you list an example of what you mean by support for blocking drivers? Thanks.
Could you list an example of what you mean by support for blocking drivers? Thanks.
I guess what he means is that for drivers like mysql which are blocking only there is little support in tornado to not freeze the tornado instance during the request response cycle.
If you do use mysql, you're in luck... here's a pure python client module that can easily be monkey-patched for gevent/eventlet:
http://code.google.com/p/pymysql/
http://code.google.com/p/pymysql/
That is correct. Eventlet has database pooling for python drivers that are written in C.
See my response to your other comment for how eventlet handles drivers written entirely in Python. In short, it converts them.
See my response to your other comment for how eventlet handles drivers written entirely in Python. In short, it converts them.
A quick question ---
can i say use pymongo in Brubeck and you are saying the blocking nature of pymongo won't affect the async nature of Brubeck ??
can i say use pymongo in Brubeck and you are saying the blocking nature of pymongo won't affect the async nature of Brubeck ??
For some reason my other account is blocked from responding so I created this one.
That is correct about pymongo. Eventlet will convert any drivers that are written entirely in Python into nonblocking driver. Gevent, an alternative to eventlet, can do the same. Brubeck supports both.
In addition to that it also makes ZeroMQ nonblocking. The combination of ZeroMQ support and pymongo, pyredis and pyriak all being available entirely as python (bson is in c tho) is what convinced me I had to write a new framework.
That is correct about pymongo. Eventlet will convert any drivers that are written entirely in Python into nonblocking driver. Gevent, an alternative to eventlet, can do the same. Brubeck supports both.
In addition to that it also makes ZeroMQ nonblocking. The combination of ZeroMQ support and pymongo, pyredis and pyriak all being available entirely as python (bson is in c tho) is what convinced me I had to write a new framework.
Your account is probably blocking you from responding because of some flamewar prevention feature Paul put in a while ago that makes it difficult for commenters to have nested discussions with one another--the deeper the discussion, the longer you have to wait before making a reply to a user who just replied to you.
Ahh...did not know that :)
wonderful.
Can you also shed light on thrift protocol ? i have struggled with the blocking nature of thrift(client).
Does eventlet convert blocking thrift client written in python to non-blocking ?
Can you also shed light on thrift protocol ? i have struggled with the blocking nature of thrift(client).
Does eventlet convert blocking thrift client written in python to non-blocking ?
You know more about thrift than I do. If the "blocking" code is in python, eventlet should be able to fix it. You can't do anything about the native machine code, though. It appears that the client library includes a c-based codec to improve performance, but it should not be a problem if the transport is based on python sockets.
I use pymongo with gevent. A small hack is necessary to prevent pymongo from making a new connection for each thread that accesses mongodb (because with gevent, you can have a massive number of threads). I'd imagine the same is true with eventlet.
BTW, how is the performance compared to sync pymongo in your use? I'm testing with eventlet and seeing it about 10-20% slower.
https://gist.github.com/1088452
https://gist.github.com/1088452
Do you mind sharing the details of that hack ??
And Bulbflow was released yesterday -- a new Python persistence framework for graph databases (http://bulbflow.com).
There are so many different web frameworks available for Python now. How would somebody make the choice over Brubeck, Flask, Bottle, web.py, Django, Pylons, etc... I know somebody will probably say something like "use the best tool for the job" but what is best for what job?
edit; That's just Python frameworks. The situation gets even messier when you start to consider every other languages for web development. Lots of 'hip' technologies I would love to put time into learning but I can't learn them all...
edit; That's just Python frameworks. The situation gets even messier when you start to consider every other languages for web development. Lots of 'hip' technologies I would love to put time into learning but I can't learn them all...
Brubeck is a Mongrel2 handler which makes it effectively a ZeroMQ Framework. But it is built with web serving in mind so I (I wrote it) attempted to also bring many of the customs we're used to with web frameworks to the Python + Mongrel2 world.
There are many python frameworks and each serves a different need. If you are a Tornado user but find the blocking / nonblocking paradigm confusing, you would probably get a lot out of Brubeck because it replaces the IOLoop (like Twisted's reactor) with a simpler system, eventlet.
I'm glad you think it's hip! It's only a couple months old.
There are many python frameworks and each serves a different need. If you are a Tornado user but find the blocking / nonblocking paradigm confusing, you would probably get a lot out of Brubeck because it replaces the IOLoop (like Twisted's reactor) with a simpler system, eventlet.
I'm glad you think it's hip! It's only a couple months old.
Agreed. For a newcommer, the question "which framework is most worth my time investment" has has become rather daunting. Seems like the safest choices are the big ones like Rails or Django, but there always seems to be something tempting and fun on the horizon.
"Spending too much Time with the Choice of Framework
This should probably go to the top. If you have a small application (say less than 10.000 lines of code) the framework probably isn't your problem anyways. And if you have more code than that, it's still not that hard to switch systems when you really have to. In fact even switching out core components like an ORM is possible and achievable if you write a little shim and get rid of that step by step. Better spend your time making the system better. The framework choice used to be a lot harder when the systems were incompatible. But this clearly no longer is the case."
http://lucumr.pocoo.org/2010/12/24/common-mistakes-as-web-de...
This should probably go to the top. If you have a small application (say less than 10.000 lines of code) the framework probably isn't your problem anyways. And if you have more code than that, it's still not that hard to switch systems when you really have to. In fact even switching out core components like an ORM is possible and achievable if you write a little shim and get rid of that step by step. Better spend your time making the system better. The framework choice used to be a lot harder when the systems were incompatible. But this clearly no longer is the case."
http://lucumr.pocoo.org/2010/12/24/common-mistakes-as-web-de...
That is a goal of Brubeck too. By using DictShield for modeling you get all your modeling needs, including serializing to Python dictionaries or JSON strings, but it doesn't have an opinion on what database you us.
Your database layer might change, for whatever reason, so you adjust a few queries for loading data and you're back in action.
In addition to this many people haphazardly choose their framework and make the mistake of using Tornado without understanding how nonblocking I/O works. If a user chooses.
Brubeck, they would get nonblocking support for free, which makes it a considerably safer haphazard choice.
Your database layer might change, for whatever reason, so you adjust a few queries for loading data and you're back in action.
In addition to this many people haphazardly choose their framework and make the mistake of using Tornado without understanding how nonblocking I/O works. If a user chooses.
Brubeck, they would get nonblocking support for free, which makes it a considerably safer haphazard choice.
Is there a Socket.IO interface, such as Tornadio (https://github.com/MrJoes/tornadio) for Tornado?
try gevent-socketio
I think it's unfortunate that the power of choice isn't as appreciated outside the Python world but I can also relate to people who feel they have to read way too much stuff.
In spite of this, however, Brubeck indeed aims to be a one-stop shop. It uses coroutines + nonblocking I/O to make scaling up easy but it also a web.py style routing system because all of us know this style already.
In spite of this, however, Brubeck indeed aims to be a one-stop shop. It uses coroutines + nonblocking I/O to make scaling up easy but it also a web.py style routing system because all of us know this style already.
Choice is great for veterans, but for someone learning Python or Ruby, how do they know what to pick? How to they know that they're not going to bet on a losing horse that stops being actively developed after a while and doesn't do anything for their career. Veterans can afford to play around and test out new frameworks, because they can always fall back to Django or whatever pays the bills, but newbies can't.
They have to do their homework, ask around, and make the best possible guess. And more importantly, focus on the more general skills than the specific framework.
If someone is getting started in a vacuum, they're doing it wrong.
If someone is getting started in a vacuum, they're doing it wrong.
Also consider that some frameworks (ala Django) go for the 'kitchen sink' approach which is better for newcomers. Others (like Express with NodeJS) just do a few minimal helpers and otherwise get out of the way.
In this case, the plethora of frameworks is an advantage for the advanced developer, while the few big obvious ones stand out for newcomers.
In this case, the plethora of frameworks is an advantage for the advanced developer, while the few big obvious ones stand out for newcomers.
It's the little things that count and I like that out of the box bcrypt is being used in auth.py.
I never got around to playing around with Mongrel2 but always wanted to.
I'm a huge Flask fan and will most likely (after looking through some of the source) be using Brubeck from now on on the Py side of things. It covers the basics very well.
Bravo!
Btw, equally impressive is DictShield.
Will you create any special cases for 'modules/extentions' or will continue to keep things agnostic?
I never got around to playing around with Mongrel2 but always wanted to.
I'm a huge Flask fan and will most likely (after looking through some of the source) be using Brubeck from now on on the Py side of things. It covers the basics very well.
Bravo!
Btw, equally impressive is DictShield.
Will you create any special cases for 'modules/extentions' or will continue to keep things agnostic?
Armin Ronacher—author of Flask and Werkzeug—pointed out earlier on Twitter[1] that (for better or for worse) it doesn't support WSGI.
[1] http://twitter.theinfo.org/92202624614539264
[1] http://twitter.theinfo.org/92202624614539264
The difference here is that Brubeck answers ZeroMQ messages from Mongrel2 instead of being a web server. So instead of using WSGI, Brubeck connects to Mongrel2 via a ZeroMQ socket and reads messages representing the web requests.
Beyond that I attempted to create a familiar interface for everyone who's used a pythonic web framework before. Indeed, a lot like Flask, but also like Tornado.
* Tornado style routing: https://github.com/j2labs/brubeck/blob/master/demos/demo_min...
* Flask style routing: https://github.com/j2labs/brubeck/blob/master/demos/demo_noc...
Beyond that I attempted to create a familiar interface for everyone who's used a pythonic web framework before. Indeed, a lot like Flask, but also like Tornado.
* Tornado style routing: https://github.com/j2labs/brubeck/blob/master/demos/demo_min...
* Flask style routing: https://github.com/j2labs/brubeck/blob/master/demos/demo_noc...
Its relatively difficult to find maintained Mongrel2 Python libraries, but projects like http://wsgid.com/ , https://github.com/rfk/m2wsgi , and (maybe unmaintained) https://github.com/berry/Mongrel2-WSGI-Handler make it fairly easy to run WSGI apps behind Mongrel2.
I wouldn't use Brubeck without WSGI solely because I don't want to write an app dependent on its web server.
I wouldn't use Brubeck without WSGI solely because I don't want to write an app dependent on its web server.
That's a fair point, but there's a lot to love about Mongrel2!
Agreed. If you limit your application to WSGI then mongrel2 is "just" another high-performance event-driven webserver. To really take advantage of the advanced features of m2 you need a framework that's more closely tied to its model of the world.
(speaking as author of the m2wsgi module linked above)
(speaking as author of the m2wsgi module linked above)
I haven't done anything significant with Mongrel2 yet, so my mental model of its operation is probably somewhat flawed, but I was thinking about my above post for a while yesterday.
It seems like WSGI supports a subset of Mongrel2 features. Any "pure" WSGI app can run fine behind Mongrel2, but Mongrel2 allows much more than the WSGI model. For example, a response in Mongrel2 is just a socket getting written to. Mongrel2 can send a request to server A, and then receive the response from a completely seperate server B. Can WSGI support something like this? As far as I know, someone would need to reimplement a lot of Mongrel2's functionality in Python (which is idiotic) just for WSGI compliance.
Again, I've yet to build anything with Mongrel2 and I haven't read the WSGI PEP in a while, so please correct me if I'm wrong about anything. After thinking about it yesterday, I find the differences between a more traditional WSGI-esque model and something like Mongrel2 to be fascinating.
It seems like WSGI supports a subset of Mongrel2 features. Any "pure" WSGI app can run fine behind Mongrel2, but Mongrel2 allows much more than the WSGI model. For example, a response in Mongrel2 is just a socket getting written to. Mongrel2 can send a request to server A, and then receive the response from a completely seperate server B. Can WSGI support something like this? As far as I know, someone would need to reimplement a lot of Mongrel2's functionality in Python (which is idiotic) just for WSGI compliance.
Again, I've yet to build anything with Mongrel2 and I haven't read the WSGI PEP in a while, so please correct me if I'm wrong about anything. After thinking about it yesterday, I find the differences between a more traditional WSGI-esque model and something like Mongrel2 to be fascinating.
It's for the better. WSGI is a piece of crap. Try getting a decent stack trace out of it's weird "we hate callbacks so we did nested function calls instead of lists" design.
I don't understand what you're saying. What do you mean by "nested function calls," and how is that antithetical to both lists and callbacks? In what way is WSGI anti-callback?
WSGI is fine if you're building short-lived, simple request/response style web apps but if you want to do any sort of async messaging, long-polling, or use websockets i.e. a modern web app - then it leaves a lot to be desired. Mongrel2/Brubeck appear to be designed with these considerations in mind.
Love the whole idea, lightweight, fast, smartly designed. Great work.
Looks very promising. I'm getting the impression that it has a better architecture than Django; I think that if and when it will match Django in level of finish, documentation, community, tools, etc., it might become a serious competitor.
Man, I haven't seen such an efficient and clean code / concepts for quite a while.
I must give it a try at the next new web project.
Having built-in support for Tornado templates will make it much easier to transform an existing one in the future.
Thank you j2lab for sharing!
I must give it a try at the next new web project.
Having built-in support for Tornado templates will make it much easier to transform an existing one in the future.
Thank you j2lab for sharing!
How does this compares to Zed's own Tir?
TIR == LUA
Brubeck == PythonOT nitpick: it's spelled "Lua". It's not an acronym.
How baked is this? Has anyone written any apps on it?
The project is still new but I have written a complete example app and have been building a far more complex app since.
Listsurf: https://github.com/j2labs/listsurf
The more complex app implements a full social network on top of Brubeck. I think it will be called Rubik, but I'm not sure yet.
It'll pop up on my github page sooner or later though: https://github.com/j2labs
Listsurf: https://github.com/j2labs/listsurf
The more complex app implements a full social network on top of Brubeck. I think it will be called Rubik, but I'm not sure yet.
It'll pop up on my github page sooner or later though: https://github.com/j2labs
I get that it follows the theme of Django and Sinatra. But Brubeck is not a very common name (the musician Dave Brubeck is my second cousin) and I don't normally see the musical Brubecks intrude into "my" domain. :)