Twisted Python's lead hacker discusses Tornado(glyph.twistedmatrix.com)
glyph.twistedmatrix.com
Twisted Python's lead hacker discusses Tornado
http://glyph.twistedmatrix.com/2009/09/what-i-wish-tornado-were.html
17 comments
I'd be pretty damn upset too if some other project got a lot of press for an announcement which included comments undermining the quality of my project. Sorry, but if you're going to strongly imply that my project's ease of use and performance are subpar, them's fighting words.
The comments were really in passing. 'Looked at it, wasn't for us', more or less. Perhaps them's fightin' words. But getting huffy about them probably helps your project even less, given they'd have otherwise gone largely unnoticed.
And not responding to disparaging comments helps your project?
I think he makes a pretty strong case in this article.
It's a bit passive aggressive though, I.e. "hey I don't have any sore feelings about it, but they went about it like real jerks". Just come out and say they were real jerks without the "I don't have sore feelings" crap. We can tell you didn't like it, and I'm completely OK with that.
I think the mindset that created Tornado is one that a lot of us share as coders. If there's an existing framework to do something but it's messy in your opinion, and the job you need to do is simple enough and not covered very well by that existing framework, well then often it costs a lot less in terms of investment to build it yourself. Then you get exactly what you need too, and nothing more (which increases complexity). It worked out for them too!
I also think twisted could market itself better, but that takes effort that could be spent elsewhere, and I doubt they're all that interested. We'll see.
I think he makes a pretty strong case in this article.
It's a bit passive aggressive though, I.e. "hey I don't have any sore feelings about it, but they went about it like real jerks". Just come out and say they were real jerks without the "I don't have sore feelings" crap. We can tell you didn't like it, and I'm completely OK with that.
I think the mindset that created Tornado is one that a lot of us share as coders. If there's an existing framework to do something but it's messy in your opinion, and the job you need to do is simple enough and not covered very well by that existing framework, well then often it costs a lot less in terms of investment to build it yourself. Then you get exactly what you need too, and nothing more (which increases complexity). It worked out for them too!
I also think twisted could market itself better, but that takes effort that could be spent elsewhere, and I doubt they're all that interested. We'll see.
Exactly. Exactly. Exactly.
And not everything is a cold impersonal engineering decision. Programmers are people (really! we are!) and people like greenfields and blank canvases and fresh starts.
And I'd also submit that a team of talented developers almost certainly will make a superior product if they have the ability to use the original as a reference and resource. The saying "Pioneers get the arrows, settlers get the land" comes to mind.
And not everything is a cold impersonal engineering decision. Programmers are people (really! we are!) and people like greenfields and blank canvases and fresh starts.
And I'd also submit that a team of talented developers almost certainly will make a superior product if they have the ability to use the original as a reference and resource. The saying "Pioneers get the arrows, settlers get the land" comes to mind.
Agreed. Greenfield projects are far more enjoyable, and you tend to learn more. Obviously we can't always do that and succeed, but they did well enough in this case. Preferably they'd go easier on the remarks about existing open source frameworks though.
"In general, it seems like Twisted is full of demo-quality stuff, but most of the protocols have tons of bugs."
http://bret.appspot.com/entry/tornado-web-server#comment-163...
That's not "wasn't for us" that's a very vague comment insulting the project. If that were true, I think it is fair to at least describe what they were -- or the nature of them -- or something.
http://bret.appspot.com/entry/tornado-web-server#comment-163...
That's not "wasn't for us" that's a very vague comment insulting the project. If that were true, I think it is fair to at least describe what they were -- or the nature of them -- or something.
Ok, but did you click thru to that listserv thread here https://garage.maemo.org/pipermail/brisa-develop/2009-August... ?
I'm not familiar with the project or this guy in particular, but man, he seems awfully abrasive. Sarcastic and bitter came to mind as I read them. I have no dog in this fight, and one could fairly assume that these 2 discussions are just not his best moments. But from what I've seen...
I'm not familiar with the project or this guy in particular, but man, he seems awfully abrasive. Sarcastic and bitter came to mind as I read them. I have no dog in this fight, and one could fairly assume that these 2 discussions are just not his best moments. But from what I've seen...
The three emails I can read by clicking through to that link don't seem overly abrasive.
Ok, but there are products, and there are products. This is a product that you can hack to your heart's content, not something sitting on a store shelf that you can take or leave. Oh, yeah, and you get it completely for free. And with a license that you can base your N-million dollar company on it with no ifs ands or buts. And could potentially save you thousands of dollars worth of work. And with developers willing to contribute their own time to make it work better for you. And yet you can't write an email saying "hi, this almost works for us, but has a few issues I'd like to talk about (and potentially help fix)".
The assumption isn't that they owe you feedback: the assumption is that it was not in their best interest to create another instance of something that already exists. That assumption is supported by the fact that Tornado has the exact same goal as Twisted. Effectively, it is a rewrite, especially if they have obviously looked closely at Twisted's code for inspiration.
A CTO wasted money here.
A CTO wasted money here.
No, a CTO didn't waste money; he got exactly what he needed, cleaner, simpler. No - it doesn't handle every single edge case, nuance, platform - but it does exactly what they needed. They didn't need to spend time patching and grokking twisted, or needing to deal with any rough edges to ramping personnel up on it. They had a job to do, and they did it.
Now that it's out in the wild, maybe someone will cross-pollinate, maybe changes will go in both projects. But the good thing is that they got their job done.
Now that it's out in the wild, maybe someone will cross-pollinate, maybe changes will go in both projects. But the good thing is that they got their job done.
I made tornado work on top of twisted by removing 1,389 lines of code and adding 92 (and I believe most of that was adding compatibility wrappers to keep the API mostly sane).
My effort is not done -- I do have a lot more code to remove. Where it is now, with having mostly just removed code, I can do far more than tornado can do out of the box, and I can do it starting with the same API. I did not modify twisted.
And seriously, these are all insanely smart guys. If they have to "ramp people up", they have to do that with their new framework anyway, but I can't believe that if I can understand twisted, they can't.
My effort is not done -- I do have a lot more code to remove. Where it is now, with having mostly just removed code, I can do far more than tornado can do out of the box, and I can do it starting with the same API. I did not modify twisted.
And seriously, these are all insanely smart guys. If they have to "ramp people up", they have to do that with their new framework anyway, but I can't believe that if I can understand twisted, they can't.
I haven't yet looked at tornado, and it's been years since I seriously investigated twisted. That said, even though it's less code to use twisted, not counting twisted, it's going to be quite a lot more after counting twisted, most of which, as you note with "can do far more than tornado can", they didn't want in the first place.
When there's a bug in tornado as it is, they can just fix it. If there was a bug in tornado-on-twisted, and if the bug were in twisted, they'd either have to wait until the twisted maintainers fixed it, or fix it themselves. If themselves, then they end up maintaining the internals of twisted (at least for the time between fix and having the patch accepted), which means they have to keep track of and understand a lot of code that, as you note, they don't have any use for in the first place! In that situation, I might choose to write my own as well.
When there's a bug in tornado as it is, they can just fix it. If there was a bug in tornado-on-twisted, and if the bug were in twisted, they'd either have to wait until the twisted maintainers fixed it, or fix it themselves. If themselves, then they end up maintaining the internals of twisted (at least for the time between fix and having the patch accepted), which means they have to keep track of and understand a lot of code that, as you note, they don't have any use for in the first place! In that situation, I might choose to write my own as well.
How does tornado on twisted perform relative to tornado?
Interfacing with an another team to solve problems in a framework they didn't like (even if just aesthetically)
I think dislike for aesthetic reasons is one of the best reasons to do your own rewrite. I can imagine that proposing radical changes to e.g. API style of an existing FOSS project would be the most difficult up-hill battle.
I think dislike for aesthetic reasons is one of the best reasons to do your own rewrite. I can imagine that proposing radical changes to e.g. API style of an existing FOSS project would be the most difficult up-hill battle.
If you accept the validity of the cathedral vs. bazaar premise, then the pattern of OSS fragmentation ultimately works against the general community's interests.
On the other hand, competition in the course of establishing best of breed constructs for emerging paradigms is healthy, as a preliminary stage before community building per Raymond's model.
On the other hand, competition in the course of establishing best of breed constructs for emerging paradigms is healthy, as a preliminary stage before community building per Raymond's model.
I like the idea of the performance and simplicity of Tornado, as I primarily use web.py. After reading this blog entry I decided to give Twisted another look. It took me about 2 minutes to get to the info about the web module and the examples were not very clear. I've walked away wondering a few things:
If I use twisted, should I use web or web2? How do I map URI's to functions?
When I go to the tornado homepage, I scroll and in < 1 minute I see how I would code for it and can be off and running. To develop web apps, I shouldn't have to spend time figuring out an async network communications framework, I should be worried about programming my app.
My perception from the Twisted site is that I would need to spend time figuring out the twisted way of doing things, then figure out the web add-on way of doing things, then worry about my app. With Tornado, I'm very close to a drop in replacement for web.py and I can be off to the races.
If I use twisted, should I use web or web2? How do I map URI's to functions?
When I go to the tornado homepage, I scroll and in < 1 minute I see how I would code for it and can be off and running. To develop web apps, I shouldn't have to spend time figuring out an async network communications framework, I should be worried about programming my app.
My perception from the Twisted site is that I would need to spend time figuring out the twisted way of doing things, then figure out the web add-on way of doing things, then worry about my app. With Tornado, I'm very close to a drop in replacement for web.py and I can be off to the races.
Your perception of twisted may be as it is because you can do SO MUCH with twisted. If it's a web.py replacement you want, it doesn't add complexity.
Take a look at the diff to go from a friendfeed ioloop to a twisted reactor loop (with my adaptor) in tornado:
http://pastebin.com/f707e5ae9
You replace two imports with two other imports, and three lines of startup code with three other lines of startup code. Your app doesn't change.
Take a look at the diff to go from a friendfeed ioloop to a twisted reactor loop (with my adaptor) in tornado:
http://pastebin.com/f707e5ae9
You replace two imports with two other imports, and three lines of startup code with three other lines of startup code. Your app doesn't change.
I was eyeing up the code earlier for this. This is great and is a good counter for my argument about Tornado's near drop in replacement. However, that's not my main point.
Ultimately, I think Twisted, due to the fact it does so much, is intimidating. The documentation lacks succinct examples of how to implement the very basic things people want to implement. The documentation to figure out how to make an app is buried under at least 3 clicks and even then isn't clear. The comments, when there are comments, in the examples don't truly explain the why of the actions just that they are there.
If twisted wants to be considered as a serious contender as foundation to develop web apps on, they need to take a look at how important very good documentation is to Django's success or how the simple, uncommented example at http://www.tornadoweb.org/ is more clear than the first bit of web example code at twisted: http://twistedmatrix.com/projects/web/documentation/examples....
I'm not knocking twisted, I'm trying to explain why when I went around evaluating what I would use in my projects, I did not choose Twisted. If you find my decisions and observations valid or not is up to you.
Ultimately, I think Twisted, due to the fact it does so much, is intimidating. The documentation lacks succinct examples of how to implement the very basic things people want to implement. The documentation to figure out how to make an app is buried under at least 3 clicks and even then isn't clear. The comments, when there are comments, in the examples don't truly explain the why of the actions just that they are there.
If twisted wants to be considered as a serious contender as foundation to develop web apps on, they need to take a look at how important very good documentation is to Django's success or how the simple, uncommented example at http://www.tornadoweb.org/ is more clear than the first bit of web example code at twisted: http://twistedmatrix.com/projects/web/documentation/examples....
I'm not knocking twisted, I'm trying to explain why when I went around evaluating what I would use in my projects, I did not choose Twisted. If you find my decisions and observations valid or not is up to you.
I'm not reading anything you're saying as being negative towards twisted.
The thing, is twisted isn't a web framework, so nobody should find it surprising that it doesn't make it really easy and obvious how one would go about making a web site with it.
A lot of people nowadays seem to look at a programming language and think "yeah, but where do my HTML templates go?" Most of the things I've built in twisted don't have web interfaces. There's a guy in the list now trying to figure out how to get his IRC MUC sending out ICQ alerts through OSCAR -- now HOWTO there, either. :)
This feels more like a python vs. ruby discussion than anything else to me. One has a community that focused really hard on making a very rich web framework and the other can do web stuff, but isn't quite there. In both camps, you have extremists who can't see why anyone would use the other. In both cases, they can be better. :)
So I'm hoping others will contribute to making the twisted web story better.
The thing, is twisted isn't a web framework, so nobody should find it surprising that it doesn't make it really easy and obvious how one would go about making a web site with it.
A lot of people nowadays seem to look at a programming language and think "yeah, but where do my HTML templates go?" Most of the things I've built in twisted don't have web interfaces. There's a guy in the list now trying to figure out how to get his IRC MUC sending out ICQ alerts through OSCAR -- now HOWTO there, either. :)
This feels more like a python vs. ruby discussion than anything else to me. One has a community that focused really hard on making a very rich web framework and the other can do web stuff, but isn't quite there. In both camps, you have extremists who can't see why anyone would use the other. In both cases, they can be better. :)
So I'm hoping others will contribute to making the twisted web story better.
Which is the same as saying it's a failure of documentation and explanation.
Is the code for tornado.twister available anywhere?
Yes, Dustin forked it on github. You can read about it here:
http://dustin.github.com/2009/09/12/tornado.html
tornado.twister here: http://github.com/dustin/tornado/blob/205a13b3c0afac843abdeb...
tornado.twister here: http://github.com/dustin/tornado/blob/205a13b3c0afac843abdeb...
You might be missing the point. Had FriendFeed worked on improving Twisted and it's documentation to make it easier to use, they would have achieved what they needed in less time and would have avoided fragmenting the resources of the open source community among two projects.
I do get that point, but to that end, perhaps they made their decision, in part, based upon the same type of thought process.
This is progress, and as a result, the Twisted people, if they're listening, have gotten valuable feedback out of the process from many people other than myself as a result.
There has also been much more about Twisted on places like Hacker News than I've seen, relatively speaking. With Dustin porting the meat of Tornado on top of Twisted's IO handler, there is much more interesting things to come out of it all.
It is in no way a negative thing for Twisted for there to be competition in an area of cross-over such as this.
This is progress, and as a result, the Twisted people, if they're listening, have gotten valuable feedback out of the process from many people other than myself as a result.
There has also been much more about Twisted on places like Hacker News than I've seen, relatively speaking. With Dustin porting the meat of Tornado on top of Twisted's IO handler, there is much more interesting things to come out of it all.
It is in no way a negative thing for Twisted for there to be competition in an area of cross-over such as this.
I think you underestimate the effort it would take to write concise, accurate documentation for a project the size of twisted. I bet it took far less time to write the 5000 lines of code in Tornado than it would to rewrite the documentation for Twisted and its web modules, and get that accepted and integrated upstream. And it wouldn't fix the fact that Twisted has all sorts of functionality that Tornado developers don't need, aren't interested in documenting, and that web developers would have to either avoid or wade through.
An observation: Python coders seem to believe that when someone has solved a problem in Python and open sourced that problem every Python coder must use their solution or else they are somehow doing an injustice, perhaps even undermining the community!
On the contrary, 1) Friendfeed can do whatever they want. There is no part of the Python license that says you must use Twisted or else. 2) I doubt FF's statements really caused damage to Twisted's user base, just as we use Cherrypy at work and didn't drop it on Thursday because of Tornado. 3) All open source contributions are a Good Thing. If Twisted devs are really feeling screwed FF is basically saying "here's how we did it, use it if you want". Sounds pretty win win to me.
Instead of complaining and accusing I would suggest Twisted devs take the publicity opportunity to show why Twisted is still great, benchmarks, what the do better than Tornado and what they can learn from it.
On the contrary, 1) Friendfeed can do whatever they want. There is no part of the Python license that says you must use Twisted or else. 2) I doubt FF's statements really caused damage to Twisted's user base, just as we use Cherrypy at work and didn't drop it on Thursday because of Tornado. 3) All open source contributions are a Good Thing. If Twisted devs are really feeling screwed FF is basically saying "here's how we did it, use it if you want". Sounds pretty win win to me.
Instead of complaining and accusing I would suggest Twisted devs take the publicity opportunity to show why Twisted is still great, benchmarks, what the do better than Tornado and what they can learn from it.
An observation: Python coders seem to believe
I doubt FF's statements really caused damage to Twisted's user base
Your suggestion in the last paragraph might cause benefit.
General comment on difficulties learning twisted - I found it difficult to grasp until I'd practically reimplemented the patterns by accident in the course of writing my own client/server application. I got stuck on a concept at one point in my non-twisted socket application, and the guys on #twisted helped me past it. For those who are interested, the O'Reilly book is obsolete but a better starting point than the online docs.
That's a poor generalisation, and I assume you've built it on the back of a single observation.
On the contrary, my observation is based on comments I've read in many Python community forums, especially when applied to Python vs (your other high-level language of choice). For example, I've read many comments to the effect of "I like Python over (Ruby, Perl, etc) because it promotes one right way of doing things and having one package for handling problem X." I am not stating that this belief is a Python language requirement, nor do I believe that this observation adheres to all Python coders so yes, it is a generalization. But my generalization is certainly not based on a single observation. Finally, I will gladly play the devil's advocate and agree that there is little way to prove this generalization, short of spending several days collecting emails on Python mailing lists, which is why it is offered as an observation only. It is an observation of the practices of a community not something codified into a PEP, at least as far as I know based on my limited research.
I completely agree with the rest of your comment and value your critique. Python is a great language that provides a diverse set of tools, which is why I also choose it to handle many tasks.
On the contrary, my observation is based on comments I've read in many Python community forums, especially when applied to Python vs (your other high-level language of choice). For example, I've read many comments to the effect of "I like Python over (Ruby, Perl, etc) because it promotes one right way of doing things and having one package for handling problem X." I am not stating that this belief is a Python language requirement, nor do I believe that this observation adheres to all Python coders so yes, it is a generalization. But my generalization is certainly not based on a single observation. Finally, I will gladly play the devil's advocate and agree that there is little way to prove this generalization, short of spending several days collecting emails on Python mailing lists, which is why it is offered as an observation only. It is an observation of the practices of a community not something codified into a PEP, at least as far as I know based on my limited research.
I completely agree with the rest of your comment and value your critique. Python is a great language that provides a diverse set of tools, which is why I also choose it to handle many tasks.
The great thing about open source is that people are free to share ideas and code. If the Twisted team likes parts of Tornado, they are free to replicate those parts back into Twisted while ignoring the parts that they don't like.
Meanwhile, users are free to choose whichever approach suits them better. Twisted is no doubt more complete in terms of protocols and such, but many people also value simplicity. One important point that I think has been overlooked here is that the core of Tornado is only a few hundred lines of code. For me, the fact that I can actually read _all_ of the code in just a few minutes is very valuable. I'm sure many others don't care though, which is why it's great that we live in a world where there doesn't always have to be a single "right" answer.
Meanwhile, users are free to choose whichever approach suits them better. Twisted is no doubt more complete in terms of protocols and such, but many people also value simplicity. One important point that I think has been overlooked here is that the core of Tornado is only a few hundred lines of code. For me, the fact that I can actually read _all_ of the code in just a few minutes is very valuable. I'm sure many others don't care though, which is why it's great that we live in a world where there doesn't always have to be a single "right" answer.
"Twisted is no doubt more complete in terms of protocols and such, but many people also value simplicity. One important point that I think has been overlooked here is that the core of Tornado is only a few hundred lines of code."
I think this is a bit misleading. The twisted port involved removing almost 1,400 lines of code and didn't change the tornado API. That is, tornado is not more simple without twisted.
I do understand the value of being able to read all of the code quickly, but there are other turtles below these. For me, it's very valuable to know that the part upon which I'm building is a well-tested codebase whose semantics I understand, and for which I can write tests when things begin to violate my understanding.
If tornado is used in more applications, it will have to grow in complexity to meet the needs of those applications. I see tornado as having two faces. The web portions can certainly continue to grow, but the network IO handing just means that people will have to port over things that already exist within twisted, and expand it in complexity (such as the kqueue work that is in progress) to handle more low-level stuff.
At some point, you can't just easily sit down and read all of it (at least, not any more easily than you can with twisted).
I think this is a bit misleading. The twisted port involved removing almost 1,400 lines of code and didn't change the tornado API. That is, tornado is not more simple without twisted.
I do understand the value of being able to read all of the code quickly, but there are other turtles below these. For me, it's very valuable to know that the part upon which I'm building is a well-tested codebase whose semantics I understand, and for which I can write tests when things begin to violate my understanding.
If tornado is used in more applications, it will have to grow in complexity to meet the needs of those applications. I see tornado as having two faces. The web portions can certainly continue to grow, but the network IO handing just means that people will have to port over things that already exist within twisted, and expand it in complexity (such as the kqueue work that is in progress) to handle more low-level stuff.
At some point, you can't just easily sit down and read all of it (at least, not any more easily than you can with twisted).
Tornado relies on a small number of well defined system calls (such as epoll). For me, the difference between reading (and debugging) a few hundred lines of code down to the system call level, and reading (and debugging) a big library like Twisted is rather significant. For you, the difference may not be important, and that's fine. I also don't think that it's necessary for Tornado to replicate all the functionality of Twisted -- if people want Twisted, they should simply use Twisted and ignore Tornado (and hopefully the Twisted team will have borrowed the best bits of Tornado anyway).
That is, tornado is not more simple without twisted.
That's not exactly what I'd say. :) Tornado without twisted is simpler than tornado with twisted, because you can't ignore the huge system you'd otherwise be able to avoid.
Edit to remove mild flame.
That's not exactly what I'd say. :) Tornado without twisted is simpler than tornado with twisted, because you can't ignore the huge system you'd otherwise be able to avoid.
Edit to remove mild flame.
I get what you're saying, but twisted does all that stuff with a great deal of testing.
It's the same reason smart companies are compelled to contribute changes to open source projects. That's a lot of maintenance you get for free.
That's why friendfeed didn't do their own python implementation on top of their own posix operating system to make sure they had a firm grasp on all the parts. A working and well-tested component is almost always better than writing a new one from scratch that you have to deal with yourself.
It's the same reason smart companies are compelled to contribute changes to open source projects. That's a lot of maintenance you get for free.
That's why friendfeed didn't do their own python implementation on top of their own posix operating system to make sure they had a firm grasp on all the parts. A working and well-tested component is almost always better than writing a new one from scratch that you have to deal with yourself.
Your point is fair, but I think you may be dramatically overestimating the amount of time that was spent writing and debugging Tornado. Overall, I'd estimate that less than 1% of our time was spent on Tornado and other similar "reinventions", and that in exchange we got simple systems that we understand and that work the way we like, which seems like a pretty good deal to me.
Oh, I don't mean to be perpetuating the argument. I think you guys did awesome stuff. Most of my twisted work doesn't even have a web interface, some of it does, but it's fine. Some could really use some help. Tornado seems to be a great answer there.
I suppose I'm just letting you know how you can better serve my personal needs next time you invent something great. :)
I suppose I'm just letting you know how you can better serve my personal needs next time you invent something great. :)
"The twisted port involved removing almost 1,400 lines of code and didn't change the tornado API."
Or, looking at it another way, you removed Tornado's simple 1400-line core, and replaced it with a 150,000-line dependency.
(Yes, I know that Tornado On Twisted doesn't depend on the entire Twisted framework. But the port does involve a lot more code in total than the original.)
Or, looking at it another way, you removed Tornado's simple 1400-line core, and replaced it with a 150,000-line dependency.
(Yes, I know that Tornado On Twisted doesn't depend on the entire Twisted framework. But the port does involve a lot more code in total than the original.)
He makes excellent points, but let's compare the two scenarios for a moment.
A) Helping out Twisted, they'd have gotten:
* Kudos from the Twisted community for their contributions to the project;
* Less code required.
B) Creating Tornado, they got:
* To avoid other people's code they didn't like in the first place.
* All the credit. They are the authors of the framework, not just contributors;
* A fresh start. A project designed to their needs;
* Ability to lead the project in the direction they like;
* No stigmas attached (e.g., "Twisted is too complex");
* Major kudos from a large community of developers interested in "real-time" web services and applications;
* Tons of free publicity to the project and FriendFeed/Facebook's open source initiatives.
A) Helping out Twisted, they'd have gotten:
* Kudos from the Twisted community for their contributions to the project;
* Less code required.
B) Creating Tornado, they got:
* To avoid other people's code they didn't like in the first place.
* All the credit. They are the authors of the framework, not just contributors;
* A fresh start. A project designed to their needs;
* Ability to lead the project in the direction they like;
* No stigmas attached (e.g., "Twisted is too complex");
* Major kudos from a large community of developers interested in "real-time" web services and applications;
* Tons of free publicity to the project and FriendFeed/Facebook's open source initiatives.
This is deeply silly. More fame and glory? So one day, Bret Taylor leaned back, rubbed his hands and whispered to himself 'Yesss, I shall write Tornado for fame and glory. Before I start I must briefly consider how I'll keep all the screaming teenage fans away and what to wear at the ticker-tape parade'. Or perhaps he just wanted to solve a problem he and his team had. And having done that, was nice enough to open source the solution.
I removed my "fame and glory" comment as it was misleading and didn't truly represent what I think. I'm not arguing that he/they premeditated it and did it for the glory. Their priority was getting FF done, undoubtedly. However, there are concrete advantages (and visibility) deriving from their choice.
To avoid other people's code they didn't like in the first place.
'Not Invented Here' syndrome. A bad reason.
A fresh start. A project designed to their needs
That's shortsighted. Maybe the short term goals had different accents, but in the long run, Twisted and Tornado have the exact same goals.
Major kudos from a large community of developers interested in "real-time" web services and applications
For rewriting something that already exists and not doing an significantly better job and arguably a worse job? Applying those resources to Twisted would also have yielded this result.
Tons of free publicity to the project and FriendFeed/Facebook's open source initiatives.
Also belongs under A. Large contributions to OSS would yield that same effect.
A fresh start. A project designed to their needs Ability to lead the project in the direction they like
You are assuming, without any reason, that they wouldn't have had that chance with Twisted. If that wouldn't work out, they could easily have forked Twisted, effectively taking over the project.
'Not Invented Here' syndrome. A bad reason.
A fresh start. A project designed to their needs
That's shortsighted. Maybe the short term goals had different accents, but in the long run, Twisted and Tornado have the exact same goals.
Major kudos from a large community of developers interested in "real-time" web services and applications
For rewriting something that already exists and not doing an significantly better job and arguably a worse job? Applying those resources to Twisted would also have yielded this result.
Tons of free publicity to the project and FriendFeed/Facebook's open source initiatives.
Also belongs under A. Large contributions to OSS would yield that same effect.
A fresh start. A project designed to their needs Ability to lead the project in the direction they like
You are assuming, without any reason, that they wouldn't have had that chance with Twisted. If that wouldn't work out, they could easily have forked Twisted, effectively taking over the project.
"'Not Invented Here' syndrome. A bad reason."
That seems a little strawman-ish, honestly. On the contrary, they embraced the technology and architecture of the app.
What's wrong with looking at something like this.. an important infrastructure piece... liking the idioms and architecture but having issues and thinking .... We have a benefit the Twisted guys didn't have... we have prior art... a map of this territory ?
I don't think anybody here has a strong conviction that FriendFeed was _right_. But you and others seem to have a strong convication that they were wrong, and all I think....and I think others here too.... all I think is that it is silly to attack them and be so certain they were wrong.
These are talented developers. Smart guys. With a little bit of business talent, too.. All of us have been in complex project cycles. Who are you or anybody to pick them off like they somehow caused somebody harm? Why the intensity? What gives?
That seems a little strawman-ish, honestly. On the contrary, they embraced the technology and architecture of the app.
What's wrong with looking at something like this.. an important infrastructure piece... liking the idioms and architecture but having issues and thinking .... We have a benefit the Twisted guys didn't have... we have prior art... a map of this territory ?
I don't think anybody here has a strong conviction that FriendFeed was _right_. But you and others seem to have a strong convication that they were wrong, and all I think....and I think others here too.... all I think is that it is silly to attack them and be so certain they were wrong.
These are talented developers. Smart guys. With a little bit of business talent, too.. All of us have been in complex project cycles. Who are you or anybody to pick them off like they somehow caused somebody harm? Why the intensity? What gives?
When the invention in question is core to your product, sometimes it does make sense to do it yourself. See Joel Spolsky's essay "In Defense of Not-Invented-Here Syndrome": http://www.joelonsoftware.com/articles/fog0000000007.html
> That's shortsighted. Maybe the short term goals had different accents, but in the long run, Twisted and Tornado have the exact same goals.
Actually they don't. Twisted is "an event-driven networking engine written in Python"
Tornado is "an open source version of the scalable, non-blocking web server and tools that power FriendFeed"
Note that the scope of these things are completely different.
Actually they don't. Twisted is "an event-driven networking engine written in Python"
Tornado is "an open source version of the scalable, non-blocking web server and tools that power FriendFeed"
Note that the scope of these things are completely different.
If you don't like someone's code, and the time cost of communication with them to change the code upstream is likely to stop you from meeting deadline (or they have a different final vision), creating something tailored to your own needs is the only solution.
Twisted has always been an "emperor has no clothes" project to me, a project that's no doubt very useful but at the same time seems to revel in its unnecessary complexity (just read its documentation - how many times do you really have to tell us that something might seem overly complicated but is worth it for its flexibility?) I'm honestly glad to see a no-bullshit alternative that doesn't have that same sense of worshipping complexity as a sign of quality.
That viewpoint seems quite narrow to me.
The documentation of Twisted attempts to introduce several event-driven programming patterns. Most people have a 'wtf' moment or two when they come across design patterns that approach things seemingly backwards.
The further I got into my Twisted app, the more I found myself reaching for things and finding that were already there for me. It was a great experience. You should try it!
The documentation of Twisted attempts to introduce several event-driven programming patterns. Most people have a 'wtf' moment or two when they come across design patterns that approach things seemingly backwards.
The further I got into my Twisted app, the more I found myself reaching for things and finding that were already there for me. It was a great experience. You should try it!
I've done plenty of work with Twisted. My complaint was not that it lacked functionality, my complaint was that the culture of the project favors 'flexibility' over everything else. It's like the C++ of networking libraries. I'm sure there are plenty of times and places where its approach is exactly what you need, but it's just rarely a good fit for what I need.
The Twisted people don't seem to get that their framework is too complicated for many uses. You can't just work on someone else's framework and make it simpler, if they don't agree that it should be simple in the first place.
"You can't just work on someone else's framework and make it simpler"
Are you suggesting that python is more complex to use than x86_64 assembler?
Similarly, when I removed code from tornado to make it work with twisted, I neither tornado more complicated to use, nor did I introduce any limitations.
The "too complicated for many uses" thing kind of bugs me, though. It's the same argument people use to use for postgres vs. mysql. Something along the lines of "it has features I don't need" (where need probably means "understand" or "need today").
That's where tornado left me. What you call complication, I call features. I use some of these features.
For example, I'd be interested in knowing how something like my tdf web server could be written in tornado. It attaches processes to resource paths, captures and transforms the output from these processes (in a stream) and streams replicates of that data to whatever connections come in.
The answer, of course, involves reimplementing reactor.spawnProcess on top of ioloop. Then you find some of the subtleties contained therein. Then you begin to understand why it seemed complicated to begin with.
Are you suggesting that python is more complex to use than x86_64 assembler?
Similarly, when I removed code from tornado to make it work with twisted, I neither tornado more complicated to use, nor did I introduce any limitations.
The "too complicated for many uses" thing kind of bugs me, though. It's the same argument people use to use for postgres vs. mysql. Something along the lines of "it has features I don't need" (where need probably means "understand" or "need today").
That's where tornado left me. What you call complication, I call features. I use some of these features.
For example, I'd be interested in knowing how something like my tdf web server could be written in tornado. It attaches processes to resource paths, captures and transforms the output from these processes (in a stream) and streams replicates of that data to whatever connections come in.
The answer, of course, involves reimplementing reactor.spawnProcess on top of ioloop. Then you find some of the subtleties contained therein. Then you begin to understand why it seemed complicated to begin with.
> Are you suggesting that python is more complex to use than x86_64 assembler?
No, but twisted is much more complicated than regular python. The inversion of control and the fact that until quite recently it was almost impossible to write code for using the framework that was easy to read/understand by experienced python ciders who were not familiar with the idims and conventions used by the project. Twisted generates more WTFs per minute than any other bit of python kit. Some of this is due to a tradition of poor documentation within the project and some due to the nature of the code. Twisted is a complicated system, and sometimes this complexity is not worth embracing.
No, but twisted is much more complicated than regular python. The inversion of control and the fact that until quite recently it was almost impossible to write code for using the framework that was easy to read/understand by experienced python ciders who were not familiar with the idims and conventions used by the project. Twisted generates more WTFs per minute than any other bit of python kit. Some of this is due to a tradition of poor documentation within the project and some due to the nature of the code. Twisted is a complicated system, and sometimes this complexity is not worth embracing.
At one time the Twisted guys were accused of walking away instead of trying to improve Python's async module. History repeats.
I find it interesting that nearly all the comments here are on Tornado vs. Twisted rather than the larger issue of communicating vs. coding that the author brings up.
There've been a lot of times - most recently yesterday - when I've hacked up a solution, mailed it off, and got back a response of "I just did this a couple days ago. With a one-line change, my code could've covered your needs too." And then of course I'm like, "Oh. Shit. Let's modify your code instead and I'll throw away my changelist."
I get the feeling this happens a lot outside of my own personal/professional sphere as well. Google's famous for its NIH syndrome. Python's got about 5 different competing web frameworks. Lots of people still write their own machine code generators instead of using LLVM. I'm kinda curious how much coding people are willing to do to save one e-mail.
Ironically, the folks who insist on building their own stuff are often right. Google's certainly done well for itself, and it's great that we can use Django instead of Zope now. So maybe the problem is that communication really is so expensive that people are better off rewriting large chunks of code to avoid it. But if that's true, we should be looking for ways to reduce the cost of communication, rather than glorifying the folks who go to heroic measures to avoid it.
It's curious - yesterday, Coders at Work was at the top of the hotlist, and most of the comments said, "I'd rather learn about the business & management aspects than the technical aspects." Yet here's an interesting business & management problem, but everyone's commenting on the technical aspects.
There've been a lot of times - most recently yesterday - when I've hacked up a solution, mailed it off, and got back a response of "I just did this a couple days ago. With a one-line change, my code could've covered your needs too." And then of course I'm like, "Oh. Shit. Let's modify your code instead and I'll throw away my changelist."
I get the feeling this happens a lot outside of my own personal/professional sphere as well. Google's famous for its NIH syndrome. Python's got about 5 different competing web frameworks. Lots of people still write their own machine code generators instead of using LLVM. I'm kinda curious how much coding people are willing to do to save one e-mail.
Ironically, the folks who insist on building their own stuff are often right. Google's certainly done well for itself, and it's great that we can use Django instead of Zope now. So maybe the problem is that communication really is so expensive that people are better off rewriting large chunks of code to avoid it. But if that's true, we should be looking for ways to reduce the cost of communication, rather than glorifying the folks who go to heroic measures to avoid it.
It's curious - yesterday, Coders at Work was at the top of the hotlist, and most of the comments said, "I'd rather learn about the business & management aspects than the technical aspects." Yet here's an interesting business & management problem, but everyone's commenting on the technical aspects.
The reason I think that folks who build their own stuff are right is that they understand their problem domain better than anyone else.
The author says communication could have cut their time down considerably. That may or may not be true. Getting into a debate with an external project about what should or shouldn't be the correct way to fix/change the external projects code can be a huge drain on time and mental energy. You can't fault a group for deciding not to bother.
Not to mention that if you become the person responsible for a piece of an external project you are suddenly not just supporting your use of that piece you are supporting everyone else's use of that piece. That leads to more debating/discussion about the right way to do it. If you have to find a way to scale fast it might actually in the long run be less time to just write it yourself for your specific needs not an entire communities needs who have different scaling issues than you and different goals. Many times One size does not fit all and convincing someone else of that is just too time consuming to try.
The author did bring up an interesting point about friendfeed developers not even trying to communicate though so I can understand what prompted him to write the article.
The author says communication could have cut their time down considerably. That may or may not be true. Getting into a debate with an external project about what should or shouldn't be the correct way to fix/change the external projects code can be a huge drain on time and mental energy. You can't fault a group for deciding not to bother.
Not to mention that if you become the person responsible for a piece of an external project you are suddenly not just supporting your use of that piece you are supporting everyone else's use of that piece. That leads to more debating/discussion about the right way to do it. If you have to find a way to scale fast it might actually in the long run be less time to just write it yourself for your specific needs not an entire communities needs who have different scaling issues than you and different goals. Many times One size does not fit all and convincing someone else of that is just too time consuming to try.
The author did bring up an interesting point about friendfeed developers not even trying to communicate though so I can understand what prompted him to write the article.
[deleted]
I think the author is misinterpreting what's going on. It's not that it's too hard to communicate with the Twisted folks, it's that the FriendFeed folks didn't think Twisted was good enough. Twisted is too buggy and too complicated.
I thought that glyph's response was appropriate and on the mark. It seemed to me that Bret had taken a few jabs at Twisted without really providing quantitate evidence. Particularly the comments about being "demo-quality" and "tons of bugs".
It would have been nice to see innovation happen collaboratively between FriendFeed and the Twisted team. Everyone would have benefited more IMHO.
Twisted has certainly done a lot for event-based architecture. As Glyph points out, the use of Deferreds have made their way into javascript frameworks like dojo. To me, that's credible and proof of twisted's impact in the development community.
It would have been nice to see innovation happen collaboratively between FriendFeed and the Twisted team. Everyone would have benefited more IMHO.
Twisted has certainly done a lot for event-based architecture. As Glyph points out, the use of Deferreds have made their way into javascript frameworks like dojo. To me, that's credible and proof of twisted's impact in the development community.
I really think that calling Deferreds a "de-facto standard" in JavaScript is a huge stretch.
I'm having trouble pinning down exactly what makes Deferreds different from any asynchronous callback system, but it seems to be the chain of callbacks?
I'm having trouble pinning down exactly what makes Deferreds different from any asynchronous callback system, but it seems to be the chain of callbacks?
One of the frustrating differences I ran into is the callback mechanism in tornado. You can't attach more than one callback, and the one kind of callback does two different things (meaning the callback has to determine whether or not the call was successful before it does its work).
To me, this makes things more complicated. What if I want to use the same error handling code for all bits of code (really common for me is `d.addErrback(log.err)` to just log it), but each successful call has to do something different?
Twisted deferreds also allow you to compose chains.
This is a somewhat trivial example, but by having the callback do both success and error cases, I had to build an adaptor to pass to callbacks to include the "is this a failure" information.
The easy way to do that is to create a wrapper class and construct it one way or another based on whether a success or error occurred. I ended up just composing it like this:
This is sort of unrolling complexity and sweeping it away. The real problem is that you can only have one callback in tornado (and you can't easily chain them). That means that what actually happens when this deferred's callback fires is that it ends up calling the original callback, which is an inner wrapper function inside of another function that calls a curried function that calls your original callback -- and you have to create these things by name when you're building out your auth class.
To me, that's complicated.
deferreds seem like there's more involved than you get with a simple callback until you actually try to use the simple callback.
The interesting properties of deferreds are
1) You can add multiple callbacks and errbacks and build chains as complicated as your system needs including composing common cases (such as success and error counters).
2) If a callback (or errback) returns a deferred, you spin off another dimension of the chain.
#2 is why the above code and about twenty lines of web.py exist for a single use case.
Here's a concrete example (close to something I do in the real world™):
Imagine a simple twitter API. You go grab a tweet and then give it to something else: `t.grabTweet(something).addCallback(processTweet)`
Now someone adds a requirement that between `grabTweet` and `processTweet` you want to perform short URL expansion. In twisted, I'd just toss an `addCallback(expandURL)` in the middle and call my awesomely well-tested URL expander -- which makes an async http call to another service and doesn't call the next callback in the chain until that service returns the data I need to fill satisfy the request.
And then after I did that, I found that the URL expansion stuff I was using was blowing up the upstream and causing me to fail to expand some, so I added async cache support to those inside the API (not as an additional toplevel callback). For this new step, the toplevel API doesn't change, but now I can do more async stuff that may lead to further async stuff without having to write additional glue code.
To me, this makes things more complicated. What if I want to use the same error handling code for all bits of code (really common for me is `d.addErrback(log.err)` to just log it), but each successful call has to do something different?
Twisted deferreds also allow you to compose chains.
This is a somewhat trivial example, but by having the callback do both success and error cases, I had to build an adaptor to pass to callbacks to include the "is this a failure" information.
The easy way to do that is to create a wrapper class and construct it one way or another based on whether a success or error occurred. I ended up just composing it like this:
deferred.addCallback(lambda x: Response(body=x))
deferred.addErrback(lambda e: Response(error=e))
deferred.addCallback(lambda r: callback(r, *args, **kwargs))
This is sort of unrolling complexity and sweeping it away. The real problem is that you can only have one callback in tornado (and you can't easily chain them). That means that what actually happens when this deferred's callback fires is that it ends up calling the original callback, which is an inner wrapper function inside of another function that calls a curried function that calls your original callback -- and you have to create these things by name when you're building out your auth class.
To me, that's complicated.
deferreds seem like there's more involved than you get with a simple callback until you actually try to use the simple callback.
The interesting properties of deferreds are
1) You can add multiple callbacks and errbacks and build chains as complicated as your system needs including composing common cases (such as success and error counters).
2) If a callback (or errback) returns a deferred, you spin off another dimension of the chain.
#2 is why the above code and about twenty lines of web.py exist for a single use case.
Here's a concrete example (close to something I do in the real world™):
Imagine a simple twitter API. You go grab a tweet and then give it to something else: `t.grabTweet(something).addCallback(processTweet)`
Now someone adds a requirement that between `grabTweet` and `processTweet` you want to perform short URL expansion. In twisted, I'd just toss an `addCallback(expandURL)` in the middle and call my awesomely well-tested URL expander -- which makes an async http call to another service and doesn't call the next callback in the chain until that service returns the data I need to fill satisfy the request.
And then after I did that, I found that the URL expansion stuff I was using was blowing up the upstream and causing me to fail to expand some, so I added async cache support to those inside the API (not as an additional toplevel callback). For this new step, the toplevel API doesn't change, but now I can do more async stuff that may lead to further async stuff without having to write additional glue code.
I agree that FF should have asked and should have used twisted under the hood. I hope the porting efforts will be successful and eventually replace the original Tornado impl.
I cannot help, though, but yet again in this blog post notice the unfortunate, defensive tone that is so omnipresent in the twisted community.
The post starts out by (politely and rightfully) pointing out FriendFeed's communication mistake and then goes on to describe how green the grass would be in twisted land.
What I miss is a bit more critical self-reflection about the reasons for FF skipping twisted, even without FF spelling them out in detail. It's not likely that FF found completely new and unseen problems with twisted. I'm more inclined to believe that they simply stumbled over the standard, long-known twisted problems; first and foremost the lack of documentation and (probably) the verbose coding-style that many parts of twisted demand.
I have the highest respect for the effort that went into twisted and for their often cited code quality. But they need to realize that they can't display a "your own fault if you're not smart enough for this"-attitude towards documentation and at the same time mock anyone who doesn't want to put up with that.
Like any OSS project twisted is ofcourse not obliged to anything. But neither are people like the FF guys obliged to choose twisted.
I cannot help, though, but yet again in this blog post notice the unfortunate, defensive tone that is so omnipresent in the twisted community.
The post starts out by (politely and rightfully) pointing out FriendFeed's communication mistake and then goes on to describe how green the grass would be in twisted land.
What I miss is a bit more critical self-reflection about the reasons for FF skipping twisted, even without FF spelling them out in detail. It's not likely that FF found completely new and unseen problems with twisted. I'm more inclined to believe that they simply stumbled over the standard, long-known twisted problems; first and foremost the lack of documentation and (probably) the verbose coding-style that many parts of twisted demand.
I have the highest respect for the effort that went into twisted and for their often cited code quality. But they need to realize that they can't display a "your own fault if you're not smart enough for this"-attitude towards documentation and at the same time mock anyone who doesn't want to put up with that.
Like any OSS project twisted is ofcourse not obliged to anything. But neither are people like the FF guys obliged to choose twisted.
I 2nd this completely re: Twisted documentation. When Twisted's communication via their documentation has been so lacking/off-putting/etc., it is hard to feel sympathy for a Twisted guy complaining about the lack of communication from people who chose a different route.
One thing to remember about Twisted is that it is basically worked on by two people. It is an enormous code base for those two to maintain, develop on, and document at the same time, even if they are ridiculously smart (which they are).
I gave kudos to Tornado, but Twisted deserves some appreciation too -- and contributors! I tried to take twisted.web into the present (the client anyway), but I unfortunately wasn't sufficiently TDD to get it contributed. Perhaps next time!
I gave kudos to Tornado, but Twisted deserves some appreciation too -- and contributors! I tried to take twisted.web into the present (the client anyway), but I unfortunately wasn't sufficiently TDD to get it contributed. Perhaps next time!
I submitted a patch to Twisted to fix their SOCKS implementation.
It was "accepted" after a month, and by accepted it means put into a branch where it will take up to a year to be incorporated into the mainline depending on whether the few maintainers feel like taking a look at it.
It was "accepted" after a month, and by accepted it means put into a branch where it will take up to a year to be incorporated into the mainline depending on whether the few maintainers feel like taking a look at it.
Two people is news to me — their website lead me to believe otherwise:
http://twistedmatrix.com/trac/wiki/TwistedMatrixLaboratories
There are several async programming libraries for Python including Allegra ( http://laurentszyster.be/blog/allegra/doc/ ), libevent-python (and other libevent bindings to python), the standard asyncore module and more. Why should Facebook use Twisted instead of one of these other libraries?
more importantly, why should it roll it's own given a plethora of more mature alternatives?
Perhaps there are many alternatives because they all have problems.
Nice article. As someone who has been involved with open source for a while, I think one of the most annoying things is indeed when people don't even tell you about their problems. I'm already doing work for you for free; I don't have money to go do market research, so I'm counting on you to step up and say something if there's a problem.
I understand the response at some level, but I also totally get the idea of making things work the way one's mind works. It makes coding much more enjoyable and speeds the process of app development once the core work is done.
And it was nice of the author to do some QA work for Tornado, too. Hopefully he'll go the extra step and submit those bugs he found as tickets so that he's walking the walk and not just talking the talk. :-)
And it was nice of the author to do some QA work for Tornado, too. Hopefully he'll go the extra step and submit those bugs he found as tickets so that he's walking the walk and not just talking the talk. :-)
It sounds very good, but I must admit I immediately remembered lots and lots of comments from open source developers that amounted to "not happy with our project (a ka 'free toilet')? well duh, hack it yourself, we owe you s*". Obviously not the fault of the Twisted developer, but I am not surprised if people actually prefer to hack their own stuff.
One reason I wouldn't have reached out, as a private company, is tipping my hand to competitors. Maybe an NDA would solve that?
Well, if this benchmark from yesterday is accurate, then Twisted can't handle as many concurrent connections as Tornado. I'd say that's a good reason not to use Twisted.
http://www.apparatusproject.org/blog/2009/09/twisted-web-vs-...
http://www.apparatusproject.org/blog/2009/09/twisted-web-vs-...
If you read the comments in the blog, it's likely just related to the platform. I did the test on my mac, which certainly isn't the final say... I'll redo the test today different hardware today.
Perhaps this is an artifact of how seriously the Twisted people and their 'lead hacker' take their project. But the twisted-centric view of the world leads to petulant posts full of odd statements:
My main point here is that if you're about to undergo a re-write of a major project
They wrote Tornado, they didn't 'undergo a rewrite of Twisted'.
For a tiny fraction of the effort invested in Tornado, FriendFeed could have worked with us to resolve many of the issues creating that chaos.
Interfacing with an another team to solve problems in a framework they didn't like (even if just aesthetically) would have probably taking significant resources for close to zero return.
There's quite a bit more in the same vein. It's wonderful that the Twisted people care. But perhaps a little too much.