Ask YC: Should I use a framework or just write the code myself?
41 comments
I worked on a large Python project before Django came out and ended up building my own framework. Recently, when I used Django, I realized that almost every feature it had was something I had ended up writing myself. Django had a lot more man hours going into the framework and was much cleaner and better tested than my implementation.
There are certain things you need for every project: -ways to handle configuration files for different environments -a pattern to map URL's to methods and templates -a basic ORM so you don't have to write SQL for every single CRUD operation -a test framework that simulates an actual web request -a way to handle SQL migrations -a way to get data to and from forms
Almost every production web site needs all of the above. Why spend time doing it yourself when people have solved these problems already? The only reason I can see for writing your own framework is as a pure learning experience.
There are certain things you need for every project: -ways to handle configuration files for different environments -a pattern to map URL's to methods and templates -a basic ORM so you don't have to write SQL for every single CRUD operation -a test framework that simulates an actual web request -a way to handle SQL migrations -a way to get data to and from forms
Almost every production web site needs all of the above. Why spend time doing it yourself when people have solved these problems already? The only reason I can see for writing your own framework is as a pure learning experience.
I started with a very light framework (web.py) and then switched to Pylons.
I'd say it's mostly based on personal preference. If you don't use a framework, you will end up reinventing a lot of stuff that's in mainstream frameworks. OTOH, the stuff you spend reinventing really won't take you much longer than learning the framework, as long as you build off a decent set of libraries (don't try to reinvent your own templating-library/HTML-escaping/database-interface etc.) And if you do switch to a framework, you'll probably be better able to understand the reasoning behind why the framework did what it did.
One other consideration: frameworks make it much easier for somebody new to get up-to-speed with your code, and often tackle thorny deployment issues that you can spend a lot of time on otherwise. Keep that in mind when deciding - for a startup, the priority is to get something up on screen so you can see if you're on the right track, but you do eventually want to be able to grow and scale.
I'd say it's mostly based on personal preference. If you don't use a framework, you will end up reinventing a lot of stuff that's in mainstream frameworks. OTOH, the stuff you spend reinventing really won't take you much longer than learning the framework, as long as you build off a decent set of libraries (don't try to reinvent your own templating-library/HTML-escaping/database-interface etc.) And if you do switch to a framework, you'll probably be better able to understand the reasoning behind why the framework did what it did.
One other consideration: frameworks make it much easier for somebody new to get up-to-speed with your code, and often tackle thorny deployment issues that you can spend a lot of time on otherwise. Keep that in mind when deciding - for a startup, the priority is to get something up on screen so you can see if you're on the right track, but you do eventually want to be able to grow and scale.
web.py is on my list to consider. It's lightness and simplicity is really attractive.
One thing that bothers me in a lot of frameworks is the object wrapping of the database. I can't see much justification for this, it just seems to be adding a layer of complexity on top of a very clearly designed model. How much easier is it to use an object abstraction when you could just do SQL directly?
John.
One thing that bothers me in a lot of frameworks is the object wrapping of the database. I can't see much justification for this, it just seems to be adding a layer of complexity on top of a very clearly designed model. How much easier is it to use an object abstraction when you could just do SQL directly?
John.
I would take a serious look at Django. You can use it as scaffolding and get rid of pieces as you write you own, the first to go will be the use of the automated admin site, and I find myself writing a custom pieces across the board.
However, being able to rapidly deploy a prototype has been priceless time and time again, and they have the basics that you need down and well tested.
If you don't like ORM, or find yourself writing a lot of custom queries for performance, that is fine. I would still use an ORM now and customize some queries for performance later. The reason? Speed of building your app...I think not writing the DB code saves me about 20% of my time.
I used to be always against using frameworks since they were massive, crappy, and relied on so much magic you didn't know what was going on behind the scenes. I am really glad I went with Django, it saved me from writing almost the identical feature set.
However, being able to rapidly deploy a prototype has been priceless time and time again, and they have the basics that you need down and well tested.
If you don't like ORM, or find yourself writing a lot of custom queries for performance, that is fine. I would still use an ORM now and customize some queries for performance later. The reason? Speed of building your app...I think not writing the DB code saves me about 20% of my time.
I used to be always against using frameworks since they were massive, crappy, and relied on so much magic you didn't know what was going on behind the scenes. I am really glad I went with Django, it saved me from writing almost the identical feature set.
It's interesting that you mention Django. Of all the frameworks it's the one I most liked as I was investigating for one major reason: the community around Django seemed welcoming and open to ideas. This was very much in contrast to the RoR community which seemed to have a "we know better than you" attitude.
John.
John.
The community is quite good overall.
It is especially good about allowing random people to contribute pieces of code that are useful. I also like how their goal is to make everything as straightforward as possible, and eliminate the 'magic' that most frameworks have. If you have any questions about it, feel free to ask.
It is especially good about allowing random people to contribute pieces of code that are useful. I also like how their goal is to make everything as straightforward as possible, and eliminate the 'magic' that most frameworks have. If you have any questions about it, feel free to ask.
Do you feel like you know what's going on behind the scenes in Django? That's my biggest concern with it. (haven't tried it yet.)
I think you'll find that the stuff Django does behind the scenes is quite clear, with very few exceptions. The basic parsing of HTTP requests is dead simple, and although the code for the ORM is hairy in places, what really matters is the SQL it generates, which is pretty clean. And like somebody else said, there's nothing that says you have to use the Django ORM.
One of the nice things about Django is that it's just Python. You can always look at the import statements in a module to see exactly what code is being run and where it's coming from. I've never used Ruby on Rails, but I have the impression it can be opaque in this area. ("Just create a file named makeasite.rb and put the single word 'create' in it, and Rails will do the rest!")
Personally, I'm vehemently against "magic" in a framework / programming environment, which is one of the reasons I love Python -- and one of the reasons we don't have a lot of magic in Django.
Let me know if you have any specific questions.
Adrian (Django cocreator / core developer)
One of the nice things about Django is that it's just Python. You can always look at the import statements in a module to see exactly what code is being run and where it's coming from. I've never used Ruby on Rails, but I have the impression it can be opaque in this area. ("Just create a file named makeasite.rb and put the single word 'create' in it, and Rails will do the rest!")
Personally, I'm vehemently against "magic" in a framework / programming environment, which is one of the reasons I love Python -- and one of the reasons we don't have a lot of magic in Django.
Let me know if you have any specific questions.
Adrian (Django cocreator / core developer)
No reason to knock the 'magicality' of rails. It works for some people; And since it is only code at the end of the day, you can always get to the bottom of the magic if you do a little digging.
While we are on the topic, I have always wondered why the Django team insists on being an MTV (Model Template View) framework..don't you guys think just conforming to MVC would make life easier for people new to Django? (This is a serious question)
While we are on the topic, I have always wondered why the Django team insists on being an MTV (Model Template View) framework..don't you guys think just conforming to MVC would make life easier for people new to Django? (This is a serious question)
> No reason to knock the 'magicality' of rails.
The problem with relying on magic is that you won't know what to do when it stops working.
The problem with relying on magic is that you won't know what to do when it stops working.
The Rails magic is well defined though, and it's all documented. It doesn't simply 'stop working'. It's really nothing more than a set of conventions that happen to make certain things really convenient.
Pretty cool. Ok, it's now on my list of frameworks to try :-)
There's a lots of answers to your question, John. I'll try giving you a couple.
One -- I might want my data to live in multiple places. For instance, I had an app I wrote recently that could persist its data to and from three places -- SQL, a funky XML format, and a private file format. That's too much read/write crap to pollute my app with.
Two -- I might want my data to live in multiple versions. If you've got multiple versions of your data format out there, such as if you were Quicken or something, you want to change/modify the data code in one spot, not all over the place.
Three -- the job of writing the UI is different than the job of knowing about data. Whether it's one box or ten thousand, one table or five, why should I care/know where the data lives? I'm writing the UI for goodness sakes. Just make the stupid stuff go away and come back when I need it. One well-constructed data tier should be able to service a hundred UI developers.
When coding imperatively and using a "short-cut" data access paradigm like RoR or CodeSmith and NetTiers (my favorite), the temptation is to start thinking well gee, if every table is an object, what's the point? Why not just hit the database directly? The answer is that you're coding just using a big shortcut -- it ain't going to work all the time like that. You have to know why the shortcut works and why it doesn't work.
One -- I might want my data to live in multiple places. For instance, I had an app I wrote recently that could persist its data to and from three places -- SQL, a funky XML format, and a private file format. That's too much read/write crap to pollute my app with.
Two -- I might want my data to live in multiple versions. If you've got multiple versions of your data format out there, such as if you were Quicken or something, you want to change/modify the data code in one spot, not all over the place.
Three -- the job of writing the UI is different than the job of knowing about data. Whether it's one box or ten thousand, one table or five, why should I care/know where the data lives? I'm writing the UI for goodness sakes. Just make the stupid stuff go away and come back when I need it. One well-constructed data tier should be able to service a hundred UI developers.
When coding imperatively and using a "short-cut" data access paradigm like RoR or CodeSmith and NetTiers (my favorite), the temptation is to start thinking well gee, if every table is an object, what's the point? Why not just hit the database directly? The answer is that you're coding just using a big shortcut -- it ain't going to work all the time like that. You have to know why the shortcut works and why it doesn't work.
I avoid ORM completely and am perfectly happy with SQL.
I use a very thin database access library over DBAPI (modeled after PEAR DB, basically just wrapping queries with logging and timing, and converting results to Python dicts instead of DBAPI cursors). It was no problem using this with Pylons, instead of the default SQLAlchemy. If you don't like a Pylons module, just swap it out; the framework is built so you can easily substitute your own components in instead of the defaults.
I used the same library with web.py. Web.py's built-in database library is pretty impractical. Global DB connection means it's very difficult to do testing/mockobjects, and it wouldn't scale beyond one machine.
I used the same library with web.py. Web.py's built-in database library is pretty impractical. Global DB connection means it's very difficult to do testing/mockobjects, and it wouldn't scale beyond one machine.
The main practical benefit of object wrapping (ActiveRecord in particular) is that it gives you a more structured way to lay out the model-related logic for your app. Callbacks, model validations, and associations in particular allow you to easily compartmentalize a lot of the details of your models. If you're looking for ways to reduce the complexity of the problem space in your head, ActiveRecord et al are great ways to do it.
(ymmv. If you eat, sleep, think, dream, and make love in SQL, I guess you might as well ditch the ORM)
(ymmv. If you eat, sleep, think, dream, and make love in SQL, I guess you might as well ditch the ORM)
[deleted]
I felt in several projects that it wouldn't have been possible to do it without an OR mapper.
Maybe if it's always just select "star" instead of select object, the difference is not so great, but if you can't use select * for some reason, it becomes unbearable for me...
Also, the automatic loading of linked objects and collections, and the caching, are nice features imo. You only have to write object.getMappedThingies() instead of another SQL request (talking about Java here).
Maybe if it's always just select "star" instead of select object, the difference is not so great, but if you can't use select * for some reason, it becomes unbearable for me...
Also, the automatic loading of linked objects and collections, and the caching, are nice features imo. You only have to write object.getMappedThingies() instead of another SQL request (talking about Java here).
SQL is extremely verbose. During my previous life as a Java programmer I moved a project from raw SQL to the hibernate ORM, and removed 10,000 lines of code in the process. Using straight SQL, the data access code was more than 50% of the entire code base. Switching to an ORM got it below 10%.
It really depends on what you're doing, but I can't imagine why anyone would want to write SQL by hand. If you really want simplicity, don't use a database at all.
It really depends on what you're doing, but I can't imagine why anyone would want to write SQL by hand. If you really want simplicity, don't use a database at all.
Are you sure that's SQL and not JDBC?
I moved from PHP/PearDB to Java/Hibernate3.0 to Python/DBAPI, and found that Java was by far the most verbose. Even with Hibernate annotations, I was writing more than twice as much code for equivalent functionality.
Perhaps the difference is that you have business objects defined already, and so the cost of Hibernate is only the annotations? My PHP and Python programs keep the SQL and drop the objects, and so they end up with incredibly tiny data access layers.
SQL is optimized for writing data access queries extremely concisely, so I don't see how it could end up as more than 50% of the codebase, unless you're using an exceptionally verbose database library (JDBC certainly counts) or your codebase is exceptionally tiny.
I moved from PHP/PearDB to Java/Hibernate3.0 to Python/DBAPI, and found that Java was by far the most verbose. Even with Hibernate annotations, I was writing more than twice as much code for equivalent functionality.
Perhaps the difference is that you have business objects defined already, and so the cost of Hibernate is only the annotations? My PHP and Python programs keep the SQL and drop the objects, and so they end up with incredibly tiny data access layers.
SQL is optimized for writing data access queries extremely concisely, so I don't see how it could end up as more than 50% of the codebase, unless you're using an exceptionally verbose database library (JDBC certainly counts) or your codebase is exceptionally tiny.
This app had something like 200 different tables, with 20 to 50 columns in most tables. Update and insert statements were 5 - 10 lines long, and there were hundreds of the things. There was some JDBC cruft too, but the vast majority was SQL.
This was an extreme case of course, and was mostly the result of bad design, but I haven't come across a database backed application that couldn't be trimmed down with some sort of SQL generator. SQL is fine for simple queries, but, for me at least, has always gotten unmanageable in the long term. Need to select, insert and update the same columns? Duplication. Conditionally group by or order by different fields? Duplication. Decide after the fact that you need to support multiple databases, or eager and lazy loading? There goes the weekend, and probably most of next week.
You could manage all of the above with some sort of home grown solution, but a good ORM just makes everything so painless. I was strongly anti-database for years, yet I now often find myself using ActiveRecord when a flat file would do, simply because ActiveRecord is easier.
Of course, that's just me. SQL and I were never a good fit, and I'm glad to be mostly rid of it, but I'm sure others have had different experiences.
This was an extreme case of course, and was mostly the result of bad design, but I haven't come across a database backed application that couldn't be trimmed down with some sort of SQL generator. SQL is fine for simple queries, but, for me at least, has always gotten unmanageable in the long term. Need to select, insert and update the same columns? Duplication. Conditionally group by or order by different fields? Duplication. Decide after the fact that you need to support multiple databases, or eager and lazy loading? There goes the weekend, and probably most of next week.
You could manage all of the above with some sort of home grown solution, but a good ORM just makes everything so painless. I was strongly anti-database for years, yet I now often find myself using ActiveRecord when a flat file would do, simply because ActiveRecord is easier.
Of course, that's just me. SQL and I were never a good fit, and I'm glad to be mostly rid of it, but I'm sure others have had different experiences.
Streamlining SQL code generation and mapping tables to objects are two entirely different feature sets in my opinion. 90 % of the reduction in code you're talking about stems from the former whereas 90 % of the complexity of ORMs stems from the latter. That's why I prefer a leaner toolset to interface with the database.
[deleted]
For me, the main benefit of ORM is that it makes it easy to plug in caching (memcache) for just about everything, which you're probably going to want when your reach scale.
I've used frameworks and written my own. Obviously what works better depends a lot on how similar your problem is to what the makers of the framework intended to solve. The dilemma is that as you build your app your problems become more specific to that particular app and you tend to gain less and less from the framework. Eventually the framework gets in your way. Not because it forces you to do it in a particular fashion, but because you feel compelled not to stray from the framework's path in the interest of consistency and simplicity. Unfortunately, the same may be true of your own framework creations, just the consequences are different.
Frameworks are great if you do a series of similar, straightforward projects like many contractors would do. A framework allows people to get acquainted with your code more quickly, so it's good for companies with high staff turnover. It also has a name that HR can put in a job ad.
If you roll your own, try to do it as a collection of independent libraries, not as a don't-call-us-we-call-you type of framework. That's my conclusion after many attempts at framework writing. It has always been the case that some parts were good and others were crap. You can't throw the crap out if everything is interconnected. The code that connects everything needs to be either the smallest part of the framework or part of the application itself.
But anyway, as long as you stay away from CMS, you're going to be fine.
Frameworks are great if you do a series of similar, straightforward projects like many contractors would do. A framework allows people to get acquainted with your code more quickly, so it's good for companies with high staff turnover. It also has a name that HR can put in a job ad.
If you roll your own, try to do it as a collection of independent libraries, not as a don't-call-us-we-call-you type of framework. That's my conclusion after many attempts at framework writing. It has always been the case that some parts were good and others were crap. You can't throw the crap out if everything is interconnected. The code that connects everything needs to be either the smallest part of the framework or part of the application itself.
But anyway, as long as you stay away from CMS, you're going to be fine.
Sorry for the double post, but I would also like to add that SQLAlchemy and Mako templates were more of a deciding factor than anything else for me to use pylons. Pick the things that you will absolutely not be reinventing (templates, orm, etc) and find where they play best together. Do you like your templates simple and with very little logic? do you like them in xml? do you like 'magic' orms? Also another thing is note how this app will be deployed. If it is purely for fun and learning, this isn't an issue. Otherwise, read up on how easily different frameworks play with apache mod_python, fcgi, scgi, etc. because limitations in that area aren't easily mended. I did have to wage a bloody war to get pylons running fcgi on my shared hosting server. Now the war is won the solution really isn't that hard/complicated or obscure. It was just lack of good documentation that needlessly killed hours and brain-cells. I'd recommend looking for as much documentation in these areas. If you can't find it fairly quickly, chances are you will be paying for it in the future.
I started with this whole web framework thing writing my own crappy code in perl. I found python, loved it, and immediately started learning django (the 0.96 version.) I quickly became dissatisfied with the limitations of the framework as my apps grew in complexity. This was somewhat in part due to new additions like newforms that were not yet completed or fully integrated. Currently I'm using the pylons framework and I am absolutely loving it. It sounds like if you are even 'considering' writing all the code yourself, you should really pick a framework akin to pylons or web.py in your language of choice. Ruby, Perl, Python, it's really all the same...
I had the very same question before, and even implemented many of the common things that my application needed. But I am now looking at Drupal.
In general whichever framework you would like to use, will have its own learning curve. Once you overcome that, it should be pretty easy and fast to start working. I think there are some obvious advantages in using a framework:
1- you don't have to reinvent a lot of the stuff
2- the solutions offered to you have passed all their tests as there are many people checking upon them, and many sites already using them
3- if you have a couple of ideas and are thinking of launching all, I think once you finish your first project you will see a huge advantage starting the other ones, and you will move a lot faster
at the end if you pick the right framework, that will satisfy your current needs (the features and options you like to offer) and future needs (such as scalability), you will feel like you are working with a huge group of developers on your site. Its kind of like you have hired other developers and formed a team for yourself.
Also make sure that the framework you pick will have a strong community as it will come handy.
You should make a list of your requirements and make a chart of the frameworks and which ones satisfy what. And in general I would take the following into consideration when choosing a framework (depending on your needs you could add to this list)
- Ease to install - Learning Curve - User Control - Session Control - Extensibility - Scalability - Themability - xHTML/CSS
In general whichever framework you would like to use, will have its own learning curve. Once you overcome that, it should be pretty easy and fast to start working. I think there are some obvious advantages in using a framework:
1- you don't have to reinvent a lot of the stuff
2- the solutions offered to you have passed all their tests as there are many people checking upon them, and many sites already using them
3- if you have a couple of ideas and are thinking of launching all, I think once you finish your first project you will see a huge advantage starting the other ones, and you will move a lot faster
at the end if you pick the right framework, that will satisfy your current needs (the features and options you like to offer) and future needs (such as scalability), you will feel like you are working with a huge group of developers on your site. Its kind of like you have hired other developers and formed a team for yourself.
Also make sure that the framework you pick will have a strong community as it will come handy.
You should make a list of your requirements and make a chart of the frameworks and which ones satisfy what. And in general I would take the following into consideration when choosing a framework (depending on your needs you could add to this list)
- Ease to install - Learning Curve - User Control - Session Control - Extensibility - Scalability - Themability - xHTML/CSS
CodeIgniter
It's lightweight, scales exceptionally well, designed in a language 95% of the web development community knows like the back of their hand, and powerful.
You'll keep yourself from reinventing the wheel, yet you won't limit yourself on capability by conforming to an obscure set of rules and development practices.
You'll keep yourself from reinventing the wheel, yet you won't limit yourself on capability by conforming to an obscure set of rules and development practices.
Ditto on CodeIgniter.
I feel like you - so far I have usually used frameworks, but there are always aspects that make me unhappy (not even RoR satisfied me). Now I wonder if the recent article about IDEs also applies to frameworks: if you insist on using a framework, you probably won't be able to use the latest, coolest programming languages, because there are no frameworks for them yet.
I am not sure how much work it is to create one's own framework, but I suspect it is less than one might think. The hardest part to me seems serious OR mapping+caching.
Personally, I would probably go the same route as Groovy on Grails and build on top of Spring+Hibernate (so leverage existing frameworks, but in my own style).
Also, I don't know any programmers who became famous for using other people's frameworks.
I am not sure how much work it is to create one's own framework, but I suspect it is less than one might think. The hardest part to me seems serious OR mapping+caching.
Personally, I would probably go the same route as Groovy on Grails and build on top of Spring+Hibernate (so leverage existing frameworks, but in my own style).
Also, I don't know any programmers who became famous for using other people's frameworks.
I'm a caveman, trying to squeeze every bit of functionality and performance out of my app. I roll out everything myself. May take a little longer, but the OCD in me is happy.
http://news.ycombinator.com/item?id=77136
http://news.ycombinator.com/item?id=77136
Flip a coin.
It doesn't have to be a balanced coin: I like Ruby, so my coin was biased toward Rails. Ergo, I learned Rails, and will probably go on to play with Merb and Camping.
Meanwhile, very smart people assure me that Python is nice, and I hope to learn it before I die.
The important thing is to just get moving. Deciding between Rails and, say, Drupal is complicated, with lots of technical, marketing, and business factors in play. But RoR, Django, and the gang are so much alike that the decision comes down to that simple coin flip.
It doesn't have to be a balanced coin: I like Ruby, so my coin was biased toward Rails. Ergo, I learned Rails, and will probably go on to play with Merb and Camping.
Meanwhile, very smart people assure me that Python is nice, and I hope to learn it before I die.
The important thing is to just get moving. Deciding between Rails and, say, Drupal is complicated, with lots of technical, marketing, and business factors in play. But RoR, Django, and the gang are so much alike that the decision comes down to that simple coin flip.
If you can find one that gels with what you're trying to do, then by all means, save yourself the time and go with it.
I think light frameworks are the best because they do not cause frameworkitis.
http://uint32t.blogspot.com/2007/11/web-programming-framewor...
http://uint32t.blogspot.com/2007/11/web-programming-framewor...
I use RoR and am very happy with it. Instead of "reading and feeling", try coding your app and see how things go. Even if you switch, by coding, you'll have made a lot of progress.
That's general good advice, and usually I try to match a project with a technology I want to learn and go code. It's the best way. In this case I'm trying to figure out do I learn python and use a python framework, learn ruby and use RoR, decide to use a PHP framework, just code the entire thing in Perl, ...
John.
John.
some points about RoR, (I checked it a few months back so they might have changed!)
- It does not offer an out of box solution to allow authenticated users to create content forming pages as part of a Web site. It offers you a web app framework and you could build upon it your custom CMS from scratch (and it will take you longer to do so)
- but there is a nice binding to the database through the framework
- at the time stability was a bit of an issue!
with all that said there are great applications built upon it such as 37Signals Base Camp
- It does not offer an out of box solution to allow authenticated users to create content forming pages as part of a Web site. It offers you a web app framework and you could build upon it your custom CMS from scratch (and it will take you longer to do so)
- but there is a nice binding to the database through the framework
- at the time stability was a bit of an issue!
with all that said there are great applications built upon it such as 37Signals Base Camp
Do you know of any great applications built on top of CMS? I don't.
I am not sure what kind of applications you are looking for, but in general there are thousands of successful and big sites out there... for instance look at the sugar network, its based on Drupal
also if you mean built on CMS out of the box, I don't either, and you will have to customize it in order to make something useful, but the point is that with RoR you have to pretty much build from scratch whereas with other CMS like Drupal you already have many functionalities included in the core and you don't have to really reinvent many things, you just customize or add the ones you want
also if you mean built on CMS out of the box, I don't either, and you will have to customize it in order to make something useful, but the point is that with RoR you have to pretty much build from scratch whereas with other CMS like Drupal you already have many functionalities included in the core and you don't have to really reinvent many things, you just customize or add the ones you want
True hackers code their Web apps in x86 assembly.
Do YCers here have experience with or without a framework? What would you choose?