Eighteen months of Django(dangoldin.com)
dangoldin.com
Eighteen months of Django
http://dangoldin.com/2013/05/07/eighteen-months-of-django/
13 comments
We've had a lot of luck just using gunicorn with a post_response hook for doing some extra work without making the client wait on it before getting a response. It avoids adding another dependency and is pretty straightforward. It makes sense for us since we don't have HUGE tasks that need to be queued, but some things that can take a maybe 0.5 to 3 seconds and where the HTTP response can be sent back before doing this work. I should open source this at some point, it's a small amount of code but pretty handy.
Note that periodic execution, one of the more useful aspects of celery, is not supported on all backends. If you're using rabbitMQ, you're golden. If you're using most or all other transports (including SQS), you're likely SOL and will have to resort to either bare cron jobs or something like django-cronograph that accomplishes the same thing.
celery is definitely nice, but it does have its limitations (like everything else).
celery is definitely nice, but it does have its limitations (like everything else).
I'm really interested in peoples opinion on Celery vs Others when using Django.
I'm about to hit a point in my pet project where I need something but I havn't chosen my approach yet. I've ehard Pyres is better than Celery.
I'm about to hit a point in my pet project where I need something but I havn't chosen my approach yet. I've ehard Pyres is better than Celery.
Once you wrap your head around one (embarrassingly, it took me a while to really understand what celery was doing -- I had an easier time getting comfortable with the multiprocessing library), it shouldn't be difficult to pick up another and compare them locally.
Celery does what it's supposed to; that's about all I can say for it. I'd suggest writing a few simple jobs and testing them in similar environments under as much load as you can. Look for things like consistency of logging and running tasks exactly once across multiple consumers despite the semantics of the queue you pick.
Celery does what it's supposed to; that's about all I can say for it. I'd suggest writing a few simple jobs and testing them in similar environments under as much load as you can. Look for things like consistency of logging and running tasks exactly once across multiple consumers despite the semantics of the queue you pick.
This is good to know. I've just started using it with redis.
I never really saw the point of that part of celery anyway. Cron's perfectly adequate on its own (and installed just about everywhere).
I never really saw the point of that part of celery anyway. Cron's perfectly adequate on its own (and installed just about everywhere).
I'm using cron instead of celery daemon for those tasks, but here are two advantages:
- being able to modify the task frequency from the admin
- cron needs special care with setting the environment variables like PATH
- being able to modify the task frequency from the admin
- cron needs special care with setting the environment variables like PATH
The nice thing about it from our point of view is the ability to keep everything in source control. Granted, you can keep your cron jobs as scripts, but celery+rabbitMQ allows you to keep it all in Python without the extra deployment step. It's not a huge deal, but it does help a bit with maintainability and getting new devs up to speed.
We keep all the deployment stuff in source control anyway. For me, it would just mean shifting it from being declared by salt in the "infrastructure" folder to being written in python in the django folder.
I agree that you should keep the code itself in python, but that can be handled just by making it a management command that gets called by cron.
Sysadmins understand and love cron, too. I think celery reinventing the wheel might make some of them a bit mad.
I agree that you should keep the code itself in python, but that can be handled just by making it a management command that gets called by cron.
Sysadmins understand and love cron, too. I think celery reinventing the wheel might make some of them a bit mad.
Yea - that's going to be one of the packages I mention in the follow up. I ended up using RabbitMQ as the backend for it which wasn't that easy to setup so I am curious to see what others are using.
What about http://pythonhosted.org/APScheduler/ ? Celery is for more massive, high capacity solutions.
Using celery for the first time, coming from PHP-land, blew my mind.
Python is now my language of choice for "getting shit done".
Python is now my language of choice for "getting shit done".
Returning to PHP land afterwards feels like you're taping your code together with duct tape and bits of string.
At the risk of this becoming a "bash PHP" thread, that's exactly how I feel. Unless I'm doing something very simple, it's always easier, cleaner, and simpler to do it in Python. Even building a basic CRUD app is night and day between the two languages; Django is beautiful and intuitive, while the PHP frameworks come close but have a few snags that make working with them frustrating.
I write this not to meaninglessly bash PHP but to hopefully convince even a single developer passing by to try out Python. I thought the same thing as you, passer-by dev, once: "Why do I need Python? It's stupid. It uses whitespace weirdly. I can do whatever I need in PHP." But then you realize you could be doing everything you're doing in PHP... better, smarter, and faster.
I write this not to meaninglessly bash PHP but to hopefully convince even a single developer passing by to try out Python. I thought the same thing as you, passer-by dev, once: "Why do I need Python? It's stupid. It uses whitespace weirdly. I can do whatever I need in PHP." But then you realize you could be doing everything you're doing in PHP... better, smarter, and faster.
I'm getting the same feeling now that I have to work in python/django after having used scala. So it's always worth trying new languages, even when you've found one that seems great at the time.
The fact is PHP has excellent frameworks. I tried Django and Flask , and love them. But for now , i'm sticking to PHP( to much legacy code to port ). For new projects ? sure. The only thing which is a bit more complicated with serverside python is deployment on client servers, the rest ( development , testing ), is easier than PHP. I only wished Python had type hinting for functions and methods like PHP does though. Aside from these issues , Python is perfect,more consice and have awesome stuffs like decorators. And a whole lot of awesome libraries.
For those who would like to get started with Django, here's a skeleton project
https://github.com/jordn/heroku-django-s3 and guide helps get you set up quickly with:
- hosting on heroku
- static files on Amazon S3
- virtualenv(wrapper) isolated packages
- environment and security critical variables removed from settings.py and stored as environment variables (following the 12-factor design ethic)I have a couple of thoughts:
1. Why not use django-admin.py startproject --template=<url>? I typically do something like django-admin.py startproject --template=https://github.com/rdegges/django-skel/zipball/master mynewproject so I don't actually have to clone
2. No runtime.txt for heroku?
3. Not specifying version numbers in your requirements.txt is very dangerous. I prefer something like "Django>=1.5.0,<=1.5.9"
1. Why not use django-admin.py startproject --template=<url>? I typically do something like django-admin.py startproject --template=https://github.com/rdegges/django-skel/zipball/master mynewproject so I don't actually have to clone
2. No runtime.txt for heroku?
3. Not specifying version numbers in your requirements.txt is very dangerous. I prefer something like "Django>=1.5.0,<=1.5.9"
Thanks! Basically the answer is that I wasn't aware of the possibility of those things and have tried to keep it as simple as possible.
I made a skeleton project for me as I couldn't find a decent guide for deploying django static's on s3. I fumbled my way through it and corrected and formalised it later so I could do it again. The https://github.com/rdegges/django-skel project looks really useful as it contains everything and more you might want to use for a django project (and more), though not yet knowing what magic's it's doing I wouldn't yet jump to use.
1. That is very neat. I hope to try and copy that.
2. Noted. I didn't realise the need, as python 2.7.4 is the default. Can see it causing headaches if this were to change however.
3. Why is it dangerous? I preferred to keep the version numbers so it would always just install latest versions without any extra effort.
I made a skeleton project for me as I couldn't find a decent guide for deploying django static's on s3. I fumbled my way through it and corrected and formalised it later so I could do it again. The https://github.com/rdegges/django-skel project looks really useful as it contains everything and more you might want to use for a django project (and more), though not yet knowing what magic's it's doing I wouldn't yet jump to use.
1. That is very neat. I hope to try and copy that.
2. Noted. I didn't realise the need, as python 2.7.4 is the default. Can see it causing headaches if this were to change however.
3. Why is it dangerous? I preferred to keep the version numbers so it would always just install latest versions without any extra effort.
The reason it's dangerous is that packages can introduce backwards incompatible changes. If you have a project that you don't work on for a few weeks/months and then need to fix a bug, you're going to have to spend additional cycles tracking down the issue and fixing either the backwards incompatible change, or adding a specific version to your requirements file anyway.
I like to try and scope the package versions at the level the maintainer promises not to introduce breakages at.
I like to try and scope the package versions at the level the maintainer promises not to introduce breakages at.
For dealing with packages such as python-psycopg2 and python-mysql, your best bet is to use --system-site-packages.
This allows the virtualenv to depend on packages provided by your distro instead of trying to compile its own. Wherever possible, I try to do this with Python packages that require a C compiler to make deployment a little easier and reduce dependencies.
Personally, I'm not a fan of Fabric and I much prefer configuration management (Salt, Chef, Puppet) because of the added flexibility you get. About the most I use Fabric for is logging into the Salt Master and kicking off a deployment process.
This allows the virtualenv to depend on packages provided by your distro instead of trying to compile its own. Wherever possible, I try to do this with Python packages that require a C compiler to make deployment a little easier and reduce dependencies.
Personally, I'm not a fan of Fabric and I much prefer configuration management (Salt, Chef, Puppet) because of the added flexibility you get. About the most I use Fabric for is logging into the Salt Master and kicking off a deployment process.
Absolutely not. An added bit of compile time is the least of your worries and if you really don't want to compile stuff on production, look at packaging your virtualenv as a compiled deb/rpm.
Never ever ever mix/depend upon system packages in your virtualenv. Ever.
Never ever ever mix/depend upon system packages in your virtualenv. Ever.
You're always depending on system packages somewhere. Compiling isn't the only reason - sometimes it's necessary just to make the damn thing work.
Some examples:
* PIL - Using Pip's version on Ubuntu, you have to hack your directory structure and re-compile the whole thing or PIL doesn't compile in support for your image libraries.
* psycopg2 - This is hit and miss (OS X comes to mind), but depending on the system version guarantees that it has been tested to work with the specific version of Postgres on your system.
* Crypto libraries - Broken packages have ended up on Pip and just cause a bloody headache. The system package is, again, tested and works.
Compiling a DEB/RPM would be a good idea, but it's not a well supported system at all. If there were a simple pip compile --type=deb or something, we would be all over it.
Some examples:
* PIL - Using Pip's version on Ubuntu, you have to hack your directory structure and re-compile the whole thing or PIL doesn't compile in support for your image libraries.
* psycopg2 - This is hit and miss (OS X comes to mind), but depending on the system version guarantees that it has been tested to work with the specific version of Postgres on your system.
* Crypto libraries - Broken packages have ended up on Pip and just cause a bloody headache. The system package is, again, tested and works.
Compiling a DEB/RPM would be a good idea, but it's not a well supported system at all. If there were a simple pip compile --type=deb or something, we would be all over it.
PIL - Don't use PIL, use Pillow.
Psycopg2 - Well, it's just a client library - surely you will be making sure your client matches your server's version (whether that's local or a clust or whatever)? Sure your system will likely need Postgres headers installed, but why is that an issue? Actually, let me just check that. e: Yeah, you need libpq-dev installed, but I fail to see the point here? You need to make sure you have python-dev and a correct version of Python installed, too.
PyCrypto - Hmm, not had a problem with this hugely.
What I was suggesting doesn't involve a deb of individual packages, but a deb of the entire app/virtualenv.
Psycopg2 - Well, it's just a client library - surely you will be making sure your client matches your server's version (whether that's local or a clust or whatever)? Sure your system will likely need Postgres headers installed, but why is that an issue? Actually, let me just check that. e: Yeah, you need libpq-dev installed, but I fail to see the point here? You need to make sure you have python-dev and a correct version of Python installed, too.
PyCrypto - Hmm, not had a problem with this hugely.
What I was suggesting doesn't involve a deb of individual packages, but a deb of the entire app/virtualenv.
Here are a few other Django packages that I haven't seen mentioned in this thread:
- Gargoyle: Feature switches in Django https://github.com/disqus/gargoyle
- webassets and django-assets: For minifying static assets
- raven + sentry for logging. Sentry is the server, so actually can be used with any project, not only Django.
- django_extensions: Provides some helper management commands, for example manage.py shell_plus which automatically imports your models
- dbbackup for backing up your db to S3 or Dropbox
- Gargoyle: Feature switches in Django https://github.com/disqus/gargoyle
- webassets and django-assets: For minifying static assets
- raven + sentry for logging. Sentry is the server, so actually can be used with any project, not only Django.
- django_extensions: Provides some helper management commands, for example manage.py shell_plus which automatically imports your models
- dbbackup for backing up your db to S3 or Dropbox
Interesting post. I decided to use Django for our startup because Python was a requirement, we're planning to have a lot of customers, and Django seems like the most appropriate choice for large scale apps. While I've mostly used non-rel databases, I'll be using postgresql as a database, because it seems to be highly compatible with Django, because Heroku makes scaling it easy, and because it's a good occasion to learn some RDBMS.
Using S3 for static media makes perfect sense, but does anyone have information on S3 when your contributors (users in the admin) are uploading lots of images for via something like django-filebrowser or a simple zipfile upload (that extracts the images into a folder)? I haven't seen anything like this in any examples.
This is supported quite well via django-storages. Unless the libs you're using are doing something manual instead of using Django's built-in file handling, this will neatly override the local-filesystem defaults and send all of your content to S3.
http://django-storages.readthedocs.org/en/latest/
http://django-storages.readthedocs.org/en/latest/
You can use S3's CORS support to allow your users to upload directly to S3 without even touching your server. Downside is that it can be hard to debug and it requires your users to use JavaScript -- but otherwise it's a big win.
See http://philfreo.com/blog/how-to-allow-direct-file-uploads-fr... for a Python example.
See http://philfreo.com/blog/how-to-allow-direct-file-uploads-fr... for a Python example.
I think the standard method of dealing with this is to have your users upload the items to the local web server and then have some sort of async task or cron job responsible for moving the files to S3 later on.
Of course this means you'll have to keep track of where the uploaded file is currently located, but if you ever want to scale to multiple web servers you'll need to do that anyway.
Of course this means you'll have to keep track of where the uploaded file is currently located, but if you ever want to scale to multiple web servers you'll need to do that anyway.
Yea - this is how I've been doing it. The server that gets the upload does some processing and then moves the content over to S3 and adds an entry to the database that references it. It also forces you to write stateless code which is nice bonus.
I'll add that you should read Two Scoops of Django at http://django.2scoops.org
As someone who is thinking about doing a python project, should I learn Django or just go with Flask? Any thoughts on those two?
Neither! Actually, as the others have said it completely depends on what you're making.
Django is great if you're making a standard cms type website. News feed, blog, etc. There's a bit of a learning curve but it gives you a lot of the scaffolding out of the box (and there are loads of examples to learn from). It has the biggest community (I guess?) so you'll always be able to find answers / libraries / sample code.
Use Flask if you really want to get a website up and running with half a dozen lines of code. You'll have to build everything else yourself; admin, authentication, etc etc - though there are good extensions for a lot of things too.
Thinking about it - if you don't know either of them, spend a day with Flask, by the end of it you should have a fairly good idea of what you get with it. Then run through the Django tutorials. It'll take longer but you'll be impressed with how quickly you can get something fairly complete up and running.
For me, I lean towards Flask (heavily). When you think about what Django does - it's basically a web container with a db/model layer and sensible scaffolding for the views (front and admin). I prefer to use SQLAlchemy (if I'm working with a relational db). In terms of the admin I think that the days of writing admin grid and edit screens that post back and forth to the server are numbered (I use REST and Angular now) - so that's not a bit of Django I use either. By the time you've ripped all that out you may as well save yourself all the config and just use Flask. Having said that - LEARN THEM BOTH. At least just a little bit so you can see for yourself where each one shines.
These days I approach it completely differently. My last two projects have involved a fair amount of complex business logic (and processing work). I've written the libraries I needed first as standalone packages, only introducing Flask as a final step. The web wrapper is just there to provide a web interface (mostly an api) access the db layer and glue it all to my libraries. I'd recommend this approach as it stops you from thinking along the lines of "ok, I'll need to create 3 Django apps, here are my models for each, should I put this code in the models.py or the views.py?" Instead you concentrate on making the real python code to do the heavy lifting, and you architect that sensibly without having to worry about how the web framework would normally prescribe it to be done. You'll end up with far more portable code in the long run.
Django is great if you're making a standard cms type website. News feed, blog, etc. There's a bit of a learning curve but it gives you a lot of the scaffolding out of the box (and there are loads of examples to learn from). It has the biggest community (I guess?) so you'll always be able to find answers / libraries / sample code.
Use Flask if you really want to get a website up and running with half a dozen lines of code. You'll have to build everything else yourself; admin, authentication, etc etc - though there are good extensions for a lot of things too.
Thinking about it - if you don't know either of them, spend a day with Flask, by the end of it you should have a fairly good idea of what you get with it. Then run through the Django tutorials. It'll take longer but you'll be impressed with how quickly you can get something fairly complete up and running.
For me, I lean towards Flask (heavily). When you think about what Django does - it's basically a web container with a db/model layer and sensible scaffolding for the views (front and admin). I prefer to use SQLAlchemy (if I'm working with a relational db). In terms of the admin I think that the days of writing admin grid and edit screens that post back and forth to the server are numbered (I use REST and Angular now) - so that's not a bit of Django I use either. By the time you've ripped all that out you may as well save yourself all the config and just use Flask. Having said that - LEARN THEM BOTH. At least just a little bit so you can see for yourself where each one shines.
These days I approach it completely differently. My last two projects have involved a fair amount of complex business logic (and processing work). I've written the libraries I needed first as standalone packages, only introducing Flask as a final step. The web wrapper is just there to provide a web interface (mostly an api) access the db layer and glue it all to my libraries. I'd recommend this approach as it stops you from thinking along the lines of "ok, I'll need to create 3 Django apps, here are my models for each, should I put this code in the models.py or the views.py?" Instead you concentrate on making the real python code to do the heavy lifting, and you architect that sensibly without having to worry about how the web framework would normally prescribe it to be done. You'll end up with far more portable code in the long run.
What is your opinion about web2py? I started exploring it a week back, and it seems to have a lot of built in functionality. I thought it would be the easiest web framework to get started with (since it was originally built to help teach students about web programming). However, it seems to have a vast number of components, and it looks like learning all of it would entail almost the same effort as learning Django...
Not the person you replied to, but my two cents.
The creator and lead developer, Massimo Di Pierro, is incredibly active and always willing to lend a hand with whatever problem you have. I've seen him around reddit (not sure if he's on HN) and even when he's met with the inevitable haters, he's always been friendly and cordial.
Just like you've said, web2py was originally made to teach people how to build websites, which really shows in its architecture. It forgoes traditional Python conventions ("explicit is better than implicit" being a major one) in order to be more beginner-friendly. Some of the more experienced developers find this annoying, but if you're coming from PHP or a fresh background, it can be much easier to get started. Just take a look at the overview on Wikipedia[0]. It gives a good outline.
This is coming from someone who doesn't even use web2py. I prefer Flask for personal projects and Django for work/group things, but I would whole-heartedly recommend web2py. Run through the book[1] and, if you don't like it, give the Django tutorial[2] a shot. If that doesn't mesh with you, there's Flask[3], Pyramid[4], TurboGears[5], and CherryPy[6]. (The last two I would not recommend for beginners, but they're still good!)
[0]: http://en.wikipedia.org/wiki/Web2py#Overview
[1]: http://web2py.com/book
[2]: https://docs.djangoproject.com/en/1.5/intro/tutorial01/
[3]: http://flask.pocoo.org/docs/
[4]: http://docs.pylonsproject.org/projects/pyramid/en/1.4-branch...
[5]: http://turbogears.readthedocs.org/en/latest/
[6]: http://docs.cherrypy.org/stable/index.html
The creator and lead developer, Massimo Di Pierro, is incredibly active and always willing to lend a hand with whatever problem you have. I've seen him around reddit (not sure if he's on HN) and even when he's met with the inevitable haters, he's always been friendly and cordial.
Just like you've said, web2py was originally made to teach people how to build websites, which really shows in its architecture. It forgoes traditional Python conventions ("explicit is better than implicit" being a major one) in order to be more beginner-friendly. Some of the more experienced developers find this annoying, but if you're coming from PHP or a fresh background, it can be much easier to get started. Just take a look at the overview on Wikipedia[0]. It gives a good outline.
This is coming from someone who doesn't even use web2py. I prefer Flask for personal projects and Django for work/group things, but I would whole-heartedly recommend web2py. Run through the book[1] and, if you don't like it, give the Django tutorial[2] a shot. If that doesn't mesh with you, there's Flask[3], Pyramid[4], TurboGears[5], and CherryPy[6]. (The last two I would not recommend for beginners, but they're still good!)
[0]: http://en.wikipedia.org/wiki/Web2py#Overview
[1]: http://web2py.com/book
[2]: https://docs.djangoproject.com/en/1.5/intro/tutorial01/
[3]: http://flask.pocoo.org/docs/
[4]: http://docs.pylonsproject.org/projects/pyramid/en/1.4-branch...
[5]: http://turbogears.readthedocs.org/en/latest/
[6]: http://docs.cherrypy.org/stable/index.html
I've never used it so I really can't comment, sorry. There are a few that I've only skimmed the documentation for to see how they compare to Django / Flask. Bottle for example - when I look through the examples it seems so close to Flask that I don't feel I need to look any closer at it (so I may be missing out on something there).
Web2py does badly with pycharm. So I don't use it.
> In terms of the admin I think that the days of writing admin grid and edit screens that post back and forth to the server are numbered (I use REST and Angular now) - so that's not a bit of Django I use either.
My feeling too. Are there any projects which are starting to create the boilerplate for this? I'm new to Angular, so looking for where to get started...
My feeling too. Are there any projects which are starting to create the boilerplate for this? I'm new to Angular, so looking for where to get started...
Not that I've seen - I'm building a specific admin interface for one of my projects at the moment using this style (but I'm not making it generic). Architecturally the REST api is on one domain, the admin system is static angular on another domain and the front of the site (/sites - 15,000 of them) is a thin layer that makes calls back to the api.
It should be possible to build a completely generic frontend that just has a grid view and an edit view that you could wire to any REST backend. Then you could use flask-restful or Django Rest Framework or whatever else you wanted.
If you were going to go that route there's an angular directive for doing grids that would get you half way there: http://angular-ui.github.io/ng-grid/#/examples
Were I still building CMSs that's the approach I would be looking into.
It should be possible to build a completely generic frontend that just has a grid view and an edit view that you could wire to any REST backend. Then you could use flask-restful or Django Rest Framework or whatever else you wanted.
If you were going to go that route there's an angular directive for doing grids that would get you half way there: http://angular-ui.github.io/ng-grid/#/examples
Were I still building CMSs that's the approach I would be looking into.
To be fair to Flask, there are lots of libraries you can add to it (Flask-Script, Flask-WTF, Flask-Auth, Flask-Admin) that replicate pretty much all of the functionality of Django, but in a more modular fashion. For example, I can use Flask-Auth regardless of if I've chosen Flask-SQLAlchemy.
Indeed - I just didn't want to push Flask too much but that's what I use and I bring in the libraries you mention as and when they're required.
Flask is so simple that it's definitely worth picking that up regardless (in any case, I think it's well done and worth having in your arsenal).
However, for large web projects, Django beats it hands down. Doing similar work with Flask would require so much wheel-reinvention that it'd be hard to justify your time or your client's money when it's all built quite well already.
Of course it's not perfect, and it's not for everything. But it's a great general-purpose, medium-performance web framework. It becomes a pain on very high-traffic sites (perhaps more so than JVM-based solutions and on par with scaling Ruby frameworks), but if you use it in the right place and/or team up with systems people that know their stuff, it's a powerful tool for quickly building complex websites in a maintainable and intuitive way.
However, for large web projects, Django beats it hands down. Doing similar work with Flask would require so much wheel-reinvention that it'd be hard to justify your time or your client's money when it's all built quite well already.
Of course it's not perfect, and it's not for everything. But it's a great general-purpose, medium-performance web framework. It becomes a pain on very high-traffic sites (perhaps more so than JVM-based solutions and on par with scaling Ruby frameworks), but if you use it in the right place and/or team up with systems people that know their stuff, it's a powerful tool for quickly building complex websites in a maintainable and intuitive way.
Good point, I'd not thought about the JVM based Ruby solutions which can be used. If they'd give more Scala-like performance..
IronPython?
IronPython?
I've been building (more or less solo) a web app marketplace for our startup for about a year now. I love Python, and had dabbled in Django and Flask prior to this, but built nothing with either to any significant degree. I went with Django because I was able to get something up relatively quickly, and I feel like that was a solid decision. I also am relatively confident in my ability to move towards Flask in the longer term if the Django app becomes too much, but I haven't experienced that yet.
Also Heroku has been mostly a godsend, as I haven't had to spend nearly any time on sysadmin stuff. One thing that caught me early on, not sure if it's a bug or what, but in Heroku, if you use git URLs for some of your packages in pip requirements (which is useful if you need to customize a package or more frequently fix a bug and can't wait for it to be pushed to pypi), Heroku's package cache has trouble knowing when to update unless you a) change the package version in its setup.py (may or may not be a good idea) or b) change the Python version in runtime.txt (kind of a pain as generally that means two deploys for an update, bump down one minor, then back up).
Also Heroku has been mostly a godsend, as I haven't had to spend nearly any time on sysadmin stuff. One thing that caught me early on, not sure if it's a bug or what, but in Heroku, if you use git URLs for some of your packages in pip requirements (which is useful if you need to customize a package or more frequently fix a bug and can't wait for it to be pushed to pypi), Heroku's package cache has trouble knowing when to update unless you a) change the package version in its setup.py (may or may not be a good idea) or b) change the Python version in runtime.txt (kind of a pain as generally that means two deploys for an update, bump down one minor, then back up).
If you are new to Python and just want to have a working project instead of tinkering, then Django is better suited. Flask is for more advanced uses where you need to have more control or be able to integrate components you choose (like SQLAlchemy instead of Django ORM, or a NoSQL DB).
I disagree somewhat.. I'm a hobbyist at best when it comes to programming. Like I've never actually shipped anything in my life. I recently decided to put an end to that to solve a problem for a family member's business.
They just wanted a simple web form, where the sales people could add notes throughout the day and then automatically email those notes to the owner at the end of the day.
I first tried Django and was quickly overwhelmed.. I switched to Flask and with the help of flask-login, flask-wtforms, and flask-mail, it only took me a long weekend to get it up and running.
Just a personal anecdote, but Flask was much easier for me.
They just wanted a simple web form, where the sales people could add notes throughout the day and then automatically email those notes to the owner at the end of the day.
I first tried Django and was quickly overwhelmed.. I switched to Flask and with the help of flask-login, flask-wtforms, and flask-mail, it only took me a long weekend to get it up and running.
Just a personal anecdote, but Flask was much easier for me.
Flask is definitely easier... it also offers far less. Which is fine in some circumstances -- perhaps many -- but the fact that Django has a steeper learning curve is due to the fact that it covers a lot more ground in terms of things many if not most web applications need.
This doesn't make it better, but it also doesn't make it worse.
This doesn't make it better, but it also doesn't make it worse.
You might also want to take a look at Pyramid. There's pros and cons for each.
I absolutely love pyramid, but I found the learning curve for it to be much higher than flask or django. If pyramid had a few more things working in a simple way out of the box I would go for it every time.
I suppose it depends on what your goals are. If your goal is just to crank out some Python, I'd go with Flask (or my personal favorite Bottle).
I think it really depends on what the project is.
If it's relatively simple and you don't really want to deal with user registration/databases I'd say definitely go with Flask. Django does a pretty good job of those two but until it "clicks" you end up being confused.
If it's relatively simple and you don't really want to deal with user registration/databases I'd say definitely go with Flask. Django does a pretty good job of those two but until it "clicks" you end up being confused.
Funny that I wrote that Flask is for advanced usage and you wrote that Flask is for simple usage. If the project is really simple - no database, no declarative forms etc. - then Flask will be better. But if you go to "medium complexity" Django gives you ready components which you need to integrate in Flask (SQLAlchemy, WTForms etc.). You also need to invent your own project structure, and Django already has a hardcoded one. When the project complexity is "high" then Django starts to be too limited/"hardcoded" and flexibility/minimalism of Flask starts to shine.
Haha yea you're right! Funny how my bias trickled through. I haven't done too much Flask customization and only use it for the one off simple project that just need a web interface.
Django is definitely more in the range of "we want you to write" this way but I've found it to be at least a bit more customizable than RoR. Flask is on the other extreme which lets you do whatever you want.. but you're the one who has to do it.
Django is definitely more in the range of "we want you to write" this way but I've found it to be at least a bit more customizable than RoR. Flask is on the other extreme which lets you do whatever you want.. but you're the one who has to do it.
I actually disagree. Flask does not necessarily shine with projects of high complexity. The framework designers themselves say as much:
"However Flask is just not designed for large applications or asynchronous servers. Flask wants to make it quick and easy to write a traditional web application."[1]
Flask is excellent when you just don't need a bunch of the stuff in Django, and don't need the batteries to be included. It's not so much that Flask is suitable to high-complexity apps, but more that it's suited to highly custom web apps. Building a music sharing app based on jQuery mobile and backed by Mongo? You're better off with Flask. But at scale you're going to end up doing something custom.
Django is highly suitable if you need to manage users, have a variety of entities that lend themselves to the admin dashboard (more broadly, that the admin/public app metaphor even makes sense in the first place), and find a plug-and-play package system compelling.
They are both great frameworks, but "complexity" (advanced/simple) is the wrong axis along which to evaluate them.
[1]http://flask.pocoo.org/docs/design/
"However Flask is just not designed for large applications or asynchronous servers. Flask wants to make it quick and easy to write a traditional web application."[1]
Flask is excellent when you just don't need a bunch of the stuff in Django, and don't need the batteries to be included. It's not so much that Flask is suitable to high-complexity apps, but more that it's suited to highly custom web apps. Building a music sharing app based on jQuery mobile and backed by Mongo? You're better off with Flask. But at scale you're going to end up doing something custom.
Django is highly suitable if you need to manage users, have a variety of entities that lend themselves to the admin dashboard (more broadly, that the admin/public app metaphor even makes sense in the first place), and find a plug-and-play package system compelling.
They are both great frameworks, but "complexity" (advanced/simple) is the wrong axis along which to evaluate them.
[1]http://flask.pocoo.org/docs/design/
The "large applications or asynchronous servers" bit might be better understood in comparison to the monolithic versus micro kernel debate. Flask is well suited for building complex applications that can be decomposed into simple moving parts with a clear separation of concerns. (My personal bias is that all complex applications can and should be developed this way.)
The documented limitation you reference is based on the WSGI server being used. "If your server uses some kind of concurrency that is not based on threads or greenlets, Flask will no longer be able to support these global proxies."[0] As nginx+gunicorn is becoming more popular, this isn't really an issue at all.
[0]http://flask.pocoo.org/docs/becomingbig/#scale-like-a-pro
The documented limitation you reference is based on the WSGI server being used. "If your server uses some kind of concurrency that is not based on threads or greenlets, Flask will no longer be able to support these global proxies."[0] As nginx+gunicorn is becoming more popular, this isn't really an issue at all.
[0]http://flask.pocoo.org/docs/becomingbig/#scale-like-a-pro
"However Flask is just not designed for large applications or asynchronous servers. Flask wants to make it quick and easy to write a traditional web application."
Armin has said before that he needs to update/clarify that statement -- as dvanduzer said, it's not really relevant anymore.
Armin has said before that he needs to update/clarify that statement -- as dvanduzer said, it's not really relevant anymore.
I really like Flask but I still go with Django when I need the admin.
How well does Django handle complex forms? The kind where "if x = y show fields p-r (and make them required), if x = z jump the next screen)?
I've just implemented one of these in PHP using the 'old' Symfony 1.4 and it was painful. I ended up hardcoding most of the frontend, and writing ridiculous modifications for "add another row" and validation.
Is Django any better at that? Complex forms have been my pain point for over 11 years, and if a framework handles them... I'm in!
I've just implemented one of these in PHP using the 'old' Symfony 1.4 and it was painful. I ended up hardcoding most of the frontend, and writing ridiculous modifications for "add another row" and validation.
Is Django any better at that? Complex forms have been my pain point for over 11 years, and if a framework handles them... I'm in!
I've recently (2-3 months) switched to Python/Django and my experience so far has been very positive.
Multi-step forms can be implemented in Django using Form wizards [1]. In order to optionally show fields or make them required I think you could build it easily using the WizardView.get_form method [2]. The functionality to skip steps is built-in and it's very easy implement using conditional views [3].
In conclusion, it's fairly easy to implement the functionality you described using only the core framework without the need to rely on external packages.
[1] - https://docs.djangoproject.com/en/dev/ref/contrib/formtools/... [2] - https://docs.djangoproject.com/en/dev/ref/contrib/formtools/... [3] - https://docs.djangoproject.com/en/dev/ref/contrib/formtools/...
Multi-step forms can be implemented in Django using Form wizards [1]. In order to optionally show fields or make them required I think you could build it easily using the WizardView.get_form method [2]. The functionality to skip steps is built-in and it's very easy implement using conditional views [3].
In conclusion, it's fairly easy to implement the functionality you described using only the core framework without the need to rely on external packages.
[1] - https://docs.djangoproject.com/en/dev/ref/contrib/formtools/... [2] - https://docs.djangoproject.com/en/dev/ref/contrib/formtools/... [3] - https://docs.djangoproject.com/en/dev/ref/contrib/formtools/...
Interesting, thanks for sharing your experience!
Have you seen anything saying it helps to build adaptive forms like [1]?
[1] - http://screencast.com/t/SutggDU5
Have you seen anything saying it helps to build adaptive forms like [1]?
[1] - http://screencast.com/t/SutggDU5
I'm not aware of any library which could help you build these kind of adaptive forms automatically.
However, the way I would go about building something like that is by using the form wizard (mentioned in my previous comment) with a combination of forms/modelforms [1] and formsets [2]. Formsets are used in situations when you need to display the same form multiple times on the same page (like the multiple addresses/benefits income in your example). However, you would have to implement all the client side interaction yourself since Django does not provide this by default (there are some libraries which are supposed to help with this but I haven't tried them myself [3]).
One thing worth mentioning is that by default, the form wizard only allows you to display either a Form, or a Formset in a step/page. However, there is a ticket open for this [4] which links to a helper class: form_container.py [5] which helps overcome this problem. Hopefully, this will be included in future versions of Django.
Hope this helps.
[1] - https://docs.djangoproject.com/en/dev/topics/forms/modelform... [2] - https://docs.djangoproject.com/en/dev/topics/forms/formsets/ [3] - https://code.google.com/p/django-dynamic-formset/ [4] - https://code.djangoproject.com/ticket/18830 [5] - https://code.djangoproject.com/attachment/ticket/18830/form_...
However, the way I would go about building something like that is by using the form wizard (mentioned in my previous comment) with a combination of forms/modelforms [1] and formsets [2]. Formsets are used in situations when you need to display the same form multiple times on the same page (like the multiple addresses/benefits income in your example). However, you would have to implement all the client side interaction yourself since Django does not provide this by default (there are some libraries which are supposed to help with this but I haven't tried them myself [3]).
One thing worth mentioning is that by default, the form wizard only allows you to display either a Form, or a Formset in a step/page. However, there is a ticket open for this [4] which links to a helper class: form_container.py [5] which helps overcome this problem. Hopefully, this will be included in future versions of Django.
Hope this helps.
[1] - https://docs.djangoproject.com/en/dev/topics/forms/modelform... [2] - https://docs.djangoproject.com/en/dev/topics/forms/formsets/ [3] - https://code.google.com/p/django-dynamic-formset/ [4] - https://code.djangoproject.com/ticket/18830 [5] - https://code.djangoproject.com/attachment/ticket/18830/form_...
I would say that it's not much better out of the box but you can combine forms, formsets, custom validation, 3rd party helpers like django-cripsy-forms and support with something like KnockoutJS in the browser and voila :)
It would be great if there was a ready-made add-on combining all these areas but I don't know any personally. I also suspect think that writing such a generic helper might not be a great idea as everyone needs something a bit different, so using the mentioned lower level components and combining them on your own could be the best way to go here.
It would be great if there was a ready-made add-on combining all these areas but I don't know any personally. I also suspect think that writing such a generic helper might not be a great idea as everyone needs something a bit different, so using the mentioned lower level components and combining them on your own could be the best way to go here.
If I'm understanding you correctly, you want to have the fields on the form depend on the contents of what was entered on the previous form e.g. if user selects name, ask for name, otherwise ask for address.
In that case, I tend to use class-based views and set different forms, templates etc. depending on those values. If the form is different enough, it's easier just to send them to a completely different URL.
In that case, I tend to use class-based views and set different forms, templates etc. depending on those values. If the form is different enough, it's easier just to send them to a completely different URL.
Not quite - that happens sometimes and I use logic in the controller (ugh) to switch the form class.
This is what I mean: an adaptive form http://screencast.com/t/SutggDU5
The server-side validation has to adapt depending on the data coming from the client-side, and varies a lot.
Perhaps others don't build forms like this much and so web frameworks don't deal with them; I get one like this every few months!
This is what I mean: an adaptive form http://screencast.com/t/SutggDU5
The server-side validation has to adapt depending on the data coming from the client-side, and varies a lot.
Perhaps others don't build forms like this much and so web frameworks don't deal with them; I get one like this every few months!
Ah, I see now. For certain cases e.g. repeating the same form over and over, you can use formsets, but that doesn't work when you have two different forms.
I still think a solution is to use multiple forms and form types rather than one big form. Perhaps you can prefix the field names on the template and use that to determine which form they go to inside the view. It makes your logic a little simpler in any case.
I still think a solution is to use multiple forms and form types rather than one big form. Perhaps you can prefix the field names on the template and use that to determine which form they go to inside the view. It makes your logic a little simpler in any case.
Note that one thing you can do is build smaller forms and mix them up. There's no issue with mixing up 3 or 4 smaller (django) forms in the same (HTML) form, quite the opposite.
Re server-side validation that adapts to the incoming data, I think it's easier to write a small mixin that fits your needs than to use a generic library (assuming there's one).
Very much this - though I've experienced this in other stacks.
Would love to know the best way people handle this.
Would love to know the best way people handle this.
Great short and concise article.
I'm in the process of writing a pet project using Django and would really appreciate your insights on what packages you find useful (as mentioned at the bottom of your post).
I'm in the process of writing a pet project using Django and would really appreciate your insights on what packages you find useful (as mentioned at the bottom of your post).
From someone who wasn't asked but has input... ;)
It's not a Python package in the sense that you might mean, but using virtualenv is absolutely fundamental to good Python development (at least the kind that lets you keep your sanity).
South is also exceptional; however, the developer recently ran a successful Kickstarter campaign dedicated to creating an improved version intended to be merged with Django itself, so that's something to watch for.
Django-CMS has proven invaluable for many of my projects and clients, though it's very heavy and more of a project-tacked-on-to-your-project than just a package. It's great to know, though, and very powerful.
django-registration can be useful if you're into full-on user profile usage, though many sites don't actually need this. There's a separate project that provides a set of default templates for this package, and it's almost more useful than django-registration itself.
livesettings is an outgrowth of django-cms that's pretty useful when you need what would otherwise be settings.py constants that can be modified at runtime without a deployment and server restart. Be careful, though, as overuse of this means you might have design issues.
feedparser is a very, very easy way to consume XML feeds.
If you want to parse something in a very user-friendly way, BeautifulSoup is hard to beat. If you're into the nitty gritty and/or need serious performance, lxml serves the same purpose but kicks far more ass.
If you need to interface with Amazon's AWS environments (EC2, S3, etc.), boto is a brilliant library.
These are more Python-in-general than Django-specific, but that's often a good thing.
It's not a Python package in the sense that you might mean, but using virtualenv is absolutely fundamental to good Python development (at least the kind that lets you keep your sanity).
South is also exceptional; however, the developer recently ran a successful Kickstarter campaign dedicated to creating an improved version intended to be merged with Django itself, so that's something to watch for.
Django-CMS has proven invaluable for many of my projects and clients, though it's very heavy and more of a project-tacked-on-to-your-project than just a package. It's great to know, though, and very powerful.
django-registration can be useful if you're into full-on user profile usage, though many sites don't actually need this. There's a separate project that provides a set of default templates for this package, and it's almost more useful than django-registration itself.
livesettings is an outgrowth of django-cms that's pretty useful when you need what would otherwise be settings.py constants that can be modified at runtime without a deployment and server restart. Be careful, though, as overuse of this means you might have design issues.
feedparser is a very, very easy way to consume XML feeds.
If you want to parse something in a very user-friendly way, BeautifulSoup is hard to beat. If you're into the nitty gritty and/or need serious performance, lxml serves the same purpose but kicks far more ass.
If you need to interface with Amazon's AWS environments (EC2, S3, etc.), boto is a brilliant library.
These are more Python-in-general than Django-specific, but that's often a good thing.
Thanks for the reply.
While all suggestion are useful the biggest take away for me is livesettings. TBH I can't think of how I would use it at the moment but I can see that it could come in handy later.
While all suggestion are useful the biggest take away for me is livesettings. TBH I can't think of how I would use it at the moment but I can see that it could come in handy later.
So a stupid question about virtualenv for you. Why does it default to full paths for everything?
Honestly, I have no idea. I haven't ever tried to move a virtualenv, and it's not something that's supposed to be part of your repo (you should build your env separately on every box; it's intended to encapsulate dependencies from the system, not to transfer them around with your code).
If your system gets updated/upgraded to the point where your virtualenv breaks, rebuild it from scratch; all it should take is a bit of time.
[edit: I am not a developer involved with virtualenv, and the statement about what it's intended for probably shouldn't be quite so strong, as this is only my impression of it and the way I use it.]
If your system gets updated/upgraded to the point where your virtualenv breaks, rebuild it from scratch; all it should take is a bit of time.
[edit: I am not a developer involved with virtualenv, and the statement about what it's intended for probably shouldn't be quite so strong, as this is only my impression of it and the way I use it.]
I actually don't "know" either. But I suspect it has to do with knowing which python to use (system generally) and if the user wants to use global packages as well.
checkout virtualenvwrapper (http://virtualenvwrapper.readthedocs.org/en/latest/install.h...) as well for handling virtualenvs.
checkout virtualenvwrapper (http://virtualenvwrapper.readthedocs.org/en/latest/install.h...) as well for handling virtualenvs.
+1, I definitely should have mentioned this. virtualenvwrapper makes it vastly more convenient.
Author here - thanks for mentioning this! I've never used it but will give it a go.
[deleted]
Nice post thanks - I'm about to tackle a fairly large Django project so it's always nice to see some best practice guidelines
celeryd is also good for scheduling tasks with either single or periodic execution.
http://docs.celeryproject.org/en/latest/getting-started/firs...
You can also use it in a typical task queue role. It will pickle or serialize the Python objects that accompany tasks for transmission over the wire.