How to take on a large Django project(gun.io)
gun.io
How to take on a large Django project
http://gun.io/blog/how-to-take-on-a-large-Django-project/
20 comments
I cleaned up the audio a little bit (using Audacity and Auphonic) and uploaded it here: http://www.youtube.com/watch?v=QVNC3q_Dqg4
nice! i'll check it out later
Is it just me or does it look like the author of the gun.io talk copied liberally from the Boston Python talk?
i'm the author of the gun.io post and i have not seen the Boston Python talk. i just added it to my list to watch. the majority of the content came from my own experiences. i also started a reddit discussion to facilitate communication among other django developers - http://www.reddit.com/r/django/comments/1h6msl/django_best_p...
I still find the project/app distinction in Django pretty confusing. I wish there were just the app standalone, and some way to join apps together if you really need to.
This is undeniably one of the more unclear parts of Django. I feel that the main issue is that an 'app' can either be a reusable, coherent collection of code - what would more normally be considered a library, or it can be a simple distinction between different areas of your website (the 'project', in Django parlance.) The first type can either be written as part of the project, or be installed from pypi, whereas the second type only ever really exists as part of the project. A project is simply a collection of apps, with configuration, that makes up a particular site.
Personally I think it's entirely possible to work quite happily with both types of apps, once you realise the distinction. My typical approach would be to start (as mentioned elsewhere by others) with one project-specific app in the project (most likely called something generic like 'core') just to get off the ground. For some projects this may be all you ever need. All other apps involved are installed from pypi (or wherever) and provide some generic functionality that you are building off/with (for example, I imagine most people install django-debug-toolbar pretty much at the same time as setting up their project.)
Once a project grows a little, however, I think there's value gained in breaking out into separate apps. They can still have dependencies on other apps (for example relationships in the models) - it's just a way of breaking up the code into more manageable sections. Once you've done this for a few projects, or after working on a large project for a while, you might start to notice common patterns being used across the apps, or some block of functionality that is used all over the place, but doesn't really belong in any one of them - perhaps this might be worth having it's own app for. This might be the sort of app that you end up tidying up and releasing to pypi - but I see far too many people getting hung up on making all apps dependency free and completely generic, when they should be worrying about it working for their use case first! It's impossible to make a reusable and useful tool unless you've had to work through some/many of the possible use-cases (insert premature optimisation quote here.)
Here's an example of this approach, which hopefully makes it slightly clearer:
Imagine you are building a kind of multi-user cms for a particular type of content - your core app to begin with might contain a custom user model (because you're using django 1.5, right ;)) along with, lets say, 3 models for the dynamic content, like a page model, a chunk model, which makes up the page content, and a revisions model that helps describe the relationship between the two. Obviously all your views etc. all belong in the one app. Alongside your core app, you might install some 3rd party apps like django-guardian to allow more granular permissions.
Time moves on and you might start adding functionality like organisations. With the new functionality, users can belong to multiple organisations, and have several different roles, allowing them to edit other people's content or administer the membership of an organisation. Perhaps there is a review process certain content has to go through before it's allowed to be published and on top of that you want to have detailed activity streams to track peoples actions ("user x did action y to item z".) All of a sudden you have more models ( an 'organisation' model, a 'review' model and a collection of related models to do with the activity logging ) as well as some fairly complicated business logic to maintain the permissions system. Of course your views file has at least doubled in size as well as your urls.py file. Realistically this is still manageable but you can see where it's heading. It's at this point I would consider breaking it out into an 'organisations' app (including the organisations, users, and activity models, as well as their logic and views) and a 'content' app, for the pages, chunks, revisions etc. Of course the relationships between the users model and the page model (for example) still exists, but there is a fairly clear separation of interest.
Lets say further on down the road, you've maybe added more functionality, and expanded into another 1-2 apps. All of the users activity is tracked for all the different apps, so you start to think that the activity functionality and models could exist in their own app - just the models and the logic, which can then be tied into all other apps that require it. As you do so, and continue to expand your project, the 'activity-stream' app becomes suitably generic and flexible you feel it might be useful for someone else, and you release it into the wider world!
YMMV of course, this is just based off my experience, but maybe it will help other people getting caught up thinking about the distinctions. Also, I'm nothing to do with the 'django-activity-stream' app, it's just an example :).
Personally I think it's entirely possible to work quite happily with both types of apps, once you realise the distinction. My typical approach would be to start (as mentioned elsewhere by others) with one project-specific app in the project (most likely called something generic like 'core') just to get off the ground. For some projects this may be all you ever need. All other apps involved are installed from pypi (or wherever) and provide some generic functionality that you are building off/with (for example, I imagine most people install django-debug-toolbar pretty much at the same time as setting up their project.)
Once a project grows a little, however, I think there's value gained in breaking out into separate apps. They can still have dependencies on other apps (for example relationships in the models) - it's just a way of breaking up the code into more manageable sections. Once you've done this for a few projects, or after working on a large project for a while, you might start to notice common patterns being used across the apps, or some block of functionality that is used all over the place, but doesn't really belong in any one of them - perhaps this might be worth having it's own app for. This might be the sort of app that you end up tidying up and releasing to pypi - but I see far too many people getting hung up on making all apps dependency free and completely generic, when they should be worrying about it working for their use case first! It's impossible to make a reusable and useful tool unless you've had to work through some/many of the possible use-cases (insert premature optimisation quote here.)
Here's an example of this approach, which hopefully makes it slightly clearer:
Imagine you are building a kind of multi-user cms for a particular type of content - your core app to begin with might contain a custom user model (because you're using django 1.5, right ;)) along with, lets say, 3 models for the dynamic content, like a page model, a chunk model, which makes up the page content, and a revisions model that helps describe the relationship between the two. Obviously all your views etc. all belong in the one app. Alongside your core app, you might install some 3rd party apps like django-guardian to allow more granular permissions.
Time moves on and you might start adding functionality like organisations. With the new functionality, users can belong to multiple organisations, and have several different roles, allowing them to edit other people's content or administer the membership of an organisation. Perhaps there is a review process certain content has to go through before it's allowed to be published and on top of that you want to have detailed activity streams to track peoples actions ("user x did action y to item z".) All of a sudden you have more models ( an 'organisation' model, a 'review' model and a collection of related models to do with the activity logging ) as well as some fairly complicated business logic to maintain the permissions system. Of course your views file has at least doubled in size as well as your urls.py file. Realistically this is still manageable but you can see where it's heading. It's at this point I would consider breaking it out into an 'organisations' app (including the organisations, users, and activity models, as well as their logic and views) and a 'content' app, for the pages, chunks, revisions etc. Of course the relationships between the users model and the page model (for example) still exists, but there is a fairly clear separation of interest.
Lets say further on down the road, you've maybe added more functionality, and expanded into another 1-2 apps. All of the users activity is tracked for all the different apps, so you start to think that the activity functionality and models could exist in their own app - just the models and the logic, which can then be tied into all other apps that require it. As you do so, and continue to expand your project, the 'activity-stream' app becomes suitably generic and flexible you feel it might be useful for someone else, and you release it into the wider world!
YMMV of course, this is just based off my experience, but maybe it will help other people getting caught up thinking about the distinctions. Also, I'm nothing to do with the 'django-activity-stream' app, it's just an example :).
I just put everything in an app called 'core'. That seems to be pretty normal.
Anything non-core obviously goes in a separate app, and usually if the project grows large enough there will be parts where it makes sense to break it apart.
Anything non-core obviously goes in a separate app, and usually if the project grows large enough there will be parts where it makes sense to break it apart.
I like this idea. Can you give us an example of something that wouldn't go in core?
anything reusable that you can package as an external utility (and would upload to pypi if it weren't for IP restrictions).
That's still too abstract for me. Does anyone have a concrete example?
Literally any Django application in the wild. Mezzanine, grappelli, south, etc.
It's purely a problem of terminology. An app is just the name for a component which has a somewhat predefined structure - model, urls, tests, etc. The django framework then loads these components and makes the various part of the structure available within the project. An app, sorry, component can be simply a model or a complete sub-system with templates, views, javascript files, etc. The general idea is to factor out the reusable stuff into components which can then be easily configured for any given project.
You can also use your "project" (effectively: a module with a "settings" module) as an "app" (effectively: a module with a "models" module and a "templatetags" module if you need custom ones) by putting its package name in INSTALLED_APPS and sticking a models.py in it.
Then, if you find a piece which would cleanly separate out into a standalone app, you just have to rip it out, add the extracted app to INSTALLED_APPS and update your imports.
Then, if you find a piece which would cleanly separate out into a standalone app, you just have to rip it out, add the extracted app to INSTALLED_APPS and update your imports.
Think of it like this: The project is your final product, comprised of separate functions from each app. The Project is really the glue holding your apps together.
[deleted]
Very informative post. I am new to python/django and am planning to use some of these code-review points to better understand django as well as my project rather than 'review' it as such.
I'm not sure about using excel to track the project layout, though I'm not sure what tool would I use for such a task. My current method of rapidly using my IDE to traverse between files while building a mental model in my mind is effective but I can retain only about 50% of the information in a week or two.
What do others do?
I'm not sure about using excel to track the project layout, though I'm not sure what tool would I use for such a task. My current method of rapidly using my IDE to traverse between files while building a mental model in my mind is effective but I can retain only about 50% of the information in a week or two.
What do others do?
I am building a stack trace app to analyze the relationship between the urls -> views -> templates in a visual manner
> Pssst! Hackers, sign up here!
just tried to register at gun.io. it doesn't want my email from github auth, then asks for email on their site anyway, but the form doesn't work.
just tried to register at gun.io. it doesn't want my email from github auth, then asks for email on their site anyway, but the form doesn't work.
http://www.youtube.com/watch?v=psCVC9BdgsA