Teaching Software Architecture with GitHub(avandeursen.com)
avandeursen.com
Teaching Software Architecture with GitHub
http://avandeursen.com/2013/12/30/teaching-software-architecture-with-github/
5 comments
Personally I think this is a good starting point: http://gorban.org/post/32873465932/software-architecture-che...
In terms of books, I really like "The Architecture of Open Source Applications" (http://aosabook.org/) which is also mentioned in the blog post. For the enterprise world, "Patterns of Enterprise Application Architecture" (http://martinfowler.com/books/eaa.html) is a good choice.
However, before applying any technical software architecture methods, it is really important to understand what your stakeholders actually want/need. Unfortunately most of the time they don't even know it themselves. In my experience most software projects fail, not because of technical reasons but simply because the original problem hasn't been understood. This is where communication is key. I assume the reason why the authors of the blog post encouraged drawing diagrams and looking at different perspectives/views is to facilitate this communication.
In terms of books, I really like "The Architecture of Open Source Applications" (http://aosabook.org/) which is also mentioned in the blog post. For the enterprise world, "Patterns of Enterprise Application Architecture" (http://martinfowler.com/books/eaa.html) is a good choice.
However, before applying any technical software architecture methods, it is really important to understand what your stakeholders actually want/need. Unfortunately most of the time they don't even know it themselves. In my experience most software projects fail, not because of technical reasons but simply because the original problem hasn't been understood. This is where communication is key. I assume the reason why the authors of the blog post encouraged drawing diagrams and looking at different perspectives/views is to facilitate this communication.
> good books for software engineering practices in the field right now
I think there is none.
I think every organization is different. The best thing to do is be communicative, responsive and honest. Moving agile means all the above. Rule one is start small and go simple. Just spent three days on making something and I got stuck so I said fuck it, start small. So I need to build this tool that reads the output of another tool. Sadly it comes out in terminal with all kinds of custom format. Instead of parsing the full "terminal output" I receive, I start small by reading and getting whatever I need. Gradually I built up the "reader/parser" I need to from this report I was trying to parse. And I finished what I was hoping to finish.
> Also when you're doing sole projects are there specific procedures you should follow and how do they differ from when you're in a team?
It's very subjective. I think if you are accustom to one way you do it that way more often. But I said in my example, I spent three days trying to go complex. That's the failure of me, but it probably happen to everyone everytime because I (or we) think I was smart so I could solve this complex issue. The only rule I have when I do sole project is to define the scope of my project. I write down whatever I want to implement, what it does. Essentially building up a set of user stories for myself. But that's also true for team project. You just can't be sloppy and acting alone when you do team work... so when you work with others, get the user stories up and running, figure out the style you want to code in... they all matter.
Last thing: it's easy to say, but it is damn hard to do it. Just relax.
I think there is none.
I think every organization is different. The best thing to do is be communicative, responsive and honest. Moving agile means all the above. Rule one is start small and go simple. Just spent three days on making something and I got stuck so I said fuck it, start small. So I need to build this tool that reads the output of another tool. Sadly it comes out in terminal with all kinds of custom format. Instead of parsing the full "terminal output" I receive, I start small by reading and getting whatever I need. Gradually I built up the "reader/parser" I need to from this report I was trying to parse. And I finished what I was hoping to finish.
> Also when you're doing sole projects are there specific procedures you should follow and how do they differ from when you're in a team?
It's very subjective. I think if you are accustom to one way you do it that way more often. But I said in my example, I spent three days trying to go complex. That's the failure of me, but it probably happen to everyone everytime because I (or we) think I was smart so I could solve this complex issue. The only rule I have when I do sole project is to define the scope of my project. I write down whatever I want to implement, what it does. Essentially building up a set of user stories for myself. But that's also true for team project. You just can't be sloppy and acting alone when you do team work... so when you work with others, get the user stories up and running, figure out the style you want to code in... they all matter.
Last thing: it's easy to say, but it is damn hard to do it. Just relax.
I was searching around for how to install Hadoop and found this information related to software patterns and how well they are understood by undergraduates leaving university.
This fellow: http://www.jonathanholloway.co.uk/work
Wrote his master's thesis on the very subject http://www.jonathanholloway.co.uk/thesis/abstract.html
[Whole thesis back in /work]
This fellow: http://www.jonathanholloway.co.uk/work
Wrote his master's thesis on the very subject http://www.jonathanholloway.co.uk/thesis/abstract.html
[Whole thesis back in /work]
I guess this is s good approach to expose students to various software architectures and what real projects end up looking like. However, I'm not sure that this teaches the students what good architecture is/does and how it evolves over time. Which is to say, I'm not sure that this explanation says that the course teaches students how to DO good software architecture.
Knowing about architecture and doing it in your own code are two different things.
Knowing about architecture and doing it in your own code are two different things.
I'm finding it very difficult getting into a large open source project. I consider myself an experienced developer, but for some reason it feel very intimidating, especially when the code base is large. I was wondering if someone could give any tips ?
The students of the course had the same problem.
We advised the students to look critically at recent open (or just closed) issues and pull requests, as well as e.g., the mailing list for the project at hand. Based on that they could get a feeling what was happening at the moment, and what a typical contribution looks like.
If you want to contribute code, we found that it works best if you work on parts of the code that are currently under active development. This also means that you do not have to have a deep understanding of the full code base: A broad overview suffices.
Some of our students also offered documentation for, e.g., installation or deployment: they struggled themselves, and offered what they learned back to the project.
We advised the students to look critically at recent open (or just closed) issues and pull requests, as well as e.g., the mailing list for the project at hand. Based on that they could get a feeling what was happening at the moment, and what a typical contribution looks like.
If you want to contribute code, we found that it works best if you work on parts of the code that are currently under active development. This also means that you do not have to have a deep understanding of the full code base: A broad overview suffices.
Some of our students also offered documentation for, e.g., installation or deployment: they struggled themselves, and offered what they learned back to the project.
Thanks for the answer, it helps a lot !
I try to work on the software I actually use. I don't jump on a random horse and start riding to the West. I think the first step is look at your tools. What problem do you have with your tools. Is the library you using too restricted? Does it have a null/nonetype exception rasied and not properly handle? If you can't solve it because you have no idea where to start, well, at least file a ticket :) I have limited experience too. I think the first real contribution was Ansible, rewriting the hg module was critical for my use. So I did. When I was an intern at Mozilla I learned about CSP so later when I found a few minor issues with the CSP implementation in Firefox I started volunteering. My internship didn't involve a single line of writing Firefox code so it did take a while to navigate through a couple files for a newbie. But grep, mxr.mozilla.org all helped.
I think even just making demos and making documentation changes is equally powerful and useful contribution.
I think even just making demos and making documentation changes is equally powerful and useful contribution.
Awesome! This is actually quite helpful and it gave me a couple of ideas.
It is actively harmful to teach students that software architecture is something that somehow arises from diagrams or that those kinds of silly pictures capture anything important about it. Powerful architectures come out of powerful ideas that in turn come from accumulated hard work of many people in different disciplines. One can learn much more from walking through the actual source code of some classic projects and from trying to understand the ideas that make them tick:
https://github.com/onetrueawk/awk - UNIX philosophy of small tools, DSLs, CS theory: state machines / regular expressions, Thompson algorithm ...
https://github.com/mirrors/emacs - Both a program and a VM for a programming language, hooks, before/after/around advices, modes, asynchronous processing with callbacks, ... Worth to think of challenges of designing interactive programs for extensibility.
https://github.com/rails/rails - Metaprogramming DSLs for creating powerful libraries, again a lesson in hooks (before_save etc.), advices (around_filter etc.), ...
https://github.com/git/git - The distributed paradigm, lots of CS theory again: hashing for ensuring consistency, DAGs everywhere, ... By the way, the sentence "yet the underlying git magic sometimes resulted in frustration with the students" is hilarious in the context of a "software architecture" course.
One of computer algebra systems - the idea of a http://en.wikipedia.org/wiki/Canonical_form
One of computer graphics engines - Linear algebra
...
There are loads of things one can learn from those projects by studying the source in some depth, but I can't think of any valuable things one could learn by just drawing pictures of the modules and connecting them with arrows. There are also several great books that explore real software design issues and not that kind of pretentious BS, they all come from acknowledged all-time master software "architects", yet all of them almost never find diagrams or "viewpoints" useful for saying the things they want to say, and they all walk you through real issues in real programs:
http://www.amazon.com/Programming-Addison-Wesley-Professiona...
http://www.amazon.com/Paradigms-Artificial-Intelligence-Prog...
http://www.amazon.com/Structure-Interpretation-Computer-Prog...
http://www.amazon.com/Unix-Programming-Environment-Prentice-...
http://www.amazon.com/Programming-Environment-Addison-Wesley...
To me, the kind of approach pictured in the post, seems like copying methods from electrical or civil engineering to appear more "serious", without giving due consideration to whether they really are helpful for anything for real-world software engineering or not. The "software engineering" class which taught those kind of diagram-drawing was about the only university class I did not ever get any use from, in fact I had enough industry experience by the point I took it that it just looked silly.
https://github.com/onetrueawk/awk - UNIX philosophy of small tools, DSLs, CS theory: state machines / regular expressions, Thompson algorithm ...
https://github.com/mirrors/emacs - Both a program and a VM for a programming language, hooks, before/after/around advices, modes, asynchronous processing with callbacks, ... Worth to think of challenges of designing interactive programs for extensibility.
https://github.com/rails/rails - Metaprogramming DSLs for creating powerful libraries, again a lesson in hooks (before_save etc.), advices (around_filter etc.), ...
https://github.com/git/git - The distributed paradigm, lots of CS theory again: hashing for ensuring consistency, DAGs everywhere, ... By the way, the sentence "yet the underlying git magic sometimes resulted in frustration with the students" is hilarious in the context of a "software architecture" course.
One of computer algebra systems - the idea of a http://en.wikipedia.org/wiki/Canonical_form
One of computer graphics engines - Linear algebra
...
There are loads of things one can learn from those projects by studying the source in some depth, but I can't think of any valuable things one could learn by just drawing pictures of the modules and connecting them with arrows. There are also several great books that explore real software design issues and not that kind of pretentious BS, they all come from acknowledged all-time master software "architects", yet all of them almost never find diagrams or "viewpoints" useful for saying the things they want to say, and they all walk you through real issues in real programs:
http://www.amazon.com/Programming-Addison-Wesley-Professiona...
http://www.amazon.com/Paradigms-Artificial-Intelligence-Prog...
http://www.amazon.com/Structure-Interpretation-Computer-Prog...
http://www.amazon.com/Unix-Programming-Environment-Prentice-...
http://www.amazon.com/Programming-Environment-Addison-Wesley...
To me, the kind of approach pictured in the post, seems like copying methods from electrical or civil engineering to appear more "serious", without giving due consideration to whether they really are helpful for anything for real-world software engineering or not. The "software engineering" class which taught those kind of diagram-drawing was about the only university class I did not ever get any use from, in fact I had enough industry experience by the point I took it that it just looked silly.
The pictures are a result of studying the systems in question.
For example, you have given links to a couple of github repos. Now if I'm trying to create an overview of how say emacs works, I would try to identify the main sub-parts and break the system down into more manageable chunks of study. This is all what 'viewpoints' are trying to structure. None of this is evident from just the codebase unless it has been captured in some documentation, or you take the trouble to understand it yourself.
You seem to have a very low opinion of software architects, but believe me, you wouldn't want to work on a 5-year old system where huge teams of developers have let loose and no one/team was responsible to ensure a cohesive, maintainable system.
Viewpoints are fantastic for communication purposes, and if you feel that the particular viewpoints shown don't make much of a difference, I agree :). A student project is also not the ideal environment, but it is really valuable to know how to create them, considering the information needs of the audience (biz vs tech vs in-depth tech). And that is what this course is all about.
For example, you have given links to a couple of github repos. Now if I'm trying to create an overview of how say emacs works, I would try to identify the main sub-parts and break the system down into more manageable chunks of study. This is all what 'viewpoints' are trying to structure. None of this is evident from just the codebase unless it has been captured in some documentation, or you take the trouble to understand it yourself.
You seem to have a very low opinion of software architects, but believe me, you wouldn't want to work on a 5-year old system where huge teams of developers have let loose and no one/team was responsible to ensure a cohesive, maintainable system.
Viewpoints are fantastic for communication purposes, and if you feel that the particular viewpoints shown don't make much of a difference, I agree :). A student project is also not the ideal environment, but it is really valuable to know how to create them, considering the information needs of the audience (biz vs tech vs in-depth tech). And that is what this course is all about.
The point is that how emacs is broken down in modules is one of the least important things about emacs. I can draw a block diagram without a "software architecture" course, thank you. In software engineering, there are of minor usefulness most of the time, if of any.
There have to be persons that guide the overall development, sure, but this does not mean that those have to be "architects" living in a castle somewhere drawing those diagrams. The best "architects" are people with long experience in the trenches that know all the ins and outs, all the relevant theory, know the domain, and supervise it down to the nitty-gritty details. Linus Torvalds is a software architect for Linux, he doesn't do much coding anymore, but he isn't busy drawing diagrams without understanding the code really well either, he just is the person with the widest understanding of the whole project:
https://lkml.org/lkml/2012/10/2/547
There have to be persons that guide the overall development, sure, but this does not mean that those have to be "architects" living in a castle somewhere drawing those diagrams. The best "architects" are people with long experience in the trenches that know all the ins and outs, all the relevant theory, know the domain, and supervise it down to the nitty-gritty details. Linus Torvalds is a software architect for Linux, he doesn't do much coding anymore, but he isn't busy drawing diagrams without understanding the code really well either, he just is the person with the widest understanding of the whole project:
https://lkml.org/lkml/2012/10/2/547
There are a LOT of practicing architects that really fit the mould you are describing! Linus has the big picture in his head, and perhaps if he were to draw them out, does it make him an ivory tower denizen ?!
Also, ignoring how a system is modularized is typically how you get tangled codebases by introducing unexpected dependencies. Just because it is 'less interesting' doesn't make it 'unimportant'.
Also, ignoring how a system is modularized is typically how you get tangled codebases by introducing unexpected dependencies. Just because it is 'less interesting' doesn't make it 'unimportant'.
Actively harmful? Diagrams?
Sorry I have to counter-react strongly. Your post is negative without a valid point and the irony is that you agree with the post.
> One can learn much more from walking through the actual source code of some classic projects and from trying to understand what makes them tick
That is the fun part about this post! You propose to mandidate a list of classical code bases. This course lets students freely pick any Github project with activity and complexity. They get graded based on pull requests.. Contributing code and interacting with an Open Source project you picked a week ago is not "drawing pictures" and "connecting them with arrows".
Sorry I have to counter-react strongly. Your post is negative without a valid point and the irony is that you agree with the post.
> One can learn much more from walking through the actual source code of some classic projects and from trying to understand what makes them tick
That is the fun part about this post! You propose to mandidate a list of classical code bases. This course lets students freely pick any Github project with activity and complexity. They get graded based on pull requests.. Contributing code and interacting with an Open Source project you picked a week ago is not "drawing pictures" and "connecting them with arrows".
Programs are powerful because of concepts. Whether or not you will be able to build a good 3d engine rests much more on the fact whether you know or able to invent a concept of a vector and have it guide all your future coding than it rests on whatever is visible from a block diagram. Both the course and the book linked to seem to focus on some abstract ideas of doubtful usefulness and never ever seem to touch upon those important things that historically have allowed us to build complex programs where other approaches have failed. Google can exist not because people invented new kinds of diagrams, but because large scale numerical linear algebra techniques were applied to the problem of search. This is the heart of Google's "architecture", not a class diagram of the crawler.
Picking arbitrary programs won't do. The point is to pick the really good designs and illustrate why they are good, and I think it boils down much more to knowing mathematics, algorithms, programming languages and having studied a lot of existing software, than to usage of any modelling techniques so far invented in formal software engineering circles.
Picking arbitrary programs won't do. The point is to pick the really good designs and illustrate why they are good, and I think it boils down much more to knowing mathematics, algorithms, programming languages and having studied a lot of existing software, than to usage of any modelling techniques so far invented in formal software engineering circles.
There's no conflict between reading and understanding a codebase and seeing the value in a diagram. You can have both. Take a look at the example you chose: [0] and [1] (about a third of the way down).
They first explained PageRank (without a diagram), then explained the flow of the data from the crawler (with a diagram). You seem to be arguing that diagrams are never useful, and that just isn't true.
[0] http://infolab.stanford.edu/~backrub/google.html [1] http://infolab.stanford.edu/~backrub/over.gif
They first explained PageRank (without a diagram), then explained the flow of the data from the crawler (with a diagram). You seem to be arguing that diagrams are never useful, and that just isn't true.
[0] http://infolab.stanford.edu/~backrub/google.html [1] http://infolab.stanford.edu/~backrub/over.gif
Here's an addition to the answers from the others -- I'm one of the authors of the post.
Thanks for your feedback.
Perhaps we did not sufficiently emphasize it in the blog post, but we do tell the students about strong existing architectures, open source as well as from industry. The books listed in further reading in the post actually contain chapters about git and emacs, and I share your enthusiasm for Bentley's little language approach.
The philosophy of the course is based on trying to apply published approaches to software architecture (such as those in Rozanski & Woods' book, but it could be others) to real systems actually maintained and used. We encourage to get in touch with the creators of those systems.
The students actually take a deep dive into a foreign code base -- I included pictures because that is easy to share in a post, but the students analyze code, report about that, execute test cases, deploy systems, etc. The pictures are indeed, as suggested by some of the others, means for communication.
Thanks for your feedback.
Perhaps we did not sufficiently emphasize it in the blog post, but we do tell the students about strong existing architectures, open source as well as from industry. The books listed in further reading in the post actually contain chapters about git and emacs, and I share your enthusiasm for Bentley's little language approach.
The philosophy of the course is based on trying to apply published approaches to software architecture (such as those in Rozanski & Woods' book, but it could be others) to real systems actually maintained and used. We encourage to get in touch with the creators of those systems.
The students actually take a deep dive into a foreign code base -- I included pictures because that is easy to share in a post, but the students analyze code, report about that, execute test cases, deploy systems, etc. The pictures are indeed, as suggested by some of the others, means for communication.
What repels me is the kind of approach taken by this Rozanski & Woods book. It's like a 500 pages of authors philosophy of software, and whether all that is of any use is completely uncertain. There is not much of a widely established corpus of knowledge on software engineering/architecture, another book would treat an almost completely different set of concepts, and whatever is commonly agreed upon is trivial and most people invent in on their own when they need it. You can publish completely anything about "Software architecture" in this spirit thats sounds smart enough and it won't ever be put to any test. I personally, from experience, don't believe any practical high-level theory of software engineering of this kind exists at all, the more "philosophical" or "theoretical" it gets the less useful it becomes. It is much better to spend this time studying established disciplines where things actually get tested out one way or another, mathematics, algorithms, operating systems, ... If there is anything specific to be taught under "software architecture", it's those ideas ("patterns") that made certain programs so much better than the alternatives, but you only really learn them by digging deep in, not by studying some very abstract models.
From your post it seems you think software architecture and -engineering are the same.. In this course -- disclaimer: I was a student taking this course -- it's not that case, Software Architecture is not the sum of a program's lines or modules. SA is the methodology of getting involved and steering a project, this includes handling team efforts and other stakeholders. Looking and learning a lot of (implementation) patterns is of course very useful, but only a subpart of SA.
I tend to think that a good SArchitect knows that the patterns they might be using are good or well suited for the current problem, a better SArchitect knows that he's perhaps using a less well suited pattern for the problem, but can layout a plan to migrate to a new and better situation without breaking continuity of the project.
Furthermore, it can't be stated clearer that in the course there are a lot of guest lecturers who _do_ show different idea's.
I tend to think that a good SArchitect knows that the patterns they might be using are good or well suited for the current problem, a better SArchitect knows that he's perhaps using a less well suited pattern for the problem, but can layout a plan to migrate to a new and better situation without breaking continuity of the project.
Furthermore, it can't be stated clearer that in the course there are a lot of guest lecturers who _do_ show different idea's.
In addition to what the other respondents have written, could it be that you are not a visual learner, and diagrams don't 'do it' for you?
What a great opportunity for these students. I wish my university had done something like this.
Lastly if it isn't too much can you post links to the course website for this,the software communications and software metrics?