Want Better Quality? Fire Your QA Team(blogs.forrester.com)
blogs.forrester.com
Want Better Quality? Fire Your QA Team
http://blogs.forrester.com/mike_gualtieri/11-02-17-want_better_quality_fire_your_qa_team
47 comments
People who know how to test are extremely valuable. Specially when doing peer review/pair programming with (solving the blind's eye thing)
Testing as a whole is not an easy task as it involves testing some really hard things: memory leaks, performance, UX, correctness of parallel and distributed algorithms/programs/systems.
However, Wealthfront engineers probably put this position more into perspective [1]:
We do not have QA team and do not want to have one, the reasoning is that if a human is involved in testing then there is a higher chance of missing things and you simply can't test all the site dozens of times a day.
Granted, Wealthfront uses continuous deployment[2], while still being regulated by the SEC. I really enjoy their lean startup approach to minimize inventory (ie: code not in production), while still keeping an immune system. And investors seem to agree it is really paying off [3].
[1] http://eng.wealthfront.com/2010/04/findbugs-husdon-and-pizza...
[2] http://www.eishay.com/2010/07/continuous-deployment-at-kachi...
[3] http://techcrunch.com/2010/10/19/kaching-gets-100m-under-man...
Testing as a whole is not an easy task as it involves testing some really hard things: memory leaks, performance, UX, correctness of parallel and distributed algorithms/programs/systems.
However, Wealthfront engineers probably put this position more into perspective [1]:
We do not have QA team and do not want to have one, the reasoning is that if a human is involved in testing then there is a higher chance of missing things and you simply can't test all the site dozens of times a day.
Granted, Wealthfront uses continuous deployment[2], while still being regulated by the SEC. I really enjoy their lean startup approach to minimize inventory (ie: code not in production), while still keeping an immune system. And investors seem to agree it is really paying off [3].
[1] http://eng.wealthfront.com/2010/04/findbugs-husdon-and-pizza...
[2] http://www.eishay.com/2010/07/continuous-deployment-at-kachi...
[3] http://techcrunch.com/2010/10/19/kaching-gets-100m-under-man...
At least in my wife's case, she generally does pre-release QA, and doesn't necessarily do continuous QA. Still though, even if you need support for continuous deployment, you probably want a QA person to set up your automated test cases to test the output of a site in a way that, in my opinion, unit tests don't seem to catch.
Having worked in Silicon Valley for 12 years, I have yet to find QA people who could successfully automate testing in a non-fragile way or without using an expensive, proprietary and crappy testing harness. Those who know how to program are invariably better at writing automated tests. One recent CS graduate should be able to put a team of QA out of a job. (Disclosure: I work for Wealthfront)
[deleted]
> This app can't go down, and it can't be wrong.
Shops who set the bar this high and mean it are exceptionally rare (I look for them when I'm on the market). Where I am now, an hour of downtime costs us my salary for weeks, and still we cut corners that don't always sit right with me.
But having generalist engineers create functional and integration test plans? Not only do we tend to cost more, but we have less experience at it than QA specialists. And a dev shouldn't test their own work, because they won't notice any unwarranted assumptions they made in design and coding.
Shops who set the bar this high and mean it are exceptionally rare (I look for them when I'm on the market). Where I am now, an hour of downtime costs us my salary for weeks, and still we cut corners that don't always sit right with me.
But having generalist engineers create functional and integration test plans? Not only do we tend to cost more, but we have less experience at it than QA specialists. And a dev shouldn't test their own work, because they won't notice any unwarranted assumptions they made in design and coding.
ridiculous.
a large fraction of errors that turn up in QA are conceptual errors made by the development team; like the blind spot in the eye, you don't know what you don't know, and there are certain problems that you'll never find yourself.
Phil Crosby would say that management is responsible for quality; management writes the specifications and has the responsibility to hire and fire developers and QA people. The buck stops with management.
Efficient quality requires practices to be work correctly across the organization, but it starts in the specification process. If you don't know where you're going, you're sure as hell never going to get there.
a large fraction of errors that turn up in QA are conceptual errors made by the development team; like the blind spot in the eye, you don't know what you don't know, and there are certain problems that you'll never find yourself.
Phil Crosby would say that management is responsible for quality; management writes the specifications and has the responsibility to hire and fire developers and QA people. The buck stops with management.
Efficient quality requires practices to be work correctly across the organization, but it starts in the specification process. If you don't know where you're going, you're sure as hell never going to get there.
"If you don't know where you're going, you're sure as hell never going to get there." -> Also not always true. Think Twitter. Think Christopher Columbus.
Columbus knew where he was going. And would have died trying to get there, if he hadn't serendipitously run into something else, first.
But he misjudged that something else (America) with his original destination (India).
Quoting from wiki (http://en.wikipedia.org/wiki/Christopher_Columbus):
"Amerigo Vespucci's travel journals, published 1502-4, convinced Martin Waldseemüller that the discovered place was not India, as Columbus always believed, but a new continent"
Quoting from wiki (http://en.wikipedia.org/wiki/Christopher_Columbus):
"Amerigo Vespucci's travel journals, published 1502-4, convinced Martin Waldseemüller that the discovered place was not India, as Columbus always believed, but a new continent"
Not sure this takes into account everyone else who died at sea. Occasional serendipitous results are to be expected, but don't represent what is probably a large body of associated failures to this approach.
Also not always true. Think Twitter.
Taken out of context. Twitter wasn't a result of good or bad QA. In fact, it's got nothing to do with that.
Taken out of context. Twitter wasn't a result of good or bad QA. In fact, it's got nothing to do with that.
If a developer's reaction to a client-facing defect is "QA didn't catch it", you have cultural problems that firing a QA team probably won't fix. If you don't have a culture of personal responsibility, a finger can always be pointed elsewhere.
Developers should absolutely write and use unit tests. Where QA really shines in the larger scale: validating actual business requirements.
Developers should absolutely write and use unit tests. Where QA really shines in the larger scale: validating actual business requirements.
Exactly. For me, every client-facing defect is an embarrassing personal failure. Having them caught in QA simply lessens the shame, and allows me to dedicate all the time I would spend freaking out about the newfound emergency to figuring out improvements that eliminate that entire class of errors.
I 100% agree with this article. Quality is everyone's job, not just the QA teams' and I believe that, in most cases, developers can and should be responsible for developing, testing, and releasing their code. When developers can't "throw code over the wall to QA" because there is no QA, they will do a better job testing it themselves.
I don't buy the popular opinion that developers can't test their own code like there is some sort of magic that goes on in the mind of QA engineers that causes them to be able to find bugs that developers couldn't find themselves. In fact, I think the opposite is true - developers are better suited to test the features they are writing because they know the code and know the risk areas, where things need to be regression tested, where more attention should be placed, etc.
I do think that it can be helpful for another set of eyes to look at a developed feature and this is where pair programming can be useful. It does not need to be QA doing this.
One caveat is that this approach does not work for all teams. This works for small, agile teams where roles and responsibilities are less defined. If you fire your QA team but your developers can't test their own code because they are set in their ways of writing crap because they expect QA to find the bugs, you may end up firing your dev team too....
I don't buy the popular opinion that developers can't test their own code like there is some sort of magic that goes on in the mind of QA engineers that causes them to be able to find bugs that developers couldn't find themselves. In fact, I think the opposite is true - developers are better suited to test the features they are writing because they know the code and know the risk areas, where things need to be regression tested, where more attention should be placed, etc.
I do think that it can be helpful for another set of eyes to look at a developed feature and this is where pair programming can be useful. It does not need to be QA doing this.
One caveat is that this approach does not work for all teams. This works for small, agile teams where roles and responsibilities are less defined. If you fire your QA team but your developers can't test their own code because they are set in their ways of writing crap because they expect QA to find the bugs, you may end up firing your dev team too....
Having been in a high pressure position in which my team and I were writing mission critical code which could "have no bugs whatsoever" first without QA, and then later with QA, I can say confidently the quality of the code and occurrence of bugs dropped dramatically after we introduced our QA engineers.
Our prior condition was very similar to telling a newspaper writer "DONT WRITE TYPOS". You can scan a page you've written several times carefully and still miss your mistakes, even if you are a conscientious writer. And the newspaper industry isn't the only industry that prevents mistakes through multi-party review processes, either.
Our prior condition was very similar to telling a newspaper writer "DONT WRITE TYPOS". You can scan a page you've written several times carefully and still miss your mistakes, even if you are a conscientious writer. And the newspaper industry isn't the only industry that prevents mistakes through multi-party review processes, either.
QA and dev are two different skills. It's like saying that we save on design by having the devs do it, or having your quarterback also play noseguard. Sure it works in high school, but not when you're a pro.
I'd much rather hire someone who knows testing and QA well and let my devs focus on writing bug-free product code.
I'd much rather hire someone who knows testing and QA well and let my devs focus on writing bug-free product code.
I disagree. Developers who don't know how to test are bad developers, they should go. And conversely, if you have QA, they should know how to develop, otherwise you're paying them to hunt, peck, and click. Why not have QA write the code to have computer do that and save like 1000x the time? That's efficiency AND quality.
Of course QA knows how to program. But their skillset is about breaking code. It's about writing this crazy C++ function that shows how your API has a security bug in this case. As developers their like dev tools developers as their customers are generally devs.
Skillset maybe isn't the right word... mindset maybe better word. Both should be good programmers -- in fact you'll often have more test code than product code. But you likely won't be able to switch your dev team to QA and vice-versa and get the same results.
Skillset maybe isn't the right word... mindset maybe better word. Both should be good programmers -- in fact you'll often have more test code than product code. But you likely won't be able to switch your dev team to QA and vice-versa and get the same results.
You're lucky to have QA people who know C++, I've never met one. Maybe I've been in too many web-oriented startups.
By the time anything gets to the QA person at my dayjob, the devs have usually tried every damn thing that they could think of to break their own code.
The usual reaction to the bugs she finds is "what made you think of trying that?"
The usual reaction to the bugs she finds is "what made you think of trying that?"
I work in QA in a high-reliability industry. Correct operation of some our products saves millions and saves lives. We need super high quality. How we do QA is pretty traditional. In particular, we have functional test teams. The perspective we take is that the testers take the specs and develop tests to exercise the functionality, and theoretically, we (QA) cover any spots the developers missed, as well as providing assurance that the developers aren't talking out of the wrong hole when they say they're done. Seems to work most of the time except when mutual brain-farts develop.
I don't know that I would go so far as to recommend firing myself to increase quality. I would have to visit with many teams and really gain experience in the way they work. I do think that we could take a step forward by having a standard across the company to have fully automated test suites running continuous builds and tests. That, however, runs facefirst into the problem of Legacy Stuff and ROI. Automation just plain takes time when you are interfacing against external programs. It's not always worth it in the short/medium term.
There's no silver bullet to the software quality problem. But me, I think that continuous integration provides a copper bullet.
I don't know that I would go so far as to recommend firing myself to increase quality. I would have to visit with many teams and really gain experience in the way they work. I do think that we could take a step forward by having a standard across the company to have fully automated test suites running continuous builds and tests. That, however, runs facefirst into the problem of Legacy Stuff and ROI. Automation just plain takes time when you are interfacing against external programs. It's not always worth it in the short/medium term.
There's no silver bullet to the software quality problem. But me, I think that continuous integration provides a copper bullet.
This works as long as your developers are talented in QA and are motivated to do it. That sounds like it requires a lucky constellation of events.
Don't be so quick to brush this off. The argument isn't so different from saying that we don't want to distinguish "front-end" and "back-end" programmers. It's probably a special circumstance that you need a team that is entirely dedicated to verification.
Feynman talks about this arrangement in his essay on the shuttle program (http://www.fotuva.org/feynman/challenger-appendix.html). In a nutshell, the core software team is responsible for checking the code. Then there is a totally removed group that is adversarial, i.e., it is measured on its ability to trip up the software team. This is the same a "setting consequences" as this author describes.
Now, the shuttle program is truly life critical. I think a good software team can produce quality results internally.
Feynman talks about this arrangement in his essay on the shuttle program (http://www.fotuva.org/feynman/challenger-appendix.html). In a nutshell, the core software team is responsible for checking the code. Then there is a totally removed group that is adversarial, i.e., it is measured on its ability to trip up the software team. This is the same a "setting consequences" as this author describes.
Now, the shuttle program is truly life critical. I think a good software team can produce quality results internally.
The argument isn't so different from saying that we don't want to distinguish "front-end" and "back-end" programmers.
I don't want to distinguish front-end and back-end programmers. I would like to distinguish front and back-end development.
I do everything currently but I still need put on a conceptual "testing hat", "GUI hat" and "back-end hat". That requires mental effort and costs time. A real Q&A would save that even if I was exactly as skilled as the Q&A person.
I don't want to distinguish front-end and back-end programmers. I would like to distinguish front and back-end development.
I do everything currently but I still need put on a conceptual "testing hat", "GUI hat" and "back-end hat". That requires mental effort and costs time. A real Q&A would save that even if I was exactly as skilled as the Q&A person.
Yeah, that's true. I think the point of the source is that the first line of defense is with the originator of the code. Sure, the context switch is expensive, but that's the price of higher quality.
It's not really about firing the QA team. Instead, it's about having a QA function that does not entirely relieve the original author of his duty to ensure quality.
It's not really about firing the QA team. Instead, it's about having a QA function that does not entirely relieve the original author of his duty to ensure quality.
Conceptual distinctions like that are great. I wish more people could be up front about the multi-modal nature of software work.
Feynman's essay is describing a QA team, is it not?
Ah it's a single anecdotal story and comes with no specifics about who it is or in what way things supposedly improved, just some broad calls for firing QA.
Forrester will ring a few more people they picked out of the phone at random, write it up as a report for their library complete with beautiful graphs, and then begin to charge thousands of dollars per copy to tell other consultants what the "Industry Best Practice" is at the moment.
> They better get it right, or heads will roll. As British author Samuel Johnson famously put it, "The prospect of being hanged focuses the mind wonderfully."
Gee, sounds like a wonderful place to work. When can I sign up?
Gee, sounds like a wonderful place to work. When can I sign up?
My head is more often buried in code than in the product's UI. Our QA team has a better nose for when interaction or UI details are off than I do. In general, they spend more time using the application, and have a better instinct for when a given behaviour is wrong.
Some of the bugs they report would fall under the automatically testable category, but many don't.
I suspect that dogfooding is more effective at companies where the product itself is useful to developers or is a consumer app. e.g. Basecamp, FogBugz, Facebook, Netflix, Gmail, etc.
Some of the bugs they report would fall under the automatically testable category, but many don't.
I suspect that dogfooding is more effective at companies where the product itself is useful to developers or is a consumer app. e.g. Basecamp, FogBugz, Facebook, Netflix, Gmail, etc.
Ah yes, Forrester.
I hope all of you reading the blog post realise that this brainfart will cost your organisation $5,500 per copy downloaded.
I hope all of you reading the blog post realise that this brainfart will cost your organisation $5,500 per copy downloaded.
I don't think this would be a good idea. When developing I don't always have the time to be as thorough as a QA team is in testing my own code. It's very helpful to have somebody else with a non-technical background to help spot problems that I am blind to.
It's also helpful to have someone that can run through endless permutations, paying careful attention to each, finding small issues that popup with countless applied revisions. It is not an easy job.
That you can afford to be blind to because you have someone doing QA?
No, that you're blind to because it's in the nature of software development. Spotting flaws in something you've spent the last $X hours convincing yourself is correct is hard to do. It's easy to make invalid assumptions and internalize them in your thinking to the point you don't notice them. This problem exists, and largely is solved by dragging in someone else to help with it. That someone else, unless you'd like to squabble over semantics, is doing QA.
Well, that's certainly helpful. But what I'm saying is more that, without a QA team, while I would I would almost certainly attempt to quality control my own code more, it would be ignorant to deny that I or anybody else has blind spots.
I believe it's a lot more difficult to see your own errors than somebody else's. I believe QA is a fulltime job. I believe the relationship between developers and QA people is mutually beneficial to both people.
I believe it's a lot more difficult to see your own errors than somebody else's. I believe QA is a fulltime job. I believe the relationship between developers and QA people is mutually beneficial to both people.
The author plays to the stereotype of developers who recklessly "commit and forget it". In a properly run organization, QA takes note of developers who consistently commit show-stopper bugs. If the QA team is positioned as a clean up crew for shabby code, this is a failing of the organization rather than the developers.
In an organization in which tests are developed for any code committed, and developers are careful not to break a build, soft issues are still caught by the QA team. Things that work but not as expected, minor cross-platform glitches, and permutations you'd never think to check, are all commonplace catches for a professional QA tech working with a well functioning development team.
The author of the article really doesn't sound like he's worked with a decent dev or QA team.
In an organization in which tests are developed for any code committed, and developers are careful not to break a build, soft issues are still caught by the QA team. Things that work but not as expected, minor cross-platform glitches, and permutations you'd never think to check, are all commonplace catches for a professional QA tech working with a well functioning development team.
The author of the article really doesn't sound like he's worked with a decent dev or QA team.
Considering I work with one of the most incompetent QA engineers I've ever had the displeasure of encountering, I couldn't agree more. This person wastes my time, and the organization's time, and has not actually increased the quality of software (as far as I can tell). It is a waste of a developer's time to decipher a bug that states "it's broken," without clearly defining "broken," or "it," or how to reproduce the behavior. Often, reproduction involves user error (like forgetting to sign into the vpn). If it weren't for this bozo, our team could get more work done in less time.
I assume that others have had better experiences.
I assume that others have had better experiences.
Do you think that your unpleasant situation is attributable to QA being an unhelpful discipline, or to a particular person performing poorly in their QA role?
Peopleware has a chapter about "the black team" that was at IBM. The chapter is about team work but they describe how in the process of gelling the QA team delighted in finding programmer bugs. They describe how creation of a reputation and ethos in the black team not only made them a little better than other QA teams but significantly better.
While it's easy to say that QA isn't important many of the QA teams I've worked with weren't respected or valued, and often saw QA as a stepping stone. If you want to increase the quality of the code, don't fire the QA team, motivate them.
While it's easy to say that QA isn't important many of the QA teams I've worked with weren't respected or valued, and often saw QA as a stepping stone. If you want to increase the quality of the code, don't fire the QA team, motivate them.
What is the metric for quality? Is the article saying that having fewer people looking for bugs resulted in more bugs being found? Or fewer bugs? Or just fewer bugs found?
Most of the other comments already voice my reaction to this post. I will only add that a "heads will roll" attitude towards motivating devs on the part of management would prompt me to switch to some other place very fast. Simply saying "you have to do X as well as Y or heads will roll" just sounds like trying to save money on QA and squeezing more work out of devs - who could be putting that time and effort to much better use improving the product.
[deleted]
A good QA analyst / team does much more than point at a problem and say 'there it is'. A good QA analyst can determine why you made an error, and can anticipate other errors you might have made in the same way. A good QA analyst will learn your formulae, learn your equations, understand your workflow and try to find root cause, even if they're not developers.
Having a good QA person with a trained eye for knowing what I'm likely to have overlooked is also FAR more cost effective than your high-paid developers doing the same task. If they know that 'heads will roll' when they get errors, I'm also betting that makes your releases even slower.
Off-topic completely, but I generally prefer non-developer QA analysts, as they generally prove themselves to 'think too much like me' to be effective.