Ask HN: Web testing is killing us - best practice?
4 comments
Do you really need to be that thorough in your testing? What are the costs when a bug gets into production and needs fixing? What are the costs of testing and fixing it before it goes into production? Balance it.
We're only doing developer ad hoc testing. We just click through a few things when we develop a feature (something you're doing anyway, I assume) and then put in production. We don't do exhaustive testing. If an edge case doesn't work and it gets reported, we'll fix it then. This has the added advantage of only working on problems customers really consider problems.
This works because a web application can be pushed online very quickly, so you can make many iterations and fixing a bug isn't inherently harder when it's in production or in development.
This is a huge YMMV-topic, where you should really think of both sides of the equation. Something that doesn't happen too often in Internet-articles, I'm afraid.
We're only doing developer ad hoc testing. We just click through a few things when we develop a feature (something you're doing anyway, I assume) and then put in production. We don't do exhaustive testing. If an edge case doesn't work and it gets reported, we'll fix it then. This has the added advantage of only working on problems customers really consider problems.
This works because a web application can be pushed online very quickly, so you can make many iterations and fixing a bug isn't inherently harder when it's in production or in development.
This is a huge YMMV-topic, where you should really think of both sides of the equation. Something that doesn't happen too often in Internet-articles, I'm afraid.
We started this way and was fine at the start, but problem is that we have many 1000s of customers now, and any bug that affects service causes us lots of hassles to clean up after.. (we had bad email problems and currency issues last month which took us a few days to recover from)...hence the need to test at least somewhat thoroughly..
Definitely an automated test bed, ideally runnable by just one command and by every developer, and then again by Buildbot when you check in code (we pushed a lot of our Selenium tests onto just Buildbot since we didn't have a lot of js and they took much longer than anything else to run).
If you don't need a lot of javascript testing, look at Twill - http://twill.idyll.org/ . Nice and simple (and very fast).
Otherwise, http://www.getwindmill.com/ and http://seleniumhq.org/ .
If you don't need a lot of javascript testing, look at Twill - http://twill.idyll.org/ . Nice and simple (and very fast).
Otherwise, http://www.getwindmill.com/ and http://seleniumhq.org/ .
Pull as much code as you can into self-contained, testable libraries that don't interact with the outside world except through defined APIs, then test the hell out of them with xUnit.
For everything else, there's Selenium.
For everything else, there's Selenium.
Right now it takes us a solid 2-3 days for 2 people to test through the site with all existing and new features... testing isn't hard, it just takes ages and reduces time we can do other stuff.
Question - what should we do?
Suck it up and keep doing it this way? Outsource the testing? Motivate some of our hardcore users to test for us? Try to build some automated test bed?