Ask HN: Are you doing automated UI testing – why or why not?
I have worked in several companies that have had goals of automated UI regression test suites, but I've never worked at a company that pulled it off successfully. I can name off a case-by-case list of reasons why these efforts have failed, but I am curious if the community is seeing similar trends.
4 comments
I'll add one example from a past company I worked for:
- We had a manager and one internal developer assigned part-time to an effort to seed an automation effort.
- The front-end of the application was Flex.
- Most QA analysts at the company had never written code before.
- We attempted to use Selenium.
As you can imagine, this was a crash and burn for several reasons. The company couldn't get enough buy-in internally to dedicate more developer time to seed the effort, and the long-term approach was not well thought out. A few tests were deployed, but they were brittle because they relied on DOM elements. This was partially because QA analysts were expected to be able to use a Selenium "capturing" tool to create tests without writing any code.
Writing this out now, I think the root problem is that at the management level, the sentiment was just that we should "do automation" and no one really understood how much that would involve. There doesn't seem to be any turn-key solution out there that is designed to fit in with the rest of a company's development processes.
When there isn't a clear path for solving a problem, it becomes a much more expensive problem to solve. In this case, the cost far out-weighed any benefit that the company would see.
With that said, I am still optimistic about the potential for effective UI automation testing and I'd love to hear more stories.
- We had a manager and one internal developer assigned part-time to an effort to seed an automation effort.
- The front-end of the application was Flex.
- Most QA analysts at the company had never written code before.
- We attempted to use Selenium.
As you can imagine, this was a crash and burn for several reasons. The company couldn't get enough buy-in internally to dedicate more developer time to seed the effort, and the long-term approach was not well thought out. A few tests were deployed, but they were brittle because they relied on DOM elements. This was partially because QA analysts were expected to be able to use a Selenium "capturing" tool to create tests without writing any code.
Writing this out now, I think the root problem is that at the management level, the sentiment was just that we should "do automation" and no one really understood how much that would involve. There doesn't seem to be any turn-key solution out there that is designed to fit in with the rest of a company's development processes.
When there isn't a clear path for solving a problem, it becomes a much more expensive problem to solve. In this case, the cost far out-weighed any benefit that the company would see.
With that said, I am still optimistic about the potential for effective UI automation testing and I'd love to hear more stories.
- We use selenium, test cases are written in python.
- We use the webdriver, recording never worked
- Followed the page model
- Had to build in retries on getting elements to make tests reliable
- You need upper management buy-in. Took us 8 months to start getting it right
- I would say our team members are intermediate with python
- Devs currently do not write tests. But they asked to start writing which is a good sign
- Yes some days you just spend fixing broken tests. But you get experience on how to write them better over time
Tips:
- Work with devs to get build the page with proper selectors (use ID if not, then us CSS selectors, don't use xpath)
- Use APIs to seed data for your tests
- Get to the page you need to test as fast as possible by hitting the urls directly, using APIs to seed what data u need. Manually injecting cookies to get to the state you need.
Costs:
- 2 full time engineers only working on automation for 8 months.
- Once you have your framework in place writing tests become a lot easier
- We use the webdriver, recording never worked
- Followed the page model
- Had to build in retries on getting elements to make tests reliable
- You need upper management buy-in. Took us 8 months to start getting it right
- I would say our team members are intermediate with python
- Devs currently do not write tests. But they asked to start writing which is a good sign
- Yes some days you just spend fixing broken tests. But you get experience on how to write them better over time
Tips:
- Work with devs to get build the page with proper selectors (use ID if not, then us CSS selectors, don't use xpath)
- Use APIs to seed data for your tests
- Get to the page you need to test as fast as possible by hitting the urls directly, using APIs to seed what data u need. Manually injecting cookies to get to the state you need.
Costs:
- 2 full time engineers only working on automation for 8 months.
- Once you have your framework in place writing tests become a lot easier
I think the question is incomplete(at least I think so).
Its not about automating the UI tests for regression testing,but to ensure that regression testing is effective.Most of the times,this is driven through the UI, which could be flaky.
As the test pyramid paradigm states , having a lot of UI tests, depending on the functionality & complexity could be an anti pattern. Essentially , a lot of the functionality can be tested using by writing unit tests & integration tests and the end to end business use cases should be automated through the UI just to ensure it works.
If it is possible to drive the state of the application using an API or URL, to the target state and then we assert elements to check the UI.
This enables faster feedback and is less flaky.This is how we approached testing in our team.Depending on the objective , we would use Selenium ,JUNit , Cucumber or corresponding ones in a given programming language.
Its not about automating the UI tests for regression testing,but to ensure that regression testing is effective.Most of the times,this is driven through the UI, which could be flaky.
As the test pyramid paradigm states , having a lot of UI tests, depending on the functionality & complexity could be an anti pattern. Essentially , a lot of the functionality can be tested using by writing unit tests & integration tests and the end to end business use cases should be automated through the UI just to ensure it works.
If it is possible to drive the state of the application using an API or URL, to the target state and then we assert elements to check the UI.
This enables faster feedback and is less flaky.This is how we approached testing in our team.Depending on the objective , we would use Selenium ,JUNit , Cucumber or corresponding ones in a given programming language.
Yes, I along with my team members have helped my company come from zero automation to decent coverage with UI tests. We have tests that run reliably for Web, iOS and Android.
It has certainly improved our deployment time.
If you describe some of the your problems. I can tell you how we overcome them.
It has certainly improved our deployment time.
If you describe some of the your problems. I can tell you how we overcome them.
Thanks for the response. I have several questions.
What is the division of labor between developers who work on the tests and (perhaps non-coding) QA analysts?
What software are you using?
How large is the application, and would you mind describing the nature of the application?
Do you have any clear cost metrics on the effects of automation testing? E.g., are you able to operate with a smaller QA team, measurably lower incidence of bugs discovered in production, etc.?
Thanks again.
What is the division of labor between developers who work on the tests and (perhaps non-coding) QA analysts?
What software are you using?
How large is the application, and would you mind describing the nature of the application?
Do you have any clear cost metrics on the effects of automation testing? E.g., are you able to operate with a smaller QA team, measurably lower incidence of bugs discovered in production, etc.?
Thanks again.