There is a pervasive problem in the software industry stemming from a misunderstanding of automated testing. Companies are investing increasing amounts of money and time into automated testing strategies and test suites, but are not seeing an increase in quality or value delivery to customers. Thankfully automated testing is now being seen as a necessary complement to manual, exploratory testing. However, many organizations don’t understand the costs and benefits of the various types of automated tests available to them.
Two big misconceptions set the stage for a risky and expensive approach to automating tests. First, is the idea that automated testing is the process of automating the steps of manual QA; automating the script the manual tester would follow. Second, is that these automated tests should be written by the QA organization, outsourced, or written by a team of “test engineers” who are not the team writing the software being tested.
🤔 Who is responsible for Quality?
Often I see this mindset take hold in organizations that are struggling with establishing cross-functional development teams. They cling to the outdated idea that “testing is QA’s responsibility” and developers should be focusing on writing production code. This disconnect makes it very difficult for organizations to improve their automated test strategy. In companies like these, I often remind them that cross-functional teams own their quality, not a department.
🛕 The Test Pyramid
The test pyramid describes an approach to creating sustainable test suites. The bottom of the pyramid consists of unit tests or, more recently known as, microtests. The majority of the test suite should consist of these tests. There should be a lot of them, they should be VERY small, verify a VERY small area of code (i.e., a few lines of production code) and they should be VERY fast (i.e., they should run in a few milliseconds). These tests should be able to run in isolation on developer computers.
The tests that occupy the middle tier are the integration and and API tests. These tests cover more system scope and are more “connected” (e.g., they connect to a database, the filesystem, etc.) than microtests, so there should be much fewer of these in the test suite.
Finally, the top is UI-driven or end-to-end tests. There should be very few of these, as they take longer to execute and have an increased likelihood of identifying false failures.
➗ The inverse relationship of scope and detail
The most important thing to understand in this pyramid is the inverse relationship between scope and detail. At the bottom of the pyramid, microtests should be testing all of the detail of the system so their scope should be very small. They should verify only a few lines of code in the system. As you move up the pyramid, the scope of the test should increase. When scope increases the detail should decrease.
Organizations adopt UI-testing tools like Selenium and attempt to wrap them in complex, homegrown testing frameworks. They use these frameworks to create comprehensive suites of end-to-end tests to replace manual QA testing, without understanding the drawbacks of this approach. I’ve seen custom frameworks that use Cucumber (which is a fine tool for BDD) to drive UI-testing-tools to eventually exercise end-to-end tests. So much of the organization’s time and money is spent creating and maintaining this complex framework and the brittle tests the framework supports.
🎉 The Test Ice Cream Cone
The result of this approach is the Test Ice Cream Cone, where a majority of the test suite is occupied by end-to-end and UI-driven tests. When I encounter this, my goal is to help the organization shift their focus towards aligning their test suite to the test pyramid.
⬇️ Push Test Down
Some organizations will resist minimizing these end-to-end tests because they managed to find an issue before it got to production. They see these tests as beneficial since they have decreased the time it takes for manual testing. It’s understandable to see why they wouldn’t want to abandon these tests, however, the fact remains that they are slower and less reliable than tests lower in the test pyramid. My advice has always been to push tests down. If a bug was caught in an end-to-end test or integration test, write a microtest that exposes the bug and then fix it. I was very pleased to see similar advice in The DevOps Handbook:
Not only are errors detected during integration testing difficult and time-consuming for developers to reproduce, even validating that it has been fixed is difficult (i.e, a developer creates a fix but then needs to wait four hours to learn whether the integration tests now pass). Therefore, whenever we find an error with an acceptance or integration test, we should create a unit test that could find the error faster, earlier, and cheaper.
– Gene Kim, Jez Humble, Patrick DeBois & John Willis – The DevOps HandBook – Chapter 10
🆘 Lacking of Unit Tests
The most common reason organizations adopt this approach is because they are lacking a high-quality and trusted unit test / microtest suite. Often teams abandon unit testing because they struggle with trying to wrap unit tests around untestable code. Other teams may fight through the challenges and wrap unit tests around untestable code and end up with a suite of tests that are a drag instead of an accelerator.
📈 Conclusion
One key attribute of successful, high-performing software teams is the reliance on fast feedback loops. Test suites are one of the main feedback loops available to teams. Focusing on fast, reliable, repeatable tests is an accelerator for software organizations. Automated test suites are essential, but be aware of the pitfalls in creating them.
Original Article: https://anthonysciamanna.com/2019/10/20/avoiding-automated-testing-pitfalls.html