“How can I be confident your code meets requirement?”
“How do we know your changes don’t break other features?”
“Can you prove to me your scripts do everything they are supposed to?”
These are some questions that can be answered by automation testing and increase our confidence in the solution.
Historically testing is often done manually. Manual testing is performed by carefully executing pre-defined test cases, comparing the results to the expected behaviour, and recording the results. This is repeated each time the source code changes, and the process lends itself to errors. The manual nature makes it difficult to execute on multiple platforms, not to mention the increased difficulty to conduct thorough testing as the code and the solution expand. The significance and benefit of automation testing will become exponential as the solution grows.
Automation testing solves many of the manual testing issues. It reduces testing efforts, enables faster delivery of capabilities at a more affordable cost. More robust testing also helps build higher quality solutions with less effort.
Automation testing involves repeatedly executing the same set of test-suites each time a change is made to the code. This exposes bugs early in the development cycle and reduces the risk of delivering a faulty solution.
There is an upfront time and effort investment needed to build the infrastructure for automation testing. Once built, these tests are reused and rerun when there is a change to the source code. This pays back dividends in terms of the reduced cost of failure. Specifically, cost in time, resources, money for the company if problems arise when deploying solutions with bugs into production, as it escaped manual testing. This can also result in the loss of customers and have a negative impact on the company’s reputation.
Once the infrastructure is in place, rerunning the tests takes significantly less time than manual testing, and costs nothing extra. It is also less prone to errors and minimal labour is required.
With CI/CD, you have the option to automatically run the tests per each sprint, commit, feature change; failures can be addressed early and fixed early. With the usual sprint of 2-3 weeks length, thorough testing to ensure the new feature works and does not break the existing functionality is challenging with manual testing. This would involve manually retesting all features from previous sprints which is pricey, and inefficient. With automated tests, retesting existing features each sprint is quick and provides assurance that the solution is working as intended.
Rerunning the tests on all servers is also a quicker task.
Automated testing provides confidence to developers, management, and customers that the product works as intended. By rerunning the same tests multiple times, getting feedback for any failures early, and fixing them. Going through this loop on the regular drives up the quality of the product on the delivery date. This process would be costly and inefficient to replicate through manual testing. Though there is an upfront cost, time, and effort needed to develop the infrastructure for automated testing; in the long term, gaining confidence in the quality of the product, reducing the risk of production issues and reputational damage associated with it, this investment will pay for itself in the long term.
Adatis has successfully implemented automation testing, catching issues early, and met with great feedback from clients.
We use Pester Framework to build and perform automation testing. To read more about the technical details involved, Regression Testing a Data Platform with Pester lays out the foundations involved.