First, What is Regression Testing?
Whenever developers change or modify their code, even a small tweak can have unexpected consequences. If these unexpected consequences amount to a bug that comes with a loss of functionality that was formerly present, that is referred to as a "regression". Regression testing is a type of software testing used to help identify if regressions have occurred and determine whether these new problems are the result of changes to the codebase or configuration.
With how complex software continues to become, you are probably wondering right now if there are regressions in your website or app that you aren’t even aware of. If you don’t utilize some form of testing (automated or otherwise) of any type then the chances are highly likely that you’ve had some bugs. Even some past Drupal core updates have introduced regressions into the code, notably the updates for 7.29 and 7.43. And Drupal core updates are tested more stringently than anything else on the Drupal platform.
The Case For Automating Your Tests
Sometimes, these regressions may not be apparent right away, perhaps they involve some kind of feature that is deep inside a workflow and not particularly obvious. That scenario is one of the main reasons you will want to automate your tests. The bigger and more complex your website is the harder it becomes to manually test everything on a regular basis.
Too Many Things to Test
Without a suite of regression tests in place to catch as many critical feature regressions as possible, something could break and you may not even know it. Fear over these kinds of uncaught issues should be reason enough to want to utilize regression testing, but it is certainly not a cheap or quick kind of process to implement. You might identify that to cover all of your critical features you'll need upwards of hundreds of tests. That is why you'll want to implement an automated regression testing scheme.
Even Careful People Make Mistakes
Another reason to implement automated regression testing, and to automated processes in general, is to prevent human error within the actual testing process. If a person is manually testing features as part of the quality assurance process, it is only a matter of time until they accidentally skip a step or get complacent with their testing workflow. With how the universe and bad luck tends to operate, that moment will be the time a bug slips through into your live website.
Ideally, you will have an automated testing system in place and use a strictly reproducible and verifiable series of tests. Before applying a change, automated tests for all critical features should be in place. After a change is applied, the website is retested in those selected areas to detect whether the change created new bugs or issues. If a problem is found, fix it, and then rerun the tests. This kind of pattern is far more reliable than a person going through a series of written down steps manually.
Ultimately, the more a system requires human input to manage a process, the more likely that human error or just plain forgetfulness will cause the whole system to break down.
Can Automated Regression Testing Be Realistically Affordable?
Automated regression testing can be kept cost-effective and simple to build out using tools like Ghost Inspector. With this service, you can use a browser-extension to record activities on your website, and automatically create your regression tests while even initially testing the features you are creating during the development process.
Additionally, the ability to schedule automated testing runs of your tests and get reports of found issues ensures that you will catch regressions if something changes later on.
As with most forms of automated testing, setting a regression-testing program on autopilot is not a surefire solution, and some conscious oversight and input is generally still needed to ensure that your tests catch all the bugs they should. As our website changes, your tests will need to be updated to match the new features. If you have hundreds of tests for your website, then this will certainly not be an easy task.
In the end, you must determine for yourself or your clients what the value is in this form of testing. If uncaught regressions and bugs could prove to be expensive and hazardous to the project's reputation, the time and cost of implement automated regression test may end up feeling like an appropriate investment to make.