Regression testing is a type of software testing used to determine whether new problems are the result of changes to the codebase or configuration. Whenever developers change or modify their code, even a small tweak can have unexpected consequences.
Before applying a change, tests for all critical features should be in place. After a change is applied, the website is retested in those selected areas to detect whether the change created new bugs or issues.
If you are wondering how often this might become an issue, even some past Drupal core updates have introduced regressions into the code, notably the updates for 7.29 and 7.43. Sometimes, these regressions (a regression is essentially a loss of functionality) may not be apparent right away.
Without a suite of regression tests in place to catch as many critical feature regressions as possible, something could break and you may not even know it. Fear over these kinds of uncaught issues should be reason enough to want to utilize regression testing, but it is certainly not a cheap or quick kind of process to implement.
You might identify that to cover all of your critical features you'll need upwards of hundreds of tests. That is why you'll want to implement an automated regression testing scheme.
Automated regression testing can be kept cost-effective and simple to built out using tools like Ghost Inspector. With this service you can use a browser-extension to record activities on your website, and automatically create your regression tests while even initially testing the features you are creating during the development process.
Additionally, the ability to schedule automated testing runs of your suite of tests, and get reports of found issues, ensures that you will catch regressions if something changes later on.
As with most forms of automated testing, setting a regression-testing program on autopilot is not a surefire solution, and some conscious oversight and input is generally still needed to ensure that your tests catch all the bugs they should. As our website changes, your tests will need to be updated to match the new features. If you have hundreds of tests for yor website, then this will certainly not be an easy task.
Ultimately though, you must determine for yourself or your clients what the value is in this form of testing. If uncaught regressions and bugs could prove to be expensive and hazardous to the project's reputation, the time and cost of implement automated regression test may end up feeling like an appropriate investment to make.