Test Coverage is secondary to defect ratio reduction

Code Test Coverage is secondary to defect ratio reduction. I see so much effort put in tracking what is called "test coverage" just to find out that the more code test coverage a project has does not necessarily has the impact one would expect on defect ratio reduction. So, while I do not oppose test coverage metrics I strongly believe those are secondary to defect ratio metrics.

Track bugs versus any other kind of issues reported by users and your existing automation tests. The ratio must go down if you have the proper test coverage. So test coverage is obtained as a result of measuring defect ratio and adjusting the system to have the correct tests in place. It is not the other way around, test coverage measurement cannot guarantee a defect ratio reduction. Test coverage must increase to address defect ratio reduction. It is way simpler and natural to build automated tests out of manual exploratory tests and measure the defect ratio than relying on complicated algorithms that must test all variants of the intrinsics of software to come up with a percentage of test coverage. So you save time when concentrating on defect ratio with the added value that your software works as expected even if the tool says the test coverage is 99%.

Often overlooked are Marketing and Sales criteria, log files and user feedback. All tests do not rank the same. There is always a crucial part of the system which cannot fail without generating tangible cost, then some part that could fail without such cost. Prioritizing what to test first is as important as prioritizing what to build first.

If we accept the fact that any piece of software is ultimately a "process integration and report unit" which needs input variables and returns output variables then we would agree we should know 100% of the cases what we should obtain as output given any combination of input. Clearly automating such tests and making sure they do not fail is the correct first measurement to apply to our software. The more bugs we get, the less quality the code has and that could be independent of any current code test coverage reported by any tool.

I believe the real issue we face nowadays is that test is still a second thought in each phase of the SDLC. That is something we need to change.

If you are transiting the agile path, if you have gotten into the lean wagon, you know that limiting WIP and knowing your cycle time is just the beginning. To provide continuous improvement you must plot also how your defect ratio is minimized over time. Without it, test coverage won't matter. Perception is reality.

No matter how much test coverage you claim to have if the software fails the user, it is failing the Business.