Introduction

The “World Quality Report” for 2013-14 was recently published by the popular triumvirate consisting of HP, Cap Gemini and Sogeti. While the “key findings” section of the report discussed many important items such as Cloud Testing, Test Environment/Test Data challenges, Agile testing, etc., the “Recommendations” segment had a very interesting and useful remark about the need for reporting “Business-Oriented Metrics to demonstrate the value of QA”.

Metrics has always been an integral part of software testing projects but the nature and type of metrics collected and shared have changed over time. About a decade back, independent testing projects reported metrics such as Schedule Variance, Effort Variance, Tester Efficiency, Testcase Effectiveness, Defects by Severity/Priority, % of valid vs invalid defects, Defect Leakage etc. Some of these are still relevant today but in today’s context of outcome based services which includes models such as pay per testcase/defect et al. it becomes important for testing vendors to share metrics that exemplifies productivity, efficiency and effectiveness in conjunction with business benefits.

Metrics as a Measure of Effectiveness

Traditional metrics, particularly the ones based on defects were used to measure the team’s effectiveness. The number of defects that got leaked to production (Defect Leakage) or what is alternatively referred to as “Defects missed during a release” reflects the team’s ability and product knowledge. Another metric used to measure team effectiveness includes the “Percentage of valid vs invalid defects”. This metric reveals the overall product knowledge of the team. These metrics can also be captured at an individual level but generally are measured at a team level.

Similarly, an individual tester’s efficiency can be assessed using the “No. of Test cases executed per day/month” and “No.of defects found per day/month” metric. The individual tester’s grasp on the product is also assessed by the “Defects found by severity” metric. It is inferred that the tester has good product knowledge if he consistently files high severity defects.

The test cases designed by a tester also reflects his product knowledge if it helps in uncovering defects. This is measured using the metric “Testcase Effectiveness”. Testcase Effectiveness at a broad level is usually derived by comparing “the number of defects that were identified through structured testing vs ad hoc testing”. If number of defects identified through structured testing is high it implies high quality of test cases. In case of high number of ad hoc defects found is high, it is generally means poor quality of test cases.

Usually a tester with good product knowledge is expected to design high quality test cases which can effectively find defects. This metric can also be measured at an individual level.

The New Age Software Testing Metrics

The metrics that we discussed in the previous section still hold good in many testing projects but their direct relevance to “business benefits” may not be very high. Apparently, many projects today have moved away from the traditional waterfall model to Agile and variants of Agile. This has opened up the possibility of a whole new breed of test metrics. The metrics that can indicate the business value of the quality assurance exercise. This is important as the management is keen to understand the value delivered on their investment.

Given the scenario, what kind of software testing metrics should vendors focus on? There could be specific defect based or test coverage based metrics that needs to be collected keeping in mind the lifecycle model, application needs etc. However, given the focal point of this article is about identifying metrics that add business value, the following can be actively considered:

How QA has contributed to reduce time to market?
Cost savings achieved by preventing defects
Test Automation ROI(Return on Investment) over a time period
Relationship between testing coverage and the risks and complexities of the systems

Other than the ones mentioned above, testing teams can also device their own business-oriented metrics that captures business effectiveness in their context.