Measuring the Success of Testing Teams

When measuring the overall success of an entire testing ODC, the first step is to establish a baseline to measure improvements against. The following examples revolve around measuring the productivity rates and quality of results found by a testing team.

Measuring the Finding of System Defects

Measuring the Finding of System Defects

There are two main factors that contribute to defect detection productivity: test case efficiency and execution efficiency. The charts above show how a team's development in Test Case and Execution Efficiency contribute to its Over time, we see the following changes:

  • Improvement of test case effectiveness
  • Improvement of test execution effectiveness
  • Increase in number of defects found

Measuring Peer Reviews of Requirements

Identifying defects in the requirement stage increases productivity because it can reduce the number of defects found in later stages of the software development lifecycle. In the example above, a project manager can see the peer review productivity is below the established control limited. By following the rate of defects found per hour, the project manager can perform a root cause analysis to develop a plan to correct the issue.

Quality of Results

Measuring the Defects Reject Rate
Defect reject rate allows managers to better understand the total cost of outsourcing by showing the percentage of the defects reviewed by the onsite team that were rejected. The result shown above is typical for many offshore engagements.  A lower defect reject rate means higher quality of work in both the testing and development teams. Rejected defects usually indicate that testers misunderstood the requirements or there was a communication breakdown between the developers and testers.

Measuring the Defects Density in Test Cases
Measuring the defect density in testing documentation is a good indication of the quality and confidence level of the testing team. Defects in test cases are usually detected during a test case review and can occur when testers misunderstand the requirements of a given test and when test documentation is incomplete or inaccurate. In the example above, the testing ODC was experiencing a high number of test case defects. Using this metric the ODC manager was able to find the root cause and create an action plan that involved peer review training.

Measuring the Post Release Defect Rate (Fault Rate)
Zero Post Release Defects means that a particular piece of software is designed as specified with acceptable performance and no downtime.  While this is the ultimate goal of every test team, few ever achieve this result. The more realistic goal is to match or perform better than the control limit. The example above shows the post release defect rate over a number of releases of an internal training project. As the test team received more training and gained experience, they were able reduce the fault rate below the control limit.

>>Send Whitepaper to a Friend