Custom Search

Metrics Used In Software Testing

Metrics Used In Testing

In this tutorial you will learn about metrics used in testing, The Product Quality Measures - 1. Customer satisfaction index, 2. Delivered defect quantities, 3. Responsiveness (turnaround time) to users, 4. Product volatility, 5. Defect ratios, 6. Defect removal efficiency, 7. Complexity of delivered product, 8. Test coverage, 9. Cost of defects, 10. Costs of quality activities, 11. Re-work, 12. Reliability and Metrics for Evaluating Application System Testing.

The Product Quality Measures:

1. Customer satisfaction index


This index is surveyed before product delivery and after product delivery
(and on-going on a periodic basis, using standard questionnaires).The following are analyzed:


  • Number of system enhancement requests per year
  • Number of maintenance fix requests per year
  • User friendliness: call volume to customer service hotline
  • User friendliness: training time per new user
  • Number of product recalls or fix releases (software vendors)
  • Number of production re-runs (in-house information systems groups)

2. Delivered defect quantities


They are normalized per function point (or per LOC) at product delivery (first 3 months or first year of operation) or Ongoing (per year of operation) by level of severity, by category or cause, e.g.: requirements defect, design defect, code defect, documentation/on-line help defect, defect introduced by fixes, etc.


3. Responsiveness (turnaround time) to users


  • Turnaround time for defect fixes, by level of severity
  • Time for minor vs. major enhancements; actual vs. planned elapsed time

4. Product volatility


  • Ratio of maintenance fixes (to repair the system & bring it into compliance with specifications), vs. enhancement requests (requests by users to enhance or change functionality)

5. Defect ratios


  • Defects found after product delivery per function point.
  • Defects found after product delivery per LOC
  • Pre-delivery defects: annual post-delivery defects
  • Defects per function point of the system modifications

6. Defect removal efficiency


  • Number of post-release defects (found by clients in field operation), categorized by level of severity
  • Ratio of defects found internally prior to release (via inspections and testing), as a percentage of all defects
  • All defects include defects found internally plus externally (by customers) in the first year after product delivery

7. Complexity of delivered product


  • McCabe's cyclomatic complexity counts across the system
  • Halstead's measure
  • Card's design complexity measures
  • Predicted defects and maintenance costs, based on complexity measures

8. Test coverage


  • Breadth of functional coverage
  • Percentage of paths, branches or conditions that were actually tested
  • Percentage by criticality level: perceived level of risk of paths
  • The ratio of the number of detected faults to the number of predicted faults.

9. Cost of defects


  • Business losses per defect that occurs during operation
  • Business interruption costs; costs of work-arounds
  • Lost sales and lost goodwill
  • Litigation costs resulting from defects
  • Annual maintenance cost (per function point)
  • Annual operating cost (per function point)
  • Measurable damage to your boss's career

10. Costs of quality activities


  • Costs of reviews, inspections and preventive measures
  • Costs of test planning and preparation
  • Costs of test execution, defect tracking, version and change control
  • Costs of diagnostics, debugging and fixing
  • Costs of tools and tool support
  • Costs of test case library maintenance
  • Costs of testing & QA education associated with the product
  • Costs of monitoring and oversight by the QA organization (if separate from the development and test organizations)

11. Re-work


  • Re-work effort (hours, as a percentage of the original coding hours)
  • Re-worked LOC (source lines of code, as a percentage of the total delivered LOC)
  • Re-worked software components (as a percentage of the total delivered components)

12. Reliability


  • Availability (percentage of time a system is available, versus the time the system is needed to be available)
  • Mean time between failure (MTBF).
  • Man time to repair (MTTR)
  • Reliability ratio (MTBF / MTTR)
  • Number of product recalls or fix releases
  • Number of production re-runs as a ratio of production runs

Metrics for Evaluating Application System Testing:

Metric = Formula


Test Coverage = Number of units (KLOC/FP) tested / total size of the system. (LOC represents Lines of Code)


Number of tests per unit size = Number of test cases per KLOC/FP (LOC represents Lines of Code).


Acceptance criteria tested = Acceptance criteria tested / total acceptance criteria


Defects per size = Defects detected / system size


Test cost (in %) = Cost of testing / total cost *100


Cost to locate defect = Cost of testing / the number of defects located


Achieving Budget = Actual cost of testing / Budgeted cost of testing


Defects detected in testing = Defects detected in testing / total system defects


Defects detected in production = Defects detected in production/system size


Quality of Testing = No of defects found during Testing/(No of defects found during testing + No of acceptance defects found after delivery) *100


Effectiveness of testing to business = Loss due to problems / total resources processed by the system.


System complaints = Number of third party complaints / number of transactions processed


Scale of Ten = Assessment of testing by giving rating in scale of 1 to 10


Source Code Analysis = Number of source code statements changed / total number of tests.


Effort Productivity = Test Planning Productivity = No of Test cases designed / Actual Effort for Design and Documentation


Test Execution Productivity = No of Test cycles executed / Actual Effort for testing


6 comments:

Vee Eee Technologies said...

Thanks for sharing your info. I really appreciate your efforts and I will be waiting for your further write ups thanks once again.

Unknown said...


Hey, nice site you have here! Keep up the excellent work!

Function Point Estimation Training

Amar said...

Thank you venu for sharing a useful information related to testing n you are maintaining a nice blog..keep posting regularly..

Unknown said...

Thanks for your information. QTP provide most precise and independent review about a software application. This automation testing tool is ideal to determine the performance and validity of a software application.
QTP Training Institutes In Chennai

Unknown said...

Thanks for sharing ERP On Cloud

Unknown said...

It is really very useful article to know the software testing. Thank you for providing the awesome information.
HP LoadRunner Training in Chennai | LoadRunner Classes in Chennai