QW2001 Paper 4A1

Mr. Henk Keesom , Dr. John Musa
(Ortho-Clinical Diagnostics)

Using Test Data to Calculate Software Reliability Growth

Key Points

Presentation Abstract

This case study describes how one company used existing product development test history to calculate the software reliability growth of three embedded real-time software products. It includes techniques that were used and can be used by others to do the same calculations with their existing product development test data. It also discusses techniques to produce better data and thus better software reliability information in the future.

The techniques described are useful to embedded product software developers and their organizations as many have access to extensive product development test performance data. They can use this data and these techniques to determine their software reliability growth during testing and their product’s current software reliability. In addition, suggestions are provided to provide better future tracking of software reliability.

Ortho-Clinical Diagnostics develops and manufactures blood analyzers for professional use in Clinical Chemistry Laboratories - containing custom developed embedded real-time control system software. The three products used as examples have been in field use and software upgraded with additional features and functions for over ten years. Thus these products provide an extensive amount of product development test history of software reliability.

The existing product development test history was used with CASRE (Computer Aided Software Reliability Engineering). CASRE is a widely accepted tool used to calculate and graph software reliability. CASRE is available on the CD-ROM that is provided with the Handbook of Software Reliability Engineering, by Michael R. Lyu.

The normalized MTTF from the three products that were used are presented and show the software reliability growth over time - and that with new upgrades (software versions) there were short term decreases (as one would expect with new additional features) in the reliability.

Techniques for converting product development test history data from calendar time to execution time are provided. This data was in calendar time but it required conversion to an approximate execution time before CASRE would produce meaningful results. Methods for handling data sets larger than CASRE can handle are included. Steps are provided to take CASRE output and produce graphs not included in CASRE.

Recommendations on how to enhance test data in the future to make calculations of software reliability more predictive are described.

About the Author

Hendrik J. Keesom is a manager of Software Verification and Validation for Ortho-Clinical Diagnostics in Rochester, NY. Keesom is a software verification and validation engineer and software engineer with over 20 years of real-time embedded software and hardware experience. Keesom is one of the co-editors of the HL-7 Chapter on Clinical Laboratory Automation and the NCCLS standard: Laboratory Automaton: Communications with Automated Clinical Laboratory Systems, Instruments, Devices , and Information Systems.

John D. Musa is one of the creators of SRE, with more than 30 years varied and extensive experience as a software development practitioner and manager. Principal author of the highly-acclaimed pioneering book Software Reliability and author of the practical Software Reliability Engineering, Musa has published more than 100 papers on SRE. Elected IEEE Fellow in 1986 for many seminal contributions, he was recognized in 1992 as the leading contributor to testing technology. His leadership has been noted by every recent edition of Who’s Who in America and American Men and Women of Science. Musa, widely recognized as a leader in SRE practice, initiated and led the effort that convinced AT&T to make SRE a “Best Current Practice.” Musa has helped a wide variety of companies with a great diversity of software-based products deploy SRE. He is an experienced international speaker and teacher (over 200 major presentations) A founder of the IEEE Technical Committee on SRE, he is closely networked with SRE leaders, providing a broad perspective.