NDLI: System Level Combinatorial Testing in Practice -- The Concurrent Maintenance Case Study

Content Provider	IEEE Xplore Digital Library
Author	Wojciak, P. Tzoref-Brill, R.
Copyright Year	2014
Description	Author affiliation: IBM Syst. & Technol. Group, Poughkeepsie, NY, USA (Wojciak, P.) \|\| Haifa Res. Lab., IBM, Haifa, Israel (Tzoref-Brill, R.)
Abstract	Combinatorial test design (CTD) is an effective test design technique that reveals faults resulting from parameter interactions in a system. CTD requires a test space definition in the form of a set of parameters, their respective values, and restrictions on the value combinations. Though CTD is considered an industry best practice, there is only a small body of work on the practical application of CTD to industrial systems, and some key elements of the CTD application process are under-explored. Specifically, little consideration has been given to the process for identifying the parameters of the test space and their interrelations, how to validate the test space definition remains an open question, the application of CTD in reported work concentrates mostly on function or interface level testing and is hardly expanded to other levels of testing, and there is a significant lack of evaluation of the degree to which CTD helped improve the quality of the system under test. In this work, we analyze the continuous application of CTD in system test of two large industrial systems: IBM® POWER7® and IBM® System z®. For POWER7, CTD was used to design test cases for server concurrent maintenance. The application of CTD was in direct response to inconsistent reliability of those features on the prior POWER6® servers, and resulted in noteworthy quality improvements on POWER7. Success with POWER7 led to application of the methods to System z Enhanced Driver Maintenance testing, also featured in this work. To the best of our knowledge, this is the first comprehensive analysis of CTD usage in system test of industrial products, and the first analysis of long term use of CTD. We describe the methodology that we followed to define the combinatorial test space, while answering some unique challenges rising from the use of CTD to design functional test cases at system level rather than at interface or function level. We also describe our methodology for evaluating the test space definition and continuously improving it over time. In addition, we describe advanced CTD features that we found helpful for achieving an effective yet affordable test plan. Finally, we quantitatively and qualitatively evaluate the overall effectiveness of CTD usage, and show that it resulted in significantly improved server concurrent maintenance features.
Starting Page	103
Ending Page	112
File Size	244525
Page Count	10
File Format	PDF
ISBN	9781479922550
DOI	10.1109/ICST.2014.23
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2014-03-31
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Software testing Combinatorial Test Design Maintenance engineering Hardware Software Combinatorial Testing Case Study Servers Combinatorial Testing Vehicles
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

From system level to defect-oriented test: a case study

Improved test and maintenance via concurrent engineering

The design of test case based on combinatorial and orthogonal experiment

Regression Test Selection of Manual System Tests in Practice

Language-based electronics systems design automation

Design, implementation, and case study of a function level unit test environment

MFQ & PPDCS - Test Analysis and Test Design for Large Embedded Software Systems

Combinatorial Testing for an Automotive Hybrid Electric Vehicle Control System: A Case Study

A method of verification in design: an operating system case study

System Level Combinatorial Testing in Practice -- The Concurrent Maintenance Case Study

Similar Documents

From system level to defect-oriented test: a case study

Improved test and maintenance via concurrent engineering

The design of test case based on combinatorial and orthogonal experiment

Regression Test Selection of Manual System Tests in Practice

Language-based electronics systems design automation

Design, implementation, and case study of a function level unit test environment

MFQ & PPDCS - Test Analysis and Test Design for Large Embedded Software Systems

Combinatorial Testing for an Automotive Hybrid Electric Vehicle Control System: A Case Study

A method of verification in design: an operating system case study

System Level Combinatorial Testing in Practice -- The Concurrent Maintenance Case Study