Six Sigma Quality Resources for Achieving Six Sigma Results
Click To Learn More About PremiumLinks
 Home > Tools & Templates  > Design Of Experiments (DOE) Search:
 
 for    
Publications
Marketplace
| iSixSigma
Stuff
| iSixSigma
Blogosphere
| Events
Calendar
| The
Dictionary
| Discussion
Forum
| Find
a Job
| Post
a Job
| Industry
News
| Newsletter
Signup
| Sigma
Calculator
| Online
Surveys
iSixSigma Magazine Signup
 iSixSigma Live!  
  Live! Home
  2010 Summit & Awards
  2010 Energy Forum
  2010 DoD Symposium
 Free Newsletters!  
  Sign Up Now!
  Manage Subscriptions
  New To Six Sigma?
  Six Sigma Q&A
  Cert. Practice Test
  Problem Solving Wizard
 Channels 
  Europe
  Financial Services
  Healthcare
  Military
  Software / IT
 Quality Directory 
  Best Practices
  Certifications/Awards
  Consultants
  Culture Evolution
  Methodologies
  News & Events
  Organizations
  Product/Service Guides
  Statistics & Analysis
  Tools & Templates
   DOE
   FMEA
   Glossary
   Histogram
   Pareto
   Poka Yoke
   SIPOC
   Software
  Voice of the Customer
  Free Whitepapers
 Related Topics 
  Innovation
  Outsourcing/Offshoring
  Business Process Mgt
 Quick Access 
  Help
  Search
  Advertise Here
  Article Archives
  Newsletter Archives
 User Feedback 
  Please suggest site
  improvements.
 
  [ larger form ]

Design Of Experiment For Software Testing

Bookmark This Page Bookmark This Page
Email This Page Email This Page
Format for Printing Format for Printing
Cite This Article Cite This Article
Submit an Article Submit an Article
Six Sigma Article Archive Read More Articles
Related Tools & Articles
  • Six Sigma Quick Poll
    What is your familiarity level with Design of Experiments (DOE)?
    Very familiar
    Familiar
    Vague familiarity
    Heard of it, but never done a DOE
    Discussion Forum
    "You write: 'insisting on a transfer function for a new design is stupid at best.' What about a DOE that takes into account price, functionality and user interface? It's obvious that there is an inflection point and maximum for the intersection of these (unless you're Microsoft and then you can bundle it with Office, charge as much as you want and build every feature imagineable into it). A DOE is a transfer function, right? It helps you understand how the various inputs affect the output (number of purchases)."

    Six Sigma Software Development
    Download Products
    By Madhav S. Phadke

    When DOE is used for software testing, there is a large amount of savings in testing time and cost. Various users in automotive, telecommunication and defense industries report big productivity improvements to their traditional testing methods. This success is due to two major important factors: 1. Focused attention on the usage of software, and 2. A mathematically sound way to span the entire operating domain with minimum set of test cases.

    The job of a software tester is to attempt to break the system in every possible way so that all faults will be detected, which will therefore increase the likelihood of delivering fault-free software to the customer.

    Let us set forth a simple scenario for software testing. Consider a function to be tested with four parameters: A, B, C, and D. These parameters could be the arguments of the command line entered from the terminal, the state of an interface, input from a connecting device, or the initial states of internal parameters. Suppose each parameter has three possible levels. This parameter-level table specifies the test domain consisting of 81 possible combinations of the test parameter levels.

    Consider the example of testing copy machine functionality for test parameters and levels. Here, parameter A could be number of original papers and levels being 1 paper, 10 papers or 51 papers. Parameter B could be Duplex with levels being one-sided copy or two-sided etc. as shown in Table 1. The software has to perform well for all possible values of these parameters.

    Table 1: Parameters And Levels
    Test ParametersLevel 1Level 2Level 3
    A. Number of Originals11051
    B. Duplex1 to 11 to 22 to 2
    C. CollatingNoneYesStapled
    D. InterruptNonePanelTray

    A thorough analysis of the requirements document and understanding of the usage of the software is needed to prepare the parameter-level table and test plans. Knowledge of how the software is written is usually not necessary to prepare this table.

    Orthogonal Array-Based Test Cases
    Table 2 shows the test plan using OA (Orthogonal Array) L9. It has nine rows and four columns. The rows correspond to test cases; the columns correspond to the test parameters. Thus, the first test case comprises Level 1 for each parameter, i.e., it represents the combination A1, B1, C1, D1. The second test case comprises combination A1, B2, C2, D2, etc.

    Table 2: Test Plan Using OA L9
    Test NumberNo. of OriginalsDuplexCollatingInterrupt
    111 to 1NoneNone
    211 to 2YesPanel
    312 to 2StapledTray
    4101 to 1YesTray
    5101 to 2StapledNone
    6102 to 2NonePanel
    7511 to 1StapledPanel
    8511 to 2NoneTray
    9512 to 2YesNone

    An orthogonal array has the balancing property that, for each pair of columns, all parameter-level combinations occur an equal number of times. In OA L9, there are nine parameter-level combinations for each pair of columns, and each combination occurs once. Taguchi [2] and Madhav S. Phadke [1] provide a comprehensive discussion of orthogonal arrays and their selection for specific applications.

    By conducting the nine tests indicated by L9, we can accomplish the following:

    1. Detect and isolate all single-mode faults. A single-mode fault is a consistent problem with any level of any single parameter. For example, if all cases of factor A at Level A1 cause error condition, it is a single-mode failure. In this example, tests 1, 2, and 3 will show errors. By analyzing the information about which tests show error, one can identify which factor level causes the fault. In this example, by noting that tests 1, 2, and 3 cause an error, one can isolate A1 as the source of the fault. Such an isolation of fault is important to fix the fault.


    2. Detect all double-mode faults. If there exists a consistent problem when specific levels of two parameters occur together, it is called a double-mode fault. Indeed, a double-mode fault is an indication of pairwise incompatibility or harmful interactions between two test parameters.


    3. Multimode faults. OAs of strength 2 can assure the detection of only the single- and double-mode faults. However, many multimode faults are also detected by these tests. This can be understood by studying the interaction tables and the geometric view of the test cases presented in the next section.

    Geometric View of Test Cases
    Software faults can be divided into two categories: region faults and isolated faults, as shown in Figure 1. Faults that occur for only one specific combination of test parameter levels, such as by a data-entry error, are called isolated faults. Assurance against isolated faults is not possible without testing all combinations. However, when the logic is faulty, there is a tendency for a region of the test domain to exhibit malfunction. Such faults are called region faults. Single-mode and double-mode faults are special cases of region faults. Orthogonal Array based testing is highly effective for the detection of region faults with a relatively small number of tests.

    Figure 1: Region Faults And Isolated Faults
    Region Faults and Isolated Faults

    A geometric picture of the test cases can provide an insight into the benefits of OA-based test cases in the detection of region faults. To facilitate geometric visualization, take the example of software with only three test parameters A, B, and C. The test domain is cube-shaped and consists of 27 lattice points. Test cases based on the OA L9 and those based on the one-factor-at-a-time method are graphically displayed in Figure 2 The OA-based test cases are dispersed uniformly throughout the test domain.

    Figure 2:
    Test Cases Based On The L9 OA
    And One Factor At A Time
    Test Cases Based On The L9 OA And One Factor At A Time

    A Case Study
    A comprehensive case study of orthogonal array based testing was published by AT&T in 1992 [3]. The study detailed regression testing of a PC (IBM format) and local area network-based electronic mail software called StarMail. Originally, a test plan was prepared that consisted of 1,500 tests that would take 18 weeks to conduct. However, because of development delays, the testing time had to be cut short to only eight weeks. The lead test engineer had, therefore, prepared an alternate plan involving 1,000 tests that would take two people eight weeks. But he was unsure about the quality of testing, i.e., its ability to detect all faults. To alleviate these problems, a test plan was prepared using the orthogonal array based Robust Testing method.

    The Robust Testing method began with the division of the functions of the StarMail software into 18 categories, test parameters and levels for each category were found, and an OA to select test cases for each category was used. For example, for the copy/move function, the selected OA L27 was used to ensure all pairwise combinations of these parameters. Different parameter tables were prepared for remaining 17 categories.

    In all, only 422 tests were needed to test the software. These tests identified 41 faults in the software, which were fixed, and the software was released. Two years of operation in the field generated no faults within the scope of testing, which indicated that relative to the scenarios encountered, the test plan was 100 percent effective in identifying faults. Had AT&T run the alternate test plan that involved 1,000 tests, the lead tester estimated they may have found only 32 of the 41 faults. Compared to the original testing plan, Robust Testing required 50 percent less testing effort, identified 28 percent more faults, and was more productive (number of faults detected per tester week) by a factor of 2.6.

    Regression Testing
    Orthogonal array based testing is very useful in regression testing. After fixing the faults, one can easily prepare a new test plan using array rotation technique. A new regression test plan is depicted in Table 3, which is different from the test plan shown in Table 2.

    Table 3: New Regression Test Plan
    Test NumberDuplexCollatingInterruptNo. of Originals
    11 to 1NoneNone1
    21 to 1YesPanel10
    31 to 1StapledTray51
    41 to 2NonePanel51
    51 to 2YesTray1
    61 to 2StapledNone10
    72 to 2NoneTray10
    82 to 2YesNone51
    92 to 2StapledPanel1

    A lot of time and cost savings are available during this phase.

    Conclusion
    Use of orthogonal array based testing has demonstrated to produce superior test plans that improve testing productivity by a factor of 2. This method is found effective in testing the incremental work done in all stages of software development. These stages include writing requirements, selecting architecture, designing the system (functional breakdown), unit testing, platform testing, integration testing, prototype testing, and system testing.

    About The Author
    Dr. Phadke pioneered the application and development of the Taguchi Method / Robust Design method in USA and is a recipient of the Taguchi Award, 1985. He has worked closely with Dr. Genichi Taguchi since 1980, and they have co-authored several articles, advancing the field of robust design.

    Dr. Phadke is the author of the first engineering text book in English on Taguchi Method, Quality Engineering Using Robust Design, published by Prentice Hall, 1989. The book, which is considered the most authoritative book in English on Robust Design, has been translated in three languages, German, Chinese, and Korean.

    Dr. Phadke can be reached at Madhav@PhadkeAssociates.com, http://www.PhadkeAssociates.com.

    References

    1. Phadke, M.S. "Quality Engineering Using Robust Design" Prentice Hall, Englewood Cliff, NJ. November 1989.


    2. Taguchi, Genichi. "System of Experimental Design" Edited by Don Clausing. New York: UNIPUB/Krass International Publications, Volume 1 & 2, 1987.


    3. Browmlie, Robert; James Prowse; Madhav S. Phadke. "Robust Testing of AT&T PMX/StarMAIL using OATS" AT&T Technical Journal, Volume 71, No. 3 May/June 1992, pp 41-47.


    4. Phadke, M.S. "Planning Efficient Software Tests" CrossTalk Journal of Defense Software Engineering, October 1997, pp 11-15.

     
    Rate This Article:  Current Rating: 3.83
      Poor    Excellent     
              1    2    3     4    5
    Copyright � 2000-2010 iSixSigma – All Rights Reserved
    Reproduction Without Permission Is Strictly Prohibited – Copyright Requests


    Publish an Article: Do you have a Six Sigma tip, learning or case study?
    Share it with the largest community of Six Sigma professionals, and be recognized by your peers.
    It's a great way to promote your expertise and/or build your resume. Read more about submitting an article.




    "The Bottom Line" Links

    BEST SELLING PRODUCTS (iSixSigma Publications)
    1. Six Sigma Black Belt (DMAIC) Training Slides - 2009 Version!
      The 2009 Six Sigma Black Belt course includes over 40 more slides than the 2008 version. Contents include: 1,220 PowerPo...
    2. Certified Lean Six Sigma Green Belt Assessment Exam
      This assessment exam is useful for students interested in assessing their knowledge of Lean Six Sigma on the Green Belt ...
    3. Certified Lean Six Sigma Black Belt Assessment Exam
      Interested in assessing your knowledge of Lean Six Sigma? Preparing for certifications? Testing your students and traine...
    4. Six Sigma Green Belt Training Slides - 2009 Version
      The 2009 Six Sigma Green Belt course is comprised of: 1122 slides (over 70 more slides than the 2008 version)Instructor...
    5. Certified Lean Six Sigma Black Belt E-book
      In 670 pages learn everything within the Lean Six Sigma DMAIC body of knowledge to successfully achieve Black Belt certi...
    6. Six Sigma DMAIC Training Slides - 2009 Version
      The complete 2009 Lean Six Sigma DMAIC course prepares participants to perform the role of a LSS Black Belt; covering wh...
    7. Kaizen Workshop E-book
      This 150+ page ebook teaches key tools and techniques of Kaizen, as well as real application to enhance learning. Kaizen...
     
    Six Sigma AdLinks
    AdLinks Information


    Google AdWords
     
    Home | Discussion Forum | Event Calendar | Job Shop
    Link To iSixSigma | Rate This Page | Report A Problem | Free Content For Your Site | Submit Article For Publishing
     Terms of Service. �2000-2010 iSixSigma. All rights reserved. v3.0lb, 0.2
    About iSixSigmaContact UsPrivacy PolicySite Map