iSixSigma

Design Of Experiment For Software Testing

Six Sigma – iSixSigma Forums Old Forums General Design Of Experiment For Software Testing

Viewing 5 posts - 1 through 5 (of 5 total)
  • Author
    Posts
  • #31147

    David L. Hallowell
    Participant

    Dr Phadke has done a nice job illustrating the benefits of fractional orthogonal arrays in software test.  I would underline two of his points that resonate with my experience.
    1. The DOE discipline of thinking about the factors and their levels that would describe the test domain – in and of itself –  can set the stage for better coverage.  While we can never test all combinations, a fractional factorial array suggests a starting point for balanced partial coverage. If we just begin writing test cases we can focus too much on certain areas and leave others out.
    2.  Fractional arrays can find region faults, while isolated faults can go completely undetected.  As many software faults are discrete events sensitive to the particular combination of factor levels, and even the sequence of state transitions leading up to the test condition this makes fractional factorials weaker in this use than they would be in their more common use in the hardware world for modeling.  There, the continuity of responses (like, for exanple, the shear strength of a bonded interface) lets us interpolate meaningful estimates of what’s going on at the design points not covered in the fractional design.  As we see in discrete event software test, the one combination we may leave out could be an isolated fault we’ll miss.
    A reasonable build on Dr. Phadke’s orthogonal arrays for test is the use of D-optimal experimental designs. These allow some further tailoring to specific test circumstances.  D-optimal designs are the results of  computer search for the ‘most orthogonal’ combination of factors and levels available given a set of coverage objectives and constraints defined by the user.  One could, for example, specify that they are interested in learning about all single and double mode faults, and specific other multi-mode faults.  They can further specify ‘inclusions’ (factor combinations to be sure to include in the test plan)  ‘exclusions’ (combinations known to be of little test value), and the number of test runs that are feasible.  The search will then return with the most orthogonal design for covering that test space.  Notice that this allows the tailoring of the test regions most thoroughly covered within the broader test space.
    Another related approach I have seen used, is to start with D-optimal/ or fractional test plan — run that and evaluate for faults found.  Then iterate with more thorough fractional designs in regions that have yielded faults. By analogy, this is like casting a broad net over the whole test space, then drilling in with progressively finer nets in areas that show faults.
    It would be interesting to see what others’ experience has been in this area.
    Dave Hallowell
     
     
     
     
     
     
     

    0
    #81922

    BB
    Participant

    I’ve done testing for a major airline’s website. The crux was that we needed to evalute ticket processing capabilities to ensure that they all completed and had correct answers. A doe would have been useful at that time, but it was back in 99 when the internet boom was hot and heavy. We probably could have cut down the 2 weeks of unit testing (10 testers) to something more manageable.
    Where was this article back then! :)

    0
    #81945

    Manee
    Participant

    In my company, Dr. Phadke gave a course. so we knew it in 1995.  But it was tedious to use orthogonal arrays.  Our manager got rdExpert tool. to make it easy  and we love it.
    Manee
     

    0
    #81946

    BB
    Participant

    What kind of applications are you using rdexpert for? How is it making your software development or testing easier? That’s the kind of information I need to take to my manager :).

    0
    #81987

    Manee
    Participant

    I personaly use it for system testing of communication products.  I like it because it gives full test script.  Some colleague use it rdexpert for integration testing and some in development area also use it.
    My manager likes it because the test plan review meetings are short and productive.
    You can contact Phadke directly for other info for your manager
    Manee
     

    0
Viewing 5 posts - 1 through 5 (of 5 total)

The forum ‘General’ is closed to new topics and replies.