iSixSigma

Quantifying the product quality based on UAT defects

Six Sigma – iSixSigma Forums Old Forums Software/IT Quantifying the product quality based on UAT defects

Viewing 9 posts - 1 through 9 (of 9 total)
  • Author
    Posts
  • #25963

    Sreenivasa
    Member

    Hi,
    Our group wants to quantify the software product quality based on the number of defects identified during user acceptance testing by using six sigma methodology. Any thoughts on where to start..
    Thanks
    Sreenivas
     

    0
    #63040

    Michael Schlueter
    Participant

    Hi Sreenivasa,
    Interesting idea. Let’s have a look at this approach.
    I suppose the software is more complex than simple. There is certainly something like a “nominal user interaction”: i.e. a main flow or similar interactions many users will do on average. This should reveal defects most of the users will encounter. Or vice versa: users shouldn’t experience any trouble along the main path.
    What’s about the unidentified defects? What’s about variability in user action? What’s about variability in the software products environment? What’s about rare circumstances (which become likely after 1E6 or 1E9 cycles of operations)?
    I’ve seen test approaches, both in HW and SW, which try to test the effect of this kind of variability by using orthogonal arrays. These arrays help you changing everything at the same time, provoking errors from weak parts more easily with reasonable effort.
    Using OA’s is an experimental approach. There is also a mental, analytical approach, called “sabotage technique” or AFD (Anticipatory Failure Determination) nowadays. The idea is as follows:

    you take the software, its computer system, its user, its data and so on;
    somebody has the task to destroy the program, its functionality, its result etc. intentionally
    but he/she is only allowed to use what is already there, ready available.
    Nothing else.
    Create as many harm as you ever can with this product. Don’t be shy. Try creating a real catastrophe with your product.
    This leads to intentionally abuse of your product; it identifies those conditions, those circumstances and those sequences of events which will inevitably create disaster – the larger, the better, because those complaints will hit your company anyway, sooner or later.
    The rational behind this sabotage approach comes from investigating real-life accidents. Those were no mysteries, but just unhappy sequences of unhappy circumstances, until at some point there was no return. Just like a snow flake falls down on an (instable) layer of snow on a sunny day at the southern mountain side were the slope is steep – starting some snow rolling – – – growing into a massive avalanche …
    Similar disasters are well known from SW as well. Nothing is wrong, each step in itself is harmless, but the chain of events becomes really bad. Some famous SW-defects at the usage level:

    The navigation module for a new fighter was taken from a missile and adapted; whenever the pilot crossed the equator, the machine rolled 180 degrees upside down …
    The Ariane rocket reused a navigation module, which was done in 16 bit, in a new 32 bit environment. On launch, the unit created an overflow, navigating the rocket downwards; the rocket had to be emergency blasted …
    A bank accounting program kept charging invoices for an account, which had a balance of +0.00 USD …
    I myself was able to prevent car accidents by this approach: I derived test cases for a software module, which accesses registers of an unimportant unit in an embedded system for car applications; it was able to allocate memory, where it shouldn’t. This could have led to unwanted behaviour of air bags, motor control, brake control etc.
    Nobody from the design team looked at it this way before. But by asking myself “how can I injure a car passenger intentionally?” finally led me to look for ways to irritate the embedded system; and memory allocation was both in place and powerful enough to perform the desired disaster. The irritated piece of silicon together with the cars motor (or other components) form a powerful people-hurting machine.
    When we knew it, we immediately corrected this problem. It never occured.
    What is common to these examples?
    All comprise rare, but possible circumstances: equator-crossing, launch, zero-balance, memory allocation are infrequent, yet normal and allowed events. SW in itself can neither create harm nor good – it is the interaction with a relevant real-life system, which turns a bug into a serious problem (like irritating a pilot, like directing a rocket to where it doesn’t belong, like annihilating huge amounts of money, like directing a cars passenger where he/she shouldn’t be etc.)
    In summary: I recommend seeking disaster from your product intentionally before you start counting defects.
    And: celebrate each disaster mechanism you’ve identified; each one found keeps a lot of trouble from your company.
     
    Best regards, Michael Schlueter

    0
    #63041

    W McCaster
    Member

    Hello,
    I’m a new Black Belt at GE Aircraft Engines.  I’m working on a Software Quality Improvement project and am currently in the Define / Measure phase.  It seems we are in the quandry.  Except that I hesitate measuring with UAT because it can be very subjective and dependent on the upfront work (e.g., good requirements gathering — requirements that can be measured, validated, and traceable). 
    So, I’m looking more into measuring quality by the # of post-production defects.  This seems to be less subjective than UAT.  I have some ideas on how to improve it, but I’m making sure I have the data before I leap into the solution :o).
    Thanks

    0
    #63042

    Sreenivasa
    Member

    Hi McCaster,
    Thanks for your reply. I think we are in the same boat. I too agree that it may be subjective to rate the product just based on UAT defects.  But when you say post-production, what will be the right duration to qualify the product.
    In our case we are trying to rate the quality of the software products delivered by different vendors based on UAT errors. If they do good upfront work, it is natural that the error rate in UAT will be less and the product deserves good quality rating.   We also want to take the factors like complexity, technology, size, number of users etc. into account.
    Would appreciate your early thoughts on this.
    Thanks & Regards
    Sreenivas

    0
    #63043

    W McCaster
    Member

    The point in the process of where to do the measure depends on what you’re trying to get after. 
    If you’re trying to rate, let’s say, the FTY of the deliverable from the vendor to you (project leader I’m assuming), you’re getting after the quality of what they are delivery after the unit testing and integration has been done by the vendor.  You’ll then go through interations of returning the code and getting fixes, etc. if needed. That should give you an idea of the quality of what is being delivered by the vendors before the customer / end user sees it.
    The next thing is that once you’re satisfied with the product and present it to your customer for UAT, the user is rating the quality of the functional requirements against what they received.  This is where you can rate how well the product met the requirements and CTQs of the customer. 
    These steps are all pre-production.  The post-production defects that I’m looking at is how well the product operates in production and has gone to the masses for use.  With the UAT, you only get a sample size of users (generally on a large project). 
    Also,  with each round of testing and error/defect rating, make sure you have a well defined scale in place for the severity of the error (pre-productio) or defect (post-production).   Enhancements are different from defects just as cosmetic changes/corrections are different than fatal errors.
    Hope that helps.

    0
    #63044

    Sreenivas
    Member

    That is fine McCaster. Lets say the product is in UAT and the user has to rate it by using six sigma methodology.  Can I get some ideas how to calculate sigma value based on the defects in UAT. This must be a common process   for all different software products with different complexities.
    Thanks & Regards
     Sreenivasa

    0
    #63049

    Sreenivas
    Member

    Thanks Michael for your reply. Your thoughts on this subject are well appreciated.  But, I am specifically looking for a six sigma methodology with a mix of statistical tool to assess the products of our vendors based on the UAT defects.   We also have to take the factors like complexity, technology, size, number of users etc. into account before making an assessment. Please do share any thoughts you have on this
    Best regards
    Sreenivas

    0
    #63050

    Michael Schlueter
    Participant

    Sreenivasa,
    Thanks for appreciating my reply. Please shed some more light on your situation; this will help me to give you a more focussed reply. – I assumed you were developing the SW 
    Let’s go a step ahed. Assume, you have the kind of evalution you are looking for: what will you do next? What will it be good for? What kind of result can you obtain? In other words: what are you heading for?
    Best regards, Michael Schlueter

    0
    #63052

    Sunil Anand
    Member

    Hi Sreenivas,
    There may be a number of ways to measure Sigma value of your process. A good starting point is defining your process itself.
    For eg, if you are measuring the sigma of sw development activity only, its sigma evaluation will be different from the evaluation for sw development and implementation.
    Once you know the exact process whose capability you want to evaluate, we can go to measurements. While you can develop a complicated model at a later stage, I would recommend that you start simple.
    Let us say, your process is “software development based on requirements”. At a high level, you are doing the following:
    1. Provide requirements
    2. Do systems analysis
    3. Development and unit test
    4. UAT
    5. Migrate to production
    Now, for this process, if you need to calculate sigma, you ask yourself “What are the outputs? What are the CTQs of customers for this process?”  Now, you may have 2 options:
    a. You may consider yourself as the customer and then the output is Unit-tested code
    OR b. You may consider business users are the customers, with output as “application in production”
    For (a) –> Ask yourself what are the opportunities for defects in unit-tested code? Answer –> Defects found in unit-tested code delivered to you.
    How do we measure that?  Answer –> Go by your functional requirements. List all the Test Cases that will be run in the UAT. Let us say O.  Ask the no. of  Test Cases where the delivered software failed. Let us say D.
    Now, you have One Unit (ie N =1) as you are testing only one delivered version of the software given to you. You had O opportunities (test cases) and D failed.
    You can calculate DPMO for this one unit. If you get multiple deliveries, you can find out all Ds, Os, and add them up. Finally get your DPMO.
    (b) Let us leave that for another day as I understand you wanted to see effectiveness of UAT. 
    Once you master the simple model, you may consider different ways of making this comprehensive…. you may like to use the “Opportunities” differently. But in my opinion, keep it simple! Let the no. of test cases determine complexity for you directly.
    Hope this helps!
    Rgds
    Sunil Anand
     
     
     
     
     
     

    0
Viewing 9 posts - 1 through 9 (of 9 total)

The forum ‘Software/IT’ is closed to new topics and replies.