Six Sigma Software Metrics, Part 2

	Bookmark This Page Email This Page Format for Printing Cite This Article Submit an Article Read More Articles
	Related Tools & Articles Six Sigma for Software - A Role Based System Six Sigma for Software… More Than a New Tool Software Quality Function Deployment
	Discussion Forum "How does one quantify the defects in software for a six sigma initiative? Is it reasonable to predict the probability of residual defects based on defects detected per test cycle?" Six Sigma Metrics For Software

B

Download Products Gage R&R Excel Template Medical Parts Delivery Project Example A Look Ahead for Six Sigma Research Report

y David L. Hallowell

Part 1 in this series on software defect metrics discussed Goals 1 and 2, which focused on identifying and removing defects in the development process as close to the point of occurrence as possible (Table 1). This installment looks at predicting defect insertion and removal dynamics early in a project and measuring predicted versus actual defect find rates during each development stage. The next and final installment in the series provides a foundation for understanding the most elusive metrics, defect density measures such as Defects per Million Opportunities (DPMO).

Goal 3: Predict Defect Find and Fix Rates During Development and After Release
Classifying and counting defects helps focus problem solving and root cause analysis efforts, however can quantify defect history in a somewhat static way. Historical defect data can assist organizations early in the project planning process to predict defect insertion and find defect counts for each development stage or iteration step. Beyond predictive tallies, it is also important to recognize the rate at which defects accumulate. Predicting both dynamics can add sensitivity to the defect diagnosis on a new project, and support project staffing plans.

Defect insertion and removal dynamics over the course of a development project are summarized in Figure 1. The left curve illustrates that defect insertion (in the form of ambiguities, misunderstandings, omissions, etc.) begins when the project effort begins, during the earliest stages of the "fuzzy front end." Defects are often tied to the intensity of the effort (e.g. number of people involved, lines of communication, decisions being made, etc.) and the insertion rate usually tracks with that contour.

The second curve illustrates that finding and fixing defects most often occurs substantially after the work-product effort. For an organization depending on the final test process to find most defects, this lag can have a negative impact. Activities like peer reviews and inspections find defects closer to their insertion point, shifting the find curve to the left, where the fix times and costs are lower.

Figure 1: Defect Insertion and Find-and-Fix Dynamics

The asymmetry in the curves suggests that it takes more time to get done than it did to get engaged. The Rayleigh Model¹ is a proven method to predict and track this time-dynamic. The application of the Model to the estimation of project effort is covered in a related article (Six Sigma Meets Project Management, Part 2). Here its use is extended to predict defect find and fix rates.

Starting Simple - Using Project History
Collecting data on the effort, duration, work-product size, and defect counts for a number of projects provides an opportunity to analyze trends, ultimately resulting in more accurate predictions for new projects. Project teams lacking trend data can still get started by using industry benchmarks as a guide. Generally speaking, if a team doesn't know "where they are" they probably aren't doing any better than representative industry averages. Applying those averages as deemed appropriate can provide a reasonable place to begin the estimating process. As each project progresses, a continual review of predicted versus actual defect counts will allow the team to refine the estimates and improve the model for the next project.

Figure 2 illustrates a case where the project team estimated the size of the new project at about 1,250 Function Points². Two additional inputs are included, one that assess the organization's Productivity Index³ and one that anticipates schedule compression. These inputs will drive project deadlines. The model computes estimates of duration, effort, and defects as total, pre-release, and released figures.

Figure 2: Estimating Model Inputs and Outputs

Applying the most likely scenario, the second line in the figure with a Total Defect estimate of 946, to a scorecard facilitates the next level of detailed predictions (Figure 3), Goals 1 and 2 provided the ability to understand and quantify Phase Containment Effectiveness (PCE), Defect Containment Effectiveness (DCE) and insertion rates. Those numbers take on a predictive value in the scorecard, where they are used to distribute the total defect count across development or iteration steps.

Building on the measurements enabled through Goals 1 and 2, project teams can use their growing database of phase containment data to estimate the number of defects expected in each phase of a new project.

Figure 3: Predictive Defect Analysis Scorecard

The circle in Figure 3 highlights the number of defects expected during the requirements stage. As development progresses the predicted versus actual defect tallies are compared. Cases where the actual is significantly higher than predicted may provide early warning of a problem. Actual tallies that are much less than predicted should prompt an investigation to ensure that leaks in the defect detection methods are not present before determining that the insertion rate was lowered.

Defect Time Contouring With The Rayleigh Model
The Rayleigh distribution offers a useful fit to real-world experience and data. The model requires two inputs, the overall total quantity contoured over time (K) and the time period necessary to estimate the quantity. The quantities most often modeled for software projects are effort and defects. The total (K) for each of these is easily estimated early in the project and the Rayleigh Model can readily provide a view of their distribution over time.

Numbers for total effort and total defects are derived from the estimating model (Figure 2). These totals, together with an anticipated time to reach peak estimate are the only quantities needed to compute the Rayleigh curve (Figure 4). For the defect plot, an additional value associated with the estimated lag behind the start of the project effort is needed to account for defect find and fix work.

Figure 4: Rayleigh Model Density Function (PDF)

Figure 5: Rayleigh Model for Effort and Defect 'Find and Fix' Activity

The pair of curves in Figure 5 illustrates the time-dynamic connection between project Effort and Defects. The Model provides support for fact-based discussions about the impact of changes such as accelerating the project delivery date. For a project under pressure to deliver within 9 months, the Model will clearly display that many defects will still remain. The Model facilitates analysis of the cost of delivering those defects versus the advantages of early delivery.

The Model's Cumulative Distribution Function (CDF) supports a more refined discussion of the impact associated with a change in delivery date. This figure describes the Total to Date effort expended or defects found at each interval. Figures 6, 7, and 8 show the Rayleigh CDF formula, chart, and values table respectively.

Figure 6: Rayleiigh Cumulative Distribution Function Scaled To Total Modeled Quantity K

Figure 7: Rayleigh CDF Chart For Cumulative Defects Found

Figure 8: Rayleigh CDF Table For Work-Product Effort And Defect Removal

Figures 7 and 8 provide quantitative, fact-based data to support a discussion on the delivery date. At 12 months the Rayleigh Model indicates the expectation that 96.9% of the defects are found. Moving delivery to 9 months could reduce the total containment effectiveness (TCE) to around 81.6%. An organization about to make that decision is well advised to weigh the benefits of early delivery against the costs associated with released defect repair and possibly, customer loyalty that are involved in that tradeoff.

Looking Ahead To Part 3
Defect counts and classifications by phase or activity, and over time provide a basic analysis platform that supports a number of Six Sigma goals as applied to Software. These basic measures fail to account for differences in the size or complexity of the work products. The last two Goals in our metrics maturity table call for the comparison of implementations within the company and across companies. In each case, defect density normalization is needed.

Goal 4: Compare Implementations Within the Company
Defects per unit size as defined within the company can support fair comparisons between projects and groups.

Goal 5: Benchmark Implementations Across Companies
Making comparisons across companies calls for a more universal approach to defect density normalization. This is the reason that Defects per Million Opportunities (DPMO) was developed for Six Sigma manufacturing environments. While the approach is not simple science, it is applicable to explore the fundamentals of the DPMO concept within the software development environment.

Read Six Sigma Software Metrics, Part 3 »

About The Author
David L. Hallowell is a Managing Partner of Six Sigma Advantage, Inc. and has 20+ years experience as an engineer, manager and Master Black Belt. As Digital’s representative to Motorola’s Six Sigma Research Institute he worked on the original courseware for Black Belts and the application of Six Sigma to software. Mr. Hallowell has supported Six Sigma deployments and trained many Black Belts and trainers worldwide.

With a special focus on Design for Six Sigma, he has led development teams to breakthrough improvements in the concept development and design of a number of commercial products. Mr. Hallowell has patents and publications in the area of microelectronics packaging and high speed interconnect. He has authored courses in Software DFSS, Design of Experiments, C++ , and Computational Intelligence Tools. He is a co-founder of Six Sigma Advantage Inc., where he co-authored the Black Belt, Green Belt, and Foundation curriculum. He can be reached via email at dhallowell@6siga.com.

Footnotes And References
1 The Rayleigh model is special case of the Weibull distributions. A good treatment of the general topic, with software application examples, can be found in Kan, Stephen, Metrics and Models in Software Quality Engineering. Addison-Wesley, 2003.
2 See www.ifpug.com, www.spr.com, and/or the work of Capers Jones for more information on Function Point sizing.
3 See Six Sigma Meets Project Management

Table 1: Software Organization Goals Versus Processes And Metrics  Six Sigma Goal  Required Processes  Enabled Processes  Metrics 1. Reduce Released Defects Unit/Integration/System Test Pre-release vs. Post-release Defect Tallies Total Containment Effectiveness (TCE) Focus defect fix/removal work Operational definitions for classifying Defects Problem-solving knowledge-base Basic causal analysis Defect stratification by type Poisson Modeling for defect clustering   (Optional) Define the "Unit" for work-products delivered Yield assessments for work-product units delivered Yield predictions for work-product units planned Total Defects per Unit (TDU) Rolled Throughput Yield (RTY) 2. Find and Fix Defects Closer to Their Origin Upstream Defect Detection Process (Inspections) Defect Insertion rates and Defect Find Rates for phases or other segments of work breakdown structure (WBS) Phase Containment Effectiveness (PCE) and Defect Containment Effectiveness (DCE) Gather data necessary for process monitoring and improvement Defect Sourcing Process Improved causal analysis Defects per Unit (DPU) for phases or other segments of WBS, contributions to TDU 3. Predict Defect Find and Fix Rates During Development and After Release Rayleigh or other time-dynamic modeling of defect insertion and repair Defect Estimating Model - calibrated to the history and current state in our process Given data on defects found during upstream development or WBS stages, predict defect find rates downstream. Predicted total defects for an upcoming project of specified size and complexity Best fit Rayleigh Model Predictive Rayleigh Model 4. Compare Implementations Within the Company Company's choice of appropriate normalizing factors(LOC, FP, etc) to convert defect counts into meaningful defect densities Defect Density comparing groups, sites, code-bases, etc. within the company Defects / Function Point Defects per KLOC 5. Benchmark Implementations Across Companies Define Opportunities for counting defects in a way that is consistent within the company and any companies being benchmarked Defect Density comparing performance with other companies (and, if applicable, other industries) DPMO Sigma Level Cpk z-score

"The Bottom Line" Links

BMG UNIVERSITY I I Lean Six Sigma Online I I • Reduce Travel Costs • Maximize Training Budget >> Get Certified Now

VILLANOVA UNIVERSITY   Earn Your Lean, Green or Black Belt Six Sigma Master Certificate Online BOOST YOUR SALARY! *$38k more than uncertified counterparts. Learn From Industry Leaders!   START NOW

SIGMAPRO MBB, Lean Sigma, & DFSS   when experience matters most...

JURAN Budget constraints halting your improvement efforts? They don't have to! Learn about our accelerated cost reduction methods.

LodeStar Institute | Public & Onsite Lean Six Sigma Also New Live WEB Training  WHY SETTLE FOR ONLINE? GET LIVE TRAINING AND PROJECT MENTORING FOR ONLINE PRICES

Finding that key person for your team is just a click away . . .     TheJobShop jobs.isixsigma.com

THE UNIVERSITY OF TEXAS AT AUSTIN 2 weeks + 1 project = Black Belt Certification

. Find us on LinkedIn Join the iSixSigma Network and Connect with Other Six Sigma Pros .

. iSixSigma Live! Summit & Awards Jan 13-16, 2009 • Miami, FL Save up to $500 • Click Here! Register by October 14 .

BEST SELLING PRODUCTS (iSixSigma Publications) Six Sigma DMAIC Training Slides The complete 2008 Lean Six Sigma DMAIC course prepares participants to perform the role of a LSS Black Belt; covering wh... Certified Lean Six Sigma Black Belt Assessment Exam Interested in assessing your knowledge of Lean Six Sigma? Preparing for certifications? Testing your students and traine... Six Sigma Black Belt (DMAIC) Training Slides - 2009 Version! The 2009 Six Sigma Black Belt course includes over 40 more slides than the 2008 version. Contents include: 1,220 PowerPo... Certified Lean Six Sigma Green Belt Assessment Exam This assessment exam is useful for students interested in assessing their knowledge of Lean Six Sigma on the Green Belt ... Process Management Training Slides The 2008 Process Management course is designed in two phases comprised of:352 Powerpoint slidesInstructor notesSlide exp... Certified Lean Six Sigma Black Belt E-book In 670 pages learn everything within the Lean Six Sigma DMAIC body of knowledge to successfully achieve Black Belt certi... Gage R&R Excel Template Gage Repeatability and Reproducibility (R&R) studies measure the amount of measurement variation that is attributabl... Sell Your Products