Design of Experiments (DOE)

Definition of Design of Experiments (DOE):

Design of experiments (DOE) can be defined as a set of statistical tools that deal with the planning, executing, analyzing, and interpretation of controlled tests to determine which factors will impact and drive the outcomes of your process. 

This article will explore two of the common approaches to DOE as well as the benefits of using DOE and offer some best practices for a successful experiment. 

Overview: What is DOE? 

Two of the most common approaches to DOE are a full factorial DOE and a fractional factorial DOE. Let’s start with a discussion of what a full factorial DOE is all about.

The purpose of the full factorial DOE is to determine at what settings of your process inputs will you optimize the values of your process outcomes. As an example, if your output is the fill level of a bottle of carbonated drink, and your primary process variables are machine speed, fill speed, and carbonation level, then what combination of those factors will give you the desired consistent fill level of the bottle?

With three variables, machine speed, fill speed, and carbonation level, how many different unique combinations would you have to test to explore all the possibilities? Which combination of machine speed, fill speed, and carbonation level will give you the most consistent fill? The experimentation using all possible factor combinations is called a full factorial design. These combinations are called Runs.  

We can calculate the total number of runs using the formula # Runs=2^k, where k is the number of variables and 2 is the number of levels, such as (High/Low) or (100 ml per minute/200 ml per minute). 

But, what if you aren’t able to run the entire set of combinations of a full factorial? What if you have monetary or time constraints, or too many variables? This is when you might choose to run a fractional factorial, also referred to as a screening DOE, which uses only a fraction of the total runs. That fraction can be one-half, one-quarter, one-eighth, and so forth depending on the number of factors or variables. 

While there is a formula to calculate the number of runs, suffice it to say you can just calculate your full factorial runs and divide by the fraction that you and your Black Belt or Master Black Belt determine is best for your experiment.

3 benefits of DOE 

Doing a designed experiment as opposed to using a trial-and-error approach has a number of benefits. 

1. Identify the main effects of your factors

A main effect is the impact of a specific variable on your output. In other words, how much does machine speed alone impact your output? Or fill speed?

2. Identifying interactions

Interactions occur if the impact of one factor on your response is dependent upon the setting of another factor. For example if you ran at a fill speed of 100 ml per minute, what machine speed should you run at to optimize your fill level? Likewise, what machine speed should you run at if your fill speed was 200 ml per minute? 

A full factorial design provides information about all the possible interactions. Fractional factorial designs will provide limited interaction information because you did not test all the possible combinations. 

3. You can determine optimal settings for your variables 

After analyzing all of your main effects and interactions, you will be able to determine what your settings should be for your factors or variables. 

Why is DOE important to understand? 

When discussing the proper settings for your process variables, people often rely on what they have always done, on what Old Joe taught them years ago, or even where they feel the best setting should be. DOE provides a more scientific approach. 

Distinguish between significant and insignificant factors

Your process variables have different impacts on your output. Some are statistically important, and some are just noise. You need to understand which is which.

The existence of interactions

Unfortunately, most process outcomes are a function of interactions rather than pure main effects. You will need to understand the implications of that when operating your processes. 

Statistical significance 

DOE statistical outputs will indicate whether your main effects and interactions are statistically significant or not. You will need to understand that so you focus on those variables that have real impact on your process.

An industry example of DOE 

A unique application of DOE in marketing is called conjoint analysis. A web-based company wanted to design its website to increase traffic and online sales. Doing a traditional DOE was not practical, so leadership decided to use conjoint analysis to help them design the optimal web page.

The marketing and IT team members identified the following variables that seemed to impact their users’ online experience: 

  • loading speed of the site
  • font of the text
  • color scheme
  • primary graphic motion
  • primary graphic size 
  • menu orientation

They enlisted the company’s Master Black Belt to help them do the experiment using a two-level approach.

In a conjoint analysis DOE, you would create mockups of the various combinations of variables. A sample of customers were selected and shown the different mockups. After viewing them, the customer then ranked the different mockups from most preferred to least preferred. The ranking provided the numerical value of that combination. To keep matters simple, they went with a quarter fraction design, or 16 different mockups. Otherwise, you’re asking customers to try and differentiate their preference and rank way too many options.

Once they gathered all the data and analyzed it, they concluded that menu orientation and loading speed were the most significant factors. This allowed them to do what they wanted with font, primary graphic, and color scheme since they were not significant.

3 best practices when thinking about DOE 

Experiments take planning and proper execution, otherwise the results may be meaningless. Here are a few hints for making sure you properly run your DOE. 

1. Carefully identify your variables

Use existing data and data analysis to try and identify the most logical factors for your experiment. Regression analysis is often a good source of selecting potentially significant factors. 

2. Prevent contamination of your experiment

During your experiment, you will have your experimental factors as well as other environmental factors around you that you aren’t interested in testing. You will need to control those to reduce the noise and contamination that might occur (which would reduce the value of your DOE).

3. Use screening experiments to reduce cost and time

Unless you’ve done some prior screening of your potential factors, you might want to start your DOE with a screening or fractional factorial design. This will provide information as to potentially significant factors without consuming your whole budget. Once you’ve identified the best potential factors, you can do a full factorial with the reduced number of factors.

Frequently Asked Questions (FAQ) about DOE

What does “main effects” refer to?

The main effects of a DOE are the individual factors that have a statistically significant effect on your output. In the common two-level DOE, an effect is measured by subtracting the response value for running at the high level from the response value for running at the low level. The difference is the effect of that factor.

How many runs do I need for a full factorial DOE?

The formula for calculating the number of runs of a full factorial DOE is # Runs=X^K where X is the number of levels or settings, and K is the number of variables for factors.

Are interactions in DOE important? 

Yes. Sometimes your DOE factors do not behave the same way when you look at them together as opposed to looking at the factor impact individually. In the world of pharmaceuticals, you hear a lot about drug interactions. You can safely take an antihistamine for your allergies. You can also safely take an antibiotic for your infection. But taking them both at the same time can cause an interaction effect that can be deadly.

In summary, DOE is the way to go

A design of experiments (DOE) is a set of statistical tools for planning, executing, analyzing, and interpreting experimental tests to determine the impact of your process factors on the outcomes of your process. 

The technique allows you to simultaneously control and manipulate multiple input factors to determine their effect on a desired output or response. By simultaneously testing multiple inputs, your DOE can identify significant interactions you might miss if you were only testing one factor at a time. 

You can either use full factorial designs with all possible factor combinations, or fractional factorial designs using smaller subsets of the combinations.

« Back to Dictionary Index