Confidence Level, Variation, How many times to run.?
Six Sigma – iSixSigma › Forums › Old Forums › General › Confidence Level, Variation, How many times to run.?
 This topic has 5 replies, 4 voices, and was last updated 13 years, 5 months ago by DrSeuss.

AuthorPosts

December 8, 2008 at 8:14 pm #51476
Hi all,
I am trying to build a business model.
I tried to actually simulate it, but I do not know how many times I have to run simulation or order to get 95% of confidence level.
I have about 25 variables, pretty much all in the same range of (plus/minus) 10%.
Could some one explain how many times I have to run simulation based on amount of variables?
Thank you,0December 9, 2008 at 4:18 pm #178480Any ideas?!!!
0December 9, 2008 at 4:22 pm #178482This might be the wrong answer but with multivariate analysis I usually assume more than 5 runs per variable.Hopefully, Robert will give you a statistically correct answer …Cheers,
Andy0December 11, 2008 at 1:44 pm #178595
Robert ButlerParticipant@rbutler Include @rbutler in your post and this person will
be notified via email.1. You are trying to build a business model
2. You have 25 variables
3. Each variable is “pretty much” in the same range of plus/minus 10 percent
4. You want to run a simulation or order to 95% confidence level.
I think you will need to provide more information before anyone can offer much of anything.
If you haven’t yet built the model then you don’t have anything to test. What kind of tools are you using to try to build a model? If we assume you are trying to build a regression model what have you done to make sure the terms can actually be included in the model (you would have to have quite a system of data gathering to guarantee the independence of 25 model variables)?
If it is a regression model and if the terms were independent and if they are all significant what does your regression analysis tell you about your model? Do you have a situation where MSE and R2 level off before all 25 terms are included? What does the residual pattern look like for the full (25 term model) and the reduced (the one where you quit adding terms when the changes in MSE and R2 became incremental)?
Again assuming a regression model and assuming you can actually have a model with 25 indepedent terms and assuming the regression analysis indicated a full model was needed then you will have a RMSE associated with the final model. Roughly 2 times the RMSE will give you the 95% CI around any prediction you may make with the model. As for simulations – you can run them as many times as you wish but they aren’t going to impact the RMSE and hence the prediction error of the model you built.
The label “business model” would suggest the terms in the final model are not something you personally can adjust to impact the outcome. If this is the case then insted of running simulations what you will need to do is wait for the equivalent of a confirming run. This means gathering additional data as it becomes available and then testing your model by seeing how well the model predicts the new data. You will want to be particularly alert to new data that has values for your model parameters that are at the extremes of your variable ranges.
If the last paragraph is an adequate summary of what you are trying to do and if you have actually done all of the things listed in the preliminary verbiage then what you are really asking is – How much new data do I need to gather before I can be confident that my model is doing an adequate job of prediction? As far as I know there isn’t any cut and dried answer to this kind of a question.
There are things, such as data partitioning and model regeneration that can be run to test term stability but the final 95% prediction CI of any model will be as described above.0December 15, 2008 at 11:43 am #178722Unless your simulation has noise built into it, you need to run it only once per experimental setup. With no noise, a (computer) simulation will return the same result every run and thus, you will not get the confidence interval you are looking for.
0December 23, 2008 at 4:20 am #179027
DrSeussParticipant@DrSeuss Include @DrSeuss in your post and this person will
be notified via email.Allen,
Is sound like you are asking the standard question; “how many data
points do I need to collect?” You need to use a sample size
calculator, like the one on this site. See iSix Sigma tools or
calculator links off the home page. What you are looking for is
something that will require to choose a confidence level for your
expected outcome for each variable. You will also need an
estimate of the standard deviation, and an acceptable margin of
error (how close is good enough). Usually with these inputs the
tool will tell you how many data points to use for your sample.
There are more complex versions that deal with alpha & beta risks,
for for starters, the simple derivations for the confidence interval
formulas should be sufficient. If the tool does not exist, the go to
the basics. Take the formula for the confidence interval for the
metric you want to estimate and then solve for n (not CI) and you
got it. Good luck, ……DrSeuss0 
AuthorPosts
The forum ‘General’ is closed to new topics and replies.