In screening experiments (model Cause and Effect 04) that are performed there are three possible results in the analysis.

  1. We run the analysis and find factors with P values below our Alpha Risk level (usually .05 or 5%). This is what we explain in the lesson and is the usual result of a screening experiment.
  2. We run the analysis and find no factors with a P value below our Alpha Risk Level. This usually happens when one of two things has occurred.
  3. First, but very rare, we have picked the wrong factors that affect the result we were looking at.
  4. Second, which is the most likely (and the one students do on the assignment) the level that are picked for each of the factors are too close to each other. Here we have to learn to pick level that are wide apart as we are running just a few runs and will have minimal data to analyze.
  5. Last is the tricky one where we run the analysis and find factors with P values that are below our Alpha Risk Level but as we remove the ones that are above that level others fall out until we only have one factor left. In this case as a Black Belt (If you do not know what I am about to tell you) you should contact a Master Black Belt or Statistician. Remember that as a black belt and you get stump, always look to an expert that can help you.

So here are a few things we need to remember.

  1. This is a screening experiment. Because it is a screening experiment we have a lot of factors and few runs to determine what is worth looking closer at. It usually is done to reduce the cost and time to run a full factorial experiment. This implies we WILL be running another experiment on the factors (and interactions) that we find significant here.
  2. A P value above .05 only means that we do not have enough data to show that these factors ARE significant. Usually we plan our design to insure we have enough data BUT here we have reduce the data amount to try and isolate just the key factors.

In your data we see what happens when the factor changes are small enough that as you eliminate factors well above a p value of .05 you find that other significant factors start to fall out until all you have is one factor. This is telling us that we do not have enough data to show any are significant per the P value alone.

We need to look at three other statistics that are found in the ANOVA analysis to help guide us to what we need and NOT have to rerun the screen with more runs. These are R-squared, R-squared (Pred). and the Model P Value. The R-aquare values look at how well the model explains the variation. These we want as high as possible. The Model P value which tell if the model is significant or not. This value you want as low as possible.

Remember this is a screening experiment so leave all the interactions out of the analysis. Remember in a screen many time the interactions are confounded with the main effects or other interactions. We will look at them in the full.

Below you will see the results that one of my students had and I discuss these three statistics (with the factor p values) below.

Factorial Regression: FlightTime versus W1, W2, L1, L2, L3, ClipSize, ...

Analysis of Variance

Source DF Adj SS AdjMS F-Value P-Value

Model 7 1.29191 0.18456 3.90 0.038

Linear 7 1.29191 0.18456 3.90 0.038

W1 1 0.02364 0.02364 0.50 0.500

W2 1 0.21506 0.21506 4.54 0.066

L1 1 0.34076 0.34076 7.19 0.028

L2 1 0.30388 0.30388 6.42 0.035

L3 1 0.02681 0.02681 0.57 0.473

ClipSize 1 0.10481 0.10481 2.21 0.175

PaperWeight 1 0.27694 0.27694 5.85 0.042

Error 8 0.37893 0.04737

Total 15 1.67084

Model Summary

S R-sq R-sq(adj) R-sq(pred)

0.217636 77.32% 57.48% 9.28%

Factors with p values less than or close to .05

Source DF Adj SS AdjMS F-Value P-Value

W2 1 0.21506 0.21506 4.54 0.066

L1 1 0.34076 0.34076 7.19 0.028

L2 1 0.30388 0.30388 6.42 0.035
PaperWeight 1 0.27694 0.27694 5.85 0.042

R-Square values

Model Summary

S R-sq R-sq(adj) R-sq(pred)

0.217636 77.32% 57.48% 9.28%

Model P Value

Source DF Adj SS AdjMS F-Value P-Value

Model 7 1.29191 0.18456 3.90 0.038

This will be our baseline!

We rerun the analysis with just these four factors.

Factorial Regression: FlightTime versus W2, L1, L2, PaperWeight

Analysis of Variance

Source DF Adj SS Adj MS F-Value P-Value

Model 4 1.1366 0.28416 5.85 0.009

Linear 4 1.1366 0.28416 5.85 0.009

W2 1 0.2151 0.21506 4.43 0.059

L1 1 0.3408 0.34076 7.02 0.023

L2 1 0.3039 0.30388 6.26 0.029

PaperWeight 1 0.2769 0.27694 5.70 0.036

Error 11 0.5342 0.04856

Total 15 1.6708

Model Summary

S R-sq R-sq(adj) R-sq(pred)

0.220370 68.03% 56.40% 32.36%

Factors with p values less than or close to .05

Source DF Adj SS Adj MS F-Value P-Value

L1 1 0.3408 0.34076 7.02 0.023

L2 1 0.3039 0.30388 6.26 0.029

PaperWeight 1 0.2769 0.27694 5.70 0.036

R-Square values

Model Summary

S R-sq R-sq(adj) R-sq(pred)

0.220370 68.03% 56.40% 32.36%

Model P Value

Source DF Adj SS Adj MS F-Value P-Value

Model 4 1.1366 0.28416 5.85 0.009

Here you can see Factor P values show we would drop W2 for next analysis. But BOTH R-sq and Model P value have improved.

Factorial Regression: FlightTime versus L1, L2, PaperWeight

Analysis of Variance

Source DF Adj SS Adj MS F-Value P-Value

Model 3 0.9216 0.30719 4.92 0.019

Linear 3 0.9216 0.30719 4.92 0.019

L1 1 0.3408 0.34076 5.46 0.038

L2 1 0.3039 0.30388 4.87 0.048

PaperWeight 1 0.2769 0.27694 4.44 0.057

Error 12 0.7493 0.06244

Total 15 1.6708

Model Summary

S R-sq R-sq(adj) R-sq(pred)

0.249876 55.16% 43.95% 20.28%

Factors with p values less than or close to .05

Source DF Adj SS Adj MS F-Value P-Value

L1 1 0.3408 0.34076 5.46 0.038

L2 1 0.3039 0.30388 4.87 0.048

R-Square values

Model Summary

S R-sq R-sq(adj) R-sq(pred)

0.249876 55.16% 43.95% 20.28%

Model P Value

Source DF Adj SS Adj MS F-Value P-Value

Model 3 0.9216 0.30719 4.92 0.019

Here you see a shift in the other three statistics to a worse condition. Both R-sq values dropped showing that the model explains less of the variation than with W2 included. The P value for the Model is now increased from .009 to .019.

This tell you that the better selection for moving on is the last analysis with the four factors of L1, L2, Paperweight, and W2.

If you continue to look at each analysis as you drop factor you will see these three factor get worse and worse.

Mr. Pyzdek pointed out to me that many times a company would only do the screening to save on cost and/or time, and that is true (but not recommended). If that were the case then looking at a cube plot of the data would show you the best setting of the four factors. This would be the best guess (give the amount of data you had).

Here you can see that the analysis shows that best flight time is 2.80406 which is found in two places in this chart. That is telling us that W2 is not in the best estimate and that the best settings for max flight time is L1=8.9; L2=6; Paper Weight= Light.