Fractional. Design of Experiments. Overview. Scenario

Size: px

Start display at page:

Download "Fractional. Design of Experiments. Overview. Scenario"

Lucinda Clark
5 years ago
Views:

1 Design of Experiments Overview We are going to learn about DOEs. Specifically, you ll learn what a DOE is, as well as, what a key concept known as Confounding is all about. Finally, you ll learn what the 4 steps to conducting a DOE are including how to plan, run, and analyze your own DOE. Scenario Full Factorial DOEs help us test factors across all possible combinations in order to determine which terms are statistically significant. Additionally, when we learned about the 2 Sample t- Test, we used the example of a company attempting to understand whether the words inside a clickable website button made any statistical difference as it relates to the number of clicks the button received. As it turns out, this same company has decided to take this same sort of website design testing to another level. Specifically, they want to conduct a Design of Experiment. After completing a Cause & Effect matrix, as well as, some excellent brainstorming, the team decided to test the following five factors; the overall page color, the color of the subscription button meaning the button people click Design of Experiments GembaAcademy.com 1

So if we wanted to do an additional replicate, we d be looking at 64 runs which would obviously take a lot of time and money to complete. So an alternative to the Full Factorial DOE is the 16 run DOE.

2 to sign up for the newsletter, the text used inside the button, the image shown on the webpage, and the customer testimonials used on the page. We want to make it clear that there are five factors in this experiment. With five factors, we re looking at 32 runs for a Full Factorial DOE and this doesn t include any replicates. So if we wanted to do an additional replicate, we d be looking at 64 runs which would obviously take a lot of time and money to complete. So an alternative to the Full Factorial DOE is the 16 run DOE. Since there are half of the number of runs, we refer to this design as a Half Fraction Design. We could also refer to this particular DOE as a 2 to the 5 minus 1 design which simply means we re testing 5 factors across 16 runs since 5 minus 1 is 4. Defining DOEs Defined, DOEs help us test many factors across varying combinations in order to identify factors worthy of deeper investigation. While DOEs can be extremely helpful, there are several things we must be vigilant of including something known as Confounding. As it turns out, when the effects of terms are indistinguishable from one another, the terms are said to be confounded. For example, if Factor A is found to be significant, but is also confounded with the BC interaction, we won t know which term is actually significant, Factor A or the Interaction of BC. In other words, it s almost like these confounded terms are identical twins who are very hard to differentiate. Confounded terms are also said to be aliased with one another. Design of Experiments GembaAcademy.com 2

Resolution Another important term related to confounding and aliasing is Resolution which measures how strongly factors are aliased with other factors and interactions.

3 Resolution Another important term related to confounding and aliasing is Resolution which measures how strongly factors are aliased with other factors and interactions. An easy way to understand resolution is by using your fingers. First, in resolution III designs, main effects are confounded with 2 way interactions. In this example, we see that factor A is confounded with the interaction of B x C. We can tell this since in this row, A is - 1. The interaction of B x C also results in - 1 since - 1 times 1 equals - 1. In other words, A is confounded with BC. Since we re almost always interested in differentiating between main effects and 2 way interactions, resolution III designs aren t typically recommended. Next, in resolution IV designs, main effects are confounded with 3 way interactions, and 2 way interactions are confounded with other 2 way interactions. In resolution V designs, main effects are confounded with 4 way interactions and 2 way interactions are confounded with 3 way interactions. Minitab does a really nice job of identifying the resolution of the various designs. Anything colored in green is going to be an excellent choice since these designs are resolution V or higher. Resolution IV designs are shaded yellow and as long as you re comfortable with main effects confounding with 3 way interactions and some 2 way interactions confounding with other 2 way interactions, these designs can also be useful. Design of Experiments GembaAcademy.com 3

4 Minitab shades all resolution III designs in red since main effects confound with 2 way interactions making the results of these designs difficult to interpret. In the end, most practitioners tend to stick with resolution IV or higher designs. That s an overview of Designs and Confounding. 4 Steps to Conducting a DOE Let s move onto the 4 steps to conducting a DOE which as you ll see are the same steps we learned about in the Full Factorial DOE module. The first step is to identify the factors we ll study, as well as, the levels of these factors. Once this is done, we ll then select the appropriate DOE design. In our website design DOE, the first factor is page color and the two levels are white and black. The second factor is button color with the two levels being orange and green. The third factor is button text with one level being the text saying Sign Up and the other level being the text saying Subscribe. The fourth factor is image type. For the first level, the picture will be of a man and for the second level, the picture will be of a woman. Finally, the fifth factor will be the testimonials where the first level is yes, testimonials will appear on the page, and the second level is no, testimonials will not appear on the page. Like the 2 Sample t- Test, the response being studied in this DOE is conversion percentage of newsletter subscribers. Minitab creates a design using default design generators. In our example, where we have 5 factors, the default generator is E = ABCD which simply means that factor E is confounded with the 4 way interaction of A x B x C x D. Minitab also shares the alias structure here. For example, we see that factor A is confounded with the interaction of BCDE. We also see as another example, that the two way interaction of A x B is confounded with the three way interaction of C x D x E. What this really means is that we wouldn t be able to distinguish the difference between the effects of the AB and CDE terms. Finally, Minitab also summarizes Design of Experiments GembaAcademy.com 4

5 the design at the top explaining that this design has 5 factors, is a resolution V design, and has 16 runs. Here s what our 2 to the 5-1 Design looks like. Please note, that we re showing the table in non randomized form for teaching purposes, but you ll always want to randomize your designs when conducting your own DOEs. For the first run of the DOE, the webpage will be configured as follows; the page background color will be white, the button color orange, the button text will say Sign Up, the image on the page will be of a man, and there will not be any testimonials on the page. Once our design has been created, it s time for step 2, plan and prepare for the experiment. This is another very important, yet sometimes neglected, step in the DOE process. If you haven t already done so, please watch the Full Factorial DOE module to hear the tips and suggestions for how to go about planning and preparing for a DOE. Design of Experiments GembaAcademy.com 5

6 Once we ve fully prepared, it s time to run the experiment. Again, we offer many tips and suggestions in the Full Factorial DOE module for how to go about this, so rather than repeating them again, we d encourage you to check that module out if you haven t already done so. For this experiment, the team was able to use standard website optimization software that automatically and randomly changed factor levels as needed until all runs were completed over the course of several weeks. While the design wasn t officially replicated, the software did in fact switch between all 16 runs several times in order to ensure zero bias between the day and time of the experimental runs. Once the experimental runs were completed, the conversion percentages were noted here. Again for this DOE, a conversion occurred when a visitor subscribed to the company newsletter. I want to point out that since this was a website based experiment, the way the DOE was executed was a bit different. If you were running a more traditional DOE where the factors were, for example, machine settings, you would be able to set the machine up accordingly and run the trial one time assuming you didn t replicate the design. At this point we re ready for step 4, analyze and interpret the results. Once we ve selected the response in Minitab, we ll need to specify the model we want to fit. As it turns out, since we re performing a Design, we can t fit all possible terms. Instead, we can only fit terms up through the second order which is what Minitab defaults to. So this is why we choose 2 in the highlighted section. You ll notice that there are no P- values. This happened since the number of runs, 16, Design of Experiments GembaAcademy.com 6

7 equals the number of terms, meaning there are no degrees of freedom left to estimate error. Saturated Models When we have this situation in either a Full Factorial or a Design, we re dealing with what s known as a Saturated Model. When we re working with a saturated model, we have to find another way to identify significant factors since there are no P- values to evaluate. Pareto Chart of the Effects (response is Conversion %, Alpha = 0.05) Term A D CE B AE AD BD C DE BC AB AC CD E BE Factor A B C D E Name Page C olor Button Color Button Text Image Testimonials Effect Lenth's PSE = As it turns out, Minitab uses a special method called Lenth s PSE which stand for Lenth s Pseudo Standard Error of the Effects. In this example, the red line drawn is based on this pseudo standard error. Just like we saw in the Full Factorial DOE module, any Pareto bar greater than the red line is statistically significant meaning in this fictitious example, Factor A or Page Color and Factor D Image type are significant. I mention this is a totally Design of Experiments GembaAcademy.com 7

8 fictitious example since we don t want anyone with a website to assume these factors will be important on your website. At this point, you may be tempted to simply remove all terms except A and D. Since we re working with a Design, we usually prefer to be a bit more conservative and start to reduce the model by removing all the 2 way interactions first while including all the main effects. The easiest way to do this is to simply choose 1 in the pull down box so only the first order, main effects, are shown. Here s what the ANOVA table looks like now. You ll now notice our P- values are back. This is because the 2- way interactions are now being used to estimate the error term. As we now see, Page Color and Image are still the only two statistically significant factors since their P- values are less than the typical Alpha value of.05 meaning we will reject Ho and conclude these main effects are significant. Here s the Pareto Chart of these same effects using a traditional P- value to denote the red line. Pareto Chart of the Standardized Effects (response is Conversion %, Alpha = 0.05) Page Color Image Term Button Color Button Text Testimonials Standardized Effect Design of Experiments GembaAcademy.com 8

9 Again, we see Page Color and Image as the only two significant factors. If the purpose of this Factorial Design was to identify critical factors worthy of deeper investigation, in a Full Factorial DOE we could probably stop at this point and conclude that these two factors are significant. For the sake of completeness, we ll go ahead and reduce the model to only show these two terms. When we do, we see that they are still significant in the ANOVA table and Pareto chart of effects. Last, but certainly not least, we'll want to check the residuals for normality, randomness, and constant variance like we've seen several times throughout the course already. Our residuals seem to be normal, random, with relatively constant variance. Again at this point, we could proceed to a Full Factorial DOE if wanted to, or perhaps we learned all we needed to with this Design and we may decide to test additional factors with another experiment. Design of Experiments GembaAcademy.com 9

Design and Analysis of Experiments Prof. Jhareswar Maiti Department of Industrial and Systems Engineering Indian Institute of Technology, Kharagpur

Design and Analysis of Experiments Prof. Jhareswar Maiti Department of Industrial and Systems Engineering Indian Institute of Technology, Kharagpur Lecture 59 Fractional Factorial Design using MINITAB