Webinar Experimental Designs for CBC

Transcription

1 Experimental Designs for CBC

2 2 Outline Background designs for linear models CBC design strategies Based on orthogonal arrays Using efficiency calculations Sawtooth Software s multi-objective optimization

3 3 BACKGROUND

4 4 Why care? In the absence of an experimental design bad and terrible things can happen: Terrible: Some of your model coefficients (utilities) may not be possible to estimate, your research becomes a crash-and-burn disaster, dollars and careers can go up in flames Bad: your utility estimates lose precision If you don t get precision through your experimental design, you have to buy it with sample size Savings in $,,,, Rs are good reasons to like precision

5 5 Attributes and Levels Attributes are the characteristics of products and levels are the discrete values those characteristics may take on Attribute (levels): Brand (Pepsi, Coke, Dr. Pepper) Capacity (16GB, 32GB, 64GB) Price ($90, $125, $159) Flame-retardant fabric (yes, no)

6 6 Design for Linear Models Orthogonal designs Full factorial designs Show every possible combination of predictors An experiment with three 4 level factors and two 5 level factors (4 3 x 5 2 ) would need observations for 4 x 4 x 4 x 5 x 5 = 1,600 combinations Fractional factorial plans require fewer observations They assume no or very few interactions When they measure main effects only, we call them orthogonal main effects plans (OMEPs) An OMEP for the (4 3 x 5 2 ) design above uses just 25 combinations or runs

7 7 Example Design 3 4 OMEP Run X1 X2 X3 X

8 8 Properties of Example Orthogonality - each level occurs with levels from each other attribute a proportional number of times Correlation matrix: X1 1 X1 X2 X3 X4 X2 0 1 X X Two-way frequencies: X2 X

9 9 Sources of Orthogonal Fractional Designs Addelman (1962) OMEP catalog Designs account for main effects (of individual attributes) not interactions Until 1994, the marketing researcher s bible for creating experimental designs Hahn and Shapiro (1966) designs allowed some interactions Kuhfeld s SAS Technical Report Orthogonal Arrays offers an extensive collection plus links to other design databases

10 10 Using Orthogonal Arrays Traditional conjoint analysis used fractional factorial experimental designs to create a set of stimuli (concepts, profiles, cards) Respondents evaluated these stimuli via ratings or rankings Regression analysis decomposed the evaluations into utilities for each level of each attribute in the study We can also use fractional factorials in Menu-based choice experiments Situational choice experiments Choice-based conjoint (CBC) experiments

11 11 CBC DESIGNS FROM ORTHOGONAL PLANS

12 12 Designs Based on Orthogonal Plans Adapting fractional factorial designs for CBC experiments Louviere and Woodworth J designs 4 more early strategies (Louviere, Hensher and Swait, 2000) Mix and match 1 Mix and match 2 Labeled mix and match L MN Simple shifting (Bunch, Louviere and Anderson 1996) Flexible shifting (Street and Burgess 2007, Street, Burgess and Louviere 2005)

13 13 2 J Designs 1. Use a fractional factorial design to create profiles 2. Use a second fractional factorial design to decide which profiles to assign to particular choice sets (levels are absent and present 3. Hadamard matrices or balanced incomplete block designs can be used for step 2 as well

14 14 Pros and Cons Pros Conceptually and mechanically simple Cons Designs can get very large Different choice sets can have different numbers of profiles

15 15 Mix and Match 1 Algorithm 1. Generate a fractional factorial plan 2. Make some number of k exact copies of this fraction 3. From each of the k+1 equivalent fractions Randomly select one profile without replacement This set of K+1 profiles is choice set 1 Prevent duplicates within a set 4. Repeat step 3 until all profiles are assigned to choice sets This strategy produces an experiment where all choice sets have k+1 profiles

16 16 Example of Mix and Match 1 The same 9 profiles appear in both the first and second alternative Alternative A Alternative B Set X1 X2 X3 X4 X1 X2 X3 X

17 17 Mix and Match 2 Algorithm 1. Generate a fractional factorial plan 2. Make some number of k equivalent (not exact) copies of this fraction* 3. From each of the k+1 equivalent fractions Randomly select one profile without replacement This set of K+1 profiles is choice set 1 4. Repeat step 3 until all profiles are assigned to choice sets This strategy covers more of the design space than Mix and Match 1 *You can do this simply by switching columns, relabeling levels (1 2, 2 3 and 3 1) or both.

18 18 Example of Mix and Match 2 In this experiment unique profiles appear in the two alternatives and we cover twice as many possible profiles Alternative A Alternative B Set X1 X2 X3 X4 X1 X2 X3 X

19 19 Versions We could create multiple versions of a mix and match design One could be the design on the previous slide One could be the previous design where columns 1&2 and 3&4 are swapped in the left hand profile One could swap codes 3 and 2 in the even numbered columns of the right hand profile and 1 and 2 in the odd columns of the right hand profile We could make several other versions in a similar way, swapping columns and/or recoding levels Then we could randomly assign different respondents to different versions When we do this we explore more of the design space We also allow estimation of some interactions

20 20 Alternative-Specific Designs A trick called labeling allows us to make alternative-specific designs In each choice set, label the profile chosen from the first of the K+1 designs with the first brand name Label the profile chosen from the second design with the second brand name Etc. Benefits Brand is now an attribute in your experiment that you didn t have to build into the fractional factorial experiment In analysis, you can have brand-specific effects of price and all other attributes In other words you can estimate interactions of brand with all other attributes

21 21 Example of Alternative-Specific Design Just label the sets of alternatives with brand names Bose JBL Set X1 X2 X3 X4 X1 X2 X3 X

22 22 L MN Designs If we have L levels per attribute, M alternatives in our choice sets and N attributes, build the design from a fractional factorial with M*N columns For example, if we have 4 alternatives with 3 attributes of 3 levels each we need a design with 12 columns worth of 3 level attributes a 27 row design from the Addelman catalog would work In this design variables are independent both within and across profiles This feature allows measurement of cross-effects

23 23 A 3 3*2 Design 3 alternatives, 2 attributes of 3 levels each - this design uses an 18 run fractional factorial design with 7 columns of 3-level variables we use two per alternative, with one unused column Note we have an unused column. We need not use every column of a design. We may also use it for other purposes, like creating blocks of questions so that no respondent has to answer all 18. Set A1 A2 B1 B2 C1 C2 Unused

24 24 Shifting Designs 1. Generate a fractional factorial plan 2. Results of this plan are the first profile of each choice set 3. Second profile is just the first with all attributes shifted 1 becomes 2, 2 becomes 3 so on, with wraparound (highest level becomes 1 again) Example: Six 3-level attributes If first profile is Second profile is and third is

25 25 Shifting Design Example Starting with a 3 4 fractional factorial we create additional alternatives through shifting Note there is no level overlap Note that a 4 th profile would duplicate the first Alternative A Alternative B Alternative C Set X1 X2 X3 X4 X1 X2 X3 X4 X1 X2 X3 X

26 26 Flexible Shifting Designs We could choose to shift different columns of the initial matrix differently For example, we could shift the 1 st and 3 rd columns by one place and the 2 nd and 4 th by 2 places Street and Burgess (2007) call this pattern of shifts, 1212, a generator They show how to use generators to create a variety of designs for main effects and interactions Like simpler shifting designs, these will not work for alternative-specific effects

27 27 Flexible Shifting Example This design uses a 1212 generator Alternative A Alternative B Alternative C Set X1 X2 X3 X4 X1 X2 X3 X4 X1 X2 X3 X

28 28 Comparison of Orthogonal Designs For generic main effects models, shifting and modified shifting give more efficient designs than do mix and match designs Mix and match designs with enough versions allow estimation of interactions For models with alternative-specific effects, use Mix and Match 2 or and L MN designs For models with cross-effects, use L MN designs Avoid shifting if you want to estimate interactions or alternative-specific effects Shifting designs cause minimal overlap

29 29 Pros and Cons of Using Orthogonal Plans Recipe-based design algorithms easy enough that you can create designs by hand or using simple Excel workbooks While the columns of a fractional factorial are orthogonal with respect to one another, they may be perfectly correlated with un-modeled interactions Focus on orthogonality can detract from efficiency Prohibitions must be added manually and there is no good way to do this

30 30 EFFICIENT DESIGNS

31 31 Designs Based on Efficiency We can calculate D-efficiency of a design mathematically Rather than retrofitting orthogonal designs to create our CBC designs, we could search for groups of choice sets that constitute efficient designs Computer search algorithms can assist this process The covariance matrix itself depends on the model we plan to run, so different models require different efficiency calculations

32 32 D-Efficiency for MNL Models Kuhfeld, Tobias and Garratt (1994) show how to use search algorithms to make efficient designs for MNL models Generate many sets of profiles with the right number of attributes and levels and let a search algorithm identify the most efficient way to combine them SAS macros MKTEX and CHOICEFF work together to create a wide range of CBC experimental designs (Kuhfeld and Wurst 2012)

33 33 Further Improvements for MNL Models You can further improve designs with iterative refinements (Huber and Zwerina 1996) called relabeling and swapping If you have reasonably good estimates of respondents utilities, you can do better still (Sandor and Wedel 2001) These are sometimes called D p -efficient designs because we have a guess about their parameters They are distinguished from D 0 -efficient designs that assume null parameters

34 34 Other Model Specifications Efficient designs for mixed logit models (Sandor and Wedel 2002) Efficient designs for hierarchical Bayesian models (Sandor and Wedel 2005) Efficient designs for two-stage models with a noncompensatory first stage (Liu and Arora 2011) Efficient designs for repeated measures logit models (Bliemer and Rose 2010)

35 35 Pros and Cons of Efficiency-Based Designs Relatively easy to implement the basic D 0 -efficient MNL model (using off-the-shelf software) Complexity grows rapidly for different models and requires Extreme programming skills and/or Specialized software and/or Prior estimates of utilities

36 36 Orthogonal versus Efficiency-Based Designs Efficiency based designs are more flexible than orthogonal designs (e.g. prohibitions are easier to accommodate) Efficiency-based designs by their nature will be a little more efficient than orthogonal designs - a lot more efficient according to Bliemer and Rose (2010) The differences may not matter much in practice A lot of sniping and grousing happens anyway

37 37 LIGHTHOUSE STUDIO DESIGNS

38 38 Sources of CBC Efficiency These properties promote efficiency (One-way) Level Balance: within each attribute, each level appears an equal number of times Orthogonality, or two-way level balance: each level appears proportionally often with every level of other attributes Minimal Overlap: minimizing the extent to which an attribute level repeats within a choice set

39 39 Lighthouse Studio s Design Strategies We can balance the sources of efficiency to produce four design strategies Good designs at the respondent level that get better as more respondents answer more versions of the design As the number of versions increases, the difference in D- efficiency between complete enumeration and efficient designs becomes minuscule (Kuhfeld and Wurst 2012) Using many versions means We can cover a larger portion of the design space We can estimate interactions without pre-planning for them Version effects can cancel out

40 40 Complete Enumeration Generation process: Maximize one- and two-way level balance Profiles are nearly orthogonal for a given respondent Features One-way level balance (each level of an attribute occurs as equally often as possible) Two-way level balance (each pair of levels occurs as equally often as possible) Minimal overlap of levels within choice sets (promotes precision of main effects)

41 41 Shortcut Generation process: Similar to Complete Enumeration, except: Profiles are built using previously least-used attribute levels (from previous tasks and versions) Two-way frequency balance is not explicitly controlled Features Minimal overlap of levels between profiles One-way level balance (each level of an attribute occurs as equally often as possible) Loose (not strict) orthogonality

42 42 Random Generation process Profiles sampled randomly, with replacement, from full factorial universe of profiles No duplicate profiles allowed within a choice set Features Only loose one-way level balance (each level occurs approximately an equal number of times) Only loose two-way level balance (each pair of levels occurs approximately an equal number of times) Significant level overlap (an advantage for precision of interactions)

43 43 Balanced Overlap Similar to Complete Enumeration but with more level overlap Features About half as much level overlap as the purely Random strategy One-way level balance (each level occurs an equal number of times) Two-way level balance (each pair of levels occurs an equal number of times) Much better for main effects than Random and much better for interaction effects than Complete Enumeration Balanced Overlap is the default method in Sawtooth Software s CBC software

44 44 Pros and Cons of Lighthouse Strategies Complete Enumeration Shortcut Random Balanced Overlap Fast + + Main Effect Efficiency Interaction Efficiency Deep Processing + + Extreme Prohibitions + +

45 45 QUESTIONS? Keith Chrzan SVP, Sawtooth Analytics Megan Peitz Ingenuity Ambassador

46 46 References Addelman, S. (1962) Orthogonal Main-effects Plans for Asymmetrical Factorial Experiments, Technometrics, 4: Bliemer, M.C.J. and J.M. Rose (2010) Construction of Experimental Designs for Mixed Logit Models Allowing for Corelation Across Choice Observations, Transportation Research Part B, 44, Bliemer, M,C.J., J.M. Rose and S. Hess (2008) Approximation of Bayesian Efficiency in Experimental Choice Designs, Journal of Choice Modeling, 1: Bunch, D.S., J.J. Louviere and D. Anderson (1996) A Comparison of Experimental Design Strategies for Multinomial Logit Models: The Case of Generic Attributes, accessed on January 26, 2013 at Chrzan, K. and B. Orme (2000) Overview and Comparison of Design Strategies for Choice-Based Conjoint Analysis, Sawtooth Software Conference Proceedings, ) Hahn GJ. Shapiro SS. (1966) A catalogue and computer program for the design and analysis of orthogonal symmetric and asymmetric fractional factorial experiments. Report 66-C-165, General Electric Research and Development Centre, Schenectady, New York.

47 47 References Huber, J. and K. Zwerina (1996) The Importance of Utility Balance in Efficient Choice Designs, Journal of Marketing Research, 33: Kanninen, B. (2002) Optimal Designs for Multinomial Choice Experiments, Journal of Marketing Research, 39: Kuhfeld, W.F. Orthogonal Arrays. Technical Report, SAS Institute. Accessed on January 26, 2013 at Kuhfeld, W.F, R. Tobias and M. Garratt (1994) Efficient Experimental Design with Marketing Research Applications, Journal of Marketing Research, 31, Kuhfeld, W.F. and J.C. Wurst (2012) An overview of the Design of Stated Choice Experiments, Sawtooth Software Conference Proceedings. Orem: Sawtooth Software Liu, Q. and N. Arora (2011) Efficient Choice Designs for a Consider-Then- Choose Model, Marketing Science, 30: Louviere, J.J., D.A. Hensher and J.D. Swait (2000) Stated Choice Methods: Analysis and Application. Cambridge: Cambridge University

48 48 References Louviere, J.J. and G.W. Woodworth (1983) Design and Analysis of Simulated Consumer Choice or Allocation Experiments: An Approach Based on Aggregate Data, Journal of Marketing Research, 20: Sandor, Z. and M. Wedel (2001) Designing Choice Experiments Using Managers Prior Beliefs, Journal of Marketing Research, 38: Sandor, Z. and M. Wedel (2002) Profile Construction in Experimental Choice Designs for Mixed Logit Models, Marketing Science, 21: Sandor, Z. and M. Wedel (2005) Heterogeneous Choice Designs, Journal of Marketing Research, 42: Street, D.J. and L. Burgess (2007) The Construction of Optimal Stated Choice Experiments. Hoboken: Wiley. Street, D.J., L. Burgess and J.J. Louviere (2005) Quick and Easy Choice Sets: Constructing Optimal and Nearly Optimal Stated Choice Experiments, International Journal of Research in Marketing, 22: