Running head: PROJECT 4: THE EVALUATION PLAN 1. Project 4: The Evaluation Plan. Rachel Murch. EDCI 577 Section 4.

Transcription

1 Running head: PROJECT 4: THE EVALUATION PLAN 1 Project 4: The Evaluation Plan Rachel Murch EDCI 577 Section 4 Purdue University

2 PROJECT 4: THE EVALUATION PLAN 2 Executive Summary MailChimp is an online marketing company that can be used by anyone who needs to send mass s or newsletters. With a wide audience of customers who continue to use the tool and pay for the service, MailChimp is sure to increase profits by increasing its customer base. To help increase the number of customers, MailChimp needs to evaluate the support resources that customers use to learn how to upload a list of addresses and send an (or campaign). The Getting Started online training videos will be evaluated to determine how effective they are in making it easy for customers to use the MailChimp service. The objective of this evaluation is to increase customer satisfaction and revenue by providing online training videos that teach customers how to use the MailChimp tool. MailChimp s evaluation plan will be executed using Kirkpatrick s 4-level evaluation model, which includes measuring customer reaction, learning, behavior, and results. The entire evaluation process will take 10 weeks, but can be repeated as necessary once the first cycle is complete. Customer evaluations will be administered over a 4-week time period, with results being tracked on a daily or weekly basis. Since interaction with customers is virtual, the evaluations will be delivered and tracked electronically, which will help to keep evaluation costs down. The level 1 evaluation will include both a customer reaction questionnaire and a continuous feedback option. These tools are designed to gather feedback about the training videos so that adjustments can be made to improve the learning experience, if necessary. Level 2 will be measured using a combination of pre- and post-tests and using a focus and experimental group of customers. Using a focus group will help to isolate the learning that occurred because of the training videos. Also, participation will be voluntary but incentives will be used to increase participation. The third level will be evaluated using Google Analytics to track customer behavior on the website. By using this tool, the evaluation team will have a good understanding of when the videos are being accessed and what section of the website the customer came from. Level 4 will be measured by taking some customer and revenue statistics before and after the evaluation program is administered. These calculations will help to determine the results due to the training. A final report will be developed to show the evaluation results and any recommendations. Expected results of the evaluations include a significant increase in revenue and customer use of the MailChimp service. If the results prove to be different, then the training program will need to be adjusted appropriately.

3 PROJECT 4: THE EVALUATION PLAN 3 Introduction MailChimp provides online marketing tools such as mass and newsletters that are simple, easy to use, and fun! These tools are used by people and businesses all over the world because MailChimp makes marketing so simple and attractive. Though the culture at MailChimp is very casual, don t be fooled, because the 100 or so employees are focused on providing the best -marketing service in the universe and providing a personal publishing platform for users. One obvious advantage that MailChimp has over its competitors is its user interface. When accessing the home page, it is hard not to feel invited in to explore. Users from the novice to the most advanced marketer are comfortable diving in to see and love what MailChimp has to offer. So what does MailChimp have to offer? This paper will focus on the section of the MailChimp website that provides user support. This area of the website includes some Getting Started videos that guide the user through the process of sending s (or campaigns in MailChimp terminology). Because the target market for MailChimp is so wide, with a variety of experience levels, it is important to support all users in a way that meets their needs and also matches the company culture. MailChimp seems to have found a way to do that through fun and informative online training videos. However, it would be useful to validate MailChimp s assumption by conducting a full evaluation program using the 4-level Kirkpatrick evaluation framework as a guide. The instructional product being evaluated in this paper is Parts 1 and 3 of MailChimp s Getting Started Videos. Part 1 covers Creating and Importing a List of Subscribers, and Part 3 covers Creating and Sending a Campaign. These support videos are available to the public and located on the Support tab of the MailChimp.com website. This document will outline an evaluation plan and framework, the tools and procedures for doing such evaluations, and the timeline for which to do so. Audience and Purpose MailChimp is an online company that caters to anyone who uses the Internet and wishes to send a large number of s (called campaigns). A tool like this can be very useful because many clients have limits on the number and size of s that can be sent. Even with these limits, a client can usually send a large list of s to friends and family for reasons such as electronic holiday cards, invitations, or announcements. In a business environment, however, the given limits can be very restricting. One possible way around this is to send s to sections or groups at a time, or to use a tool like MailChimp. Because people and companies have different needs, MailChimp caters to a diverse group of users. Its target audiences include people who want to send a campaign one time, those who want to send an occasional campaign, and those who send campaigns regularly. Regular users will not reference the training videos frequently, but casual users will reference the videos again and again. The training video modules evaluated in this paper are geared toward those people who have not used MailChimp before (or who don t use it often) and who want to understand exactly how to create an campaign and create or import a list of people to send the to. The content for these two modules covers the basic requirements for sending a campaign.

4 PROJECT 4: THE EVALUATION PLAN 4 The purposes of this evaluation program are to: 1) gather user reaction and gauge the level of learning from Parts 1 and 3 of the Getting Started training videos; and 2) determine the effectiveness of the Getting Started training videos. Evaluation Framework As mentioned above, the Kirkpatrick evaluation method is the framework used to evaluate MailChimp s Getting Started online training videos in this Evaluation Plan. The traditional 4 levels for this model include measuring reaction, learning, behavior, and results. The following sections describe the tools used for each level of evaluation, the procedures with which to implement those tools, and possible limitations. Level 1: Evaluating Reaction The first measurement instrument (Appendix C) is a questionnaire designed to capture the reaction of customers who use the training videos as a support resource. Because MailChimp is an online company, it must be able to support its customers virtually. Measuring customer reaction to the support resources is one way to gauge customer satisfaction with regard to the training videos (Kirkpatrick & Kirkpatrick, 2006, p. 27). Also, if customers know how to use the MailChimp tools, then they are more likely to have a good experience with the website and want to continue using it, which keeps the company in business. For this reason, the training videos must be a great resource to provide virtual customer service and support. Measuring customer reaction can be done in many ways, but for this situation the reaction questionnaire would launch at the end of each training video. Having the questionnaire linked to the video would help to increase the response rate (with the optimum response rate being 100%). However, because many customers could view the videos again and again, there would be an opt-out button so that customers wouldn t get upset that they are forced to provide feedback with each viewing. This opt-out option could lead to a lower response rate, but the benefit of not upsetting customers would outweigh the risk. The first part of the reaction questionnaire contains multiple choice questions designed to gather information such as which training modules the users viewed, their overall reaction to the video, and the quality, timing, and professional design of the videos. These questions can help MailChimp understand what characteristics of the videos may need to be improved upon. The second part of the questionnaire asks customers to identify both things that they felt could be improved, and things they liked best. Allowing some open-text answers will enable MailChimp to get honest feedback from customers. Another way to measure reaction in an online environment is to gather feedback as the learner is doing the training. A continuous feedback icon (Appendix D) would be available on the screen as the customer is viewing the training videos. It would offer the opportunity for customers to click and provide feedback at a particular section of the content when the thought is fresh in their minds. Implementing the reaction evaluation tools does not have to be contingent on this evaluation process. Similar feedback is customarily gathered continuously for any and all learning tools or classes. For this reason, the reaction tools could be implemented as soon as

5 PROJECT 4: THE EVALUATION PLAN 5 possible, though the formal process that includes analysis and reporting will not begin until the beginning of November However, for the purpose of this plan, all documents and timelines have been created with the assumption that all evaluations will follow the timeline created for this evaluation program. Level 2: Evaluating Learning The second measurement instrument (Appendix E) is designed to capture if and what learning occurred from watching the video training modules. Certain skills and knowledge are needed in order to efficiently and appropriately use MailChimp s service, which can also impact levels of customer satisfaction (similar to reaction). The learning instrument consists of multiple choice, fill-in, and true/false questions that directly relate to the content covered in the training videos. Using a focus group and experimental group, and administering pre-and post-tests to measure learning, are great ways to isolate what learning occurred because of the training. MailChimp learning can be evaluated using both of these methods, and though participation from users is critical, this can be successful if the right tools are used. In an online environment, there is no option to observe customer behavior in person, so it is necessary to use an analytics tool to track customer behavior on the MailChimp website. See below for the procedures and activities for the focus group and experimental group: Focus Group. The focus group would consist of customers who did not watch the training videos, but who were able to create a list and send a campaign to subscribers anyway. This group would be randomly selected to participate in a process to evaluate how user-friendly the website is without instruction. Though participation would be optional, an incentive of a free campaign would be offered to help entice customers to participate This group would first be asked to take a pre-test learning questionnaire (Appendix E). Next, they would be asked to create a list and send a campaign to that list of subscribers. Then, once the campaign was sent, this group would be prompted to complete the post-test learning questionnaire (Appendix E) that is comprised of the same questions from the pre-test. Experimental Group. The experimental group would consist of customers who are randomly selected to take part in a process to evaluate the usefulness of the training videos, with the incentive of getting a free month of campaigns. First, they would be prompted to take a pre-test learning questionnaire (same as what the focus group took). After taking the pre-test, they would be asked to watch the training videos. Once the training videos were viewed, they would be instructed to create a list and send a campaign to that list of subscribers. The last step would be for them to take the learning questionnaire as a post-test. See Table 1 for the Evaluation Tools for Levels 1 and 2. There may be some challenges with this group due to the amount of time and effort it will take to participate. Though each participant gets a free month of campaigns, it might not be enough of an incentive. The compensation needs to be worthwhile and should be increased as necessary to increase participation.

6 PROJECT 4: THE EVALUATION PLAN 6 Level 3: Evaluating Behavior Tracking customer behavior on the web can be done efficiently and accurately by using Google Analytics. This tool makes it possible to do many things, including tracking specific events that take place online. This is called Event Tracking, and will be used in this evaluation to track how many times a user clicks on a particular link, button, or webpage. In addition, tracking will include the location from which the customer originated. See Example 1 below. Example 1: Situation: Customer A is a participant in the Evaluation Program. Customer A and is on the Campaigns screen and navigates away from it, to the Resources page and clicks play on Video 3. After watching the video, Customer A navigates back to the Campaigns page and successfully creates a campaign. Google Analytics Results: Customer A came from the Campaigns page and watched Video 3, one time. The assumption, based on Example 1, is that Customer A was unaware of how to create a campaign and therefore watched a training video to learn. Because Customer A is a participant in the Evaluation Program, MailChimp also has tracking of whether he/she is a part of the focus or experimental group. If he/she was in the focus group, then the training hadn t been viewed before, and therefore the training was effective in helping create a campaign. If Customer A was

7 PROJECT 4: THE EVALUATION PLAN 7 in the experimental group, then some assumptions can be made about the training itself. It is possible that the video was too complicated to digest in one viewing, or for some other reason that may have been indicated in the reaction feedback from the level 1 evaluation. Appendix F shows a tracking table that will be used to capture customer behavior. The tracking will show which main page the customer came from: Dashboards, Campaigns, Lists, or Reports, and how many times the play button was clicked on which video. This information is valuable because if a customer is on Page X and goes to the Support page and watches Video X, then it is understood that the user didn t know or remember how to do something on that page. Although online customer behavior can be tracked, it is impossible to understand the reason behind that behavior. For example, a customer may have accidentally clicked on a link, page, or button and it was an unintended behavior. The electronic tracking of that behavior only knows that it happened, not for what reason it happened. For this reason, the evaluation team will need to look at statistically significant outliers and exclude them from the results as a part of the analysis process. Level 4: Evaluating Results Evaluating the results of the Getting Started training videos will be done electronically, using a predetermined set metrics that can be tracked and reported on automatically. These include: 1. Average number of campaigns sent per user 2. Average number of list uploads per user 3. Average number of new customers 4. Average revenue per user account. The calculations for the above information will be made at the beginning of the evaluation program, as listed in the Evaluation Plan (Appendix A) and in the Gantt Chart (Appendix B). To track progress, a pulse will be taken at the end of each week, for four weeks. At the end of the evaluation program, the final calculations will be made and compared to the baseline metrics. To do this, a report will be generated for the evaluators to use when writing the final report. In addition, the evaluators will complete a questionnaire that asks them to identify potential external reasons why the calculations are what they are. For instance, there may have been some site downtime that caused customers to not be able to login to their accounts. Another reason might be that the upload feature of the website was down for a period of time. It is important to include an option for the evaluators to provide any explanations of situations that could have impacted the results and would not have been part of the calculations on the report. See Appendix G for an example of the report and the questions that accompany it for the final results. Reaction Evaluation Expected Findings and Reporting The reaction questionnaire (Appendix C) results will be gathered and reported in an electronic format that allows for easy viewing and analysis of the data. Acceptable results for this evaluation would be an average of 4 (or equivalent) or better. For instance, the first main

8 PROJECT 4: THE EVALUATION PLAN 8 question asks how the person would describe the training, and an acceptable average reaction level would be the It was super helpful! or It was helpful answers only. If training isn t helpful, then it is useless. For the other questions, an average rating of 4 or 5 would be acceptable. The open-ended questions would be used for analyzing the questions that precede them. Using the electronic tool that tracks these responses, the training team must first determine what the average rating was for each question on a weekly basis. Based on those ratings, it might then make changes to the training videos, such as timing, narrators, and/or design. Weekly reports will be created and available for company management as necessary. See Table 2 for an example of the Level 1 Results Report. Table 2

9 PROJECT 4: THE EVALUATION PLAN 9 The results of the second reaction evaluation (Appendix D) will be monitored on a daily basis by a help desk team. This continuous feedback would either be acted on immediately, in the event that there is an issue with the videos, or it would be sent to the training team and added to the data from the last two questions on the reaction questionnaire. Learning Evaluation Pre-test and post-test scores for the focus and experimental groups will be tabulated (electronically) and compared. Expected Results: Pre-test scores for the focus group are expected to be similar (+/- 5%) to those of the experimental group. Post-test scores for the experimental group are expected to show an increase (approximately 20%) higher than those of the focus group (calculation: post-test score average minus pre-test average score). Reporting on the learning evaluations will be done based on the number of hits to the training videos. The higher the number of hits, the more often the evaluations will be tabulated, and vice versa. At a minimum, a monthly report will be generated and analyzed by the training team. The team would need to take into consideration unknown variables such as customer experience using online training tools, customer comfort with technology, incomplete evaluations, and the fact that customers may have been rushing through the questions. Behavior Evaluation Customer behavior will be tracked and reported through the Google Analytics tool using Event Tracking. The expected behavior is such that a user who has already viewed the Getting Started videos (or experimental group) will navigate back to the videos 25% less than the users who did not watch the videos in the first place (or focus group). See Appendix F for the tracking that will occur while a user is on the MailChimp website. Results Evaluation The results report (Appendix G) will be used to report weekly results to management. At the end of the evaluation period, the report will also be used to compile the final report. Automatic processes will provide the statistics, but the some of the analysis will need to be completed by one of the evaluators. The objective of the evaluation process is to increase customer satisfaction and revenue by providing online training videos that teach customers how to use the MailChimp tool. Ultimately, the management team is looking for an indication that the videos are effective at increasing the use of the MailChimp tool. An increase in the use of the tool correlates to increased revenue. The expected results include: A 25% increase in the average number of campaigns sent per user.

10 PROJECT 4: THE EVALUATION PLAN 10 A 10% increase in the average number of list uploads per user. A 5% increase in the average number of customers/users. A 20% increase in the average revenue per user account. Conclusion MailChimp has a diverse audience of people who want an marketing tool that is easy to use. Engaging and informative online training videos can help make the MailChimp service simple and attractive to a customer. If a customer is happy with the service and the support offered, he/she is more likely to continue to use the service. For this reason, Parts 1 and 3 (the tool basics) of the Getting Started training videos are critical to the success of the business. This evaluation will be used to gather information to help make the training videos more effective, and subsequently, increase revenue. Measuring customer reaction, learning, behavior, and results will help provide a pulse of how effective the videos are now, and will unveil any areas needing more attention or improvement. The procedures and tools for each of the levels of evaluation are described in this document and will serve as a roadmap for executing the evaluation program.

11 PROJECT 4: THE EVALUATION PLAN 11 References Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs: The four levels (3 rd ed.). San Francisco: Berrett-Koehler Publishers, Inc. Part 1: Creating and Importing a List of Subscribers [Video file]. Retrieved from Part 3: Creating and Sending a Campaign [Video file]. Retrieved from

12 PROJECT 4: THE EVALUATION PLAN 12 Appendix A Evaluation Plan Project Scope Creator ID Purpose Evaluation Plan Mailchimp Online Training Videos Evaluation of Parts 1 and 3 of the Getting Started Training Videos Rachel Murch Mailchimp Online Training Evaluation Plan Goals Determine the value of the training videos. Levels of Evaluation 1 Reaction 2 Learning 3 Behavior 4 Results Objectives that will be measured Increase customer satisfaction and revenue by providing online training videos that teach customers how to use the Mailchimp tool. Process Task Final Date Due Who Create evaluation plan: 10/19/2012 Eval Team Lead Develop instruments: 10/24/2012 Evaluation Team Level 1 Reaction 10/16/2012 Person 1 Level 2 Learning 10/18/2012 Person 1 Level 3 Behavior 10/23/2012 Person 2 Level 4 Results 10/24/2012 Person 3 Set up Google Analytics Event Tracking: 11/2/2012 Tech Team Collect and store baseline metrics: 11/2/2012 Evaluation Team Launch Level 1 & 2 Evals in Production: 11/5/2012 Tech Team Collect and Analyze data (weekly) 11/5-12/4/2012 Person 2 and Person 3 Prepare final report 12/13/2012 Eval Team Lead and Eval Team Review/present results to Steering Committee Tech Resources: $9300 Evaluation Resources: $15000 Google Analytics: $ /14/2012 Evaluation Budget Eval Team Lead, Tech Team, and Eval Team TOTAL: $26800

13 PROJECT 4: THE EVALUATION PLAN 13 Data Item Continuous Feedback Reaction Pre-test Post-test Event tracking Company revenue Weekly avg of new customers Weekly avg campaigns sent per user Weekly avg of lists uploaded per user Target Measures Reaction Learning Behavior Results Data Collection When How Collected From Whom During training Electronic Survey Customer/User After each training video Electronic Survey Customer/User Before training Electronic Survey Customer/User After watching the training Electronic Survey Customer/User After training, while customer is using the Mailchimp tool Before launching the eval program and after each of 4 weeks Before launching the eval program and after each of 4 weeks Before launching the eval program and after each of 4 weeks Before launching the eval program and after each of 4 weeks Average scores Electronic tracking Ranking and comparison using output averages Internal finance tools Internal finance tools Internal finance tools Data Analysis Methods of performing calculations Average scores and common themes Google Analytics Internal finance tools Report of Findings How to control for other factors Company health report Company health report Company health report Material only covers what is in training videos Google Analytics tracking specific behavior Manual explanations Customer/User Company health report By Whom Electronic Electronic Electronic Electronic Google Analytics/ Electronic Finance and Eval Team Finance and Eval Team Finance and Eval Team Finance and Eval Team Who will perform analysis Continuous feedback link Person 1 Person 1 Person 2 Eval Team Lead and Person 3 What results are reported To whom How communicated 1. Transfer of learning metrics 2. Value of training 3. Action plan for modifications Steering Committee Electronic report and presentation

14 Running head: PROJECT 4: THE EVALUATION PLAN 14 Appendix B Gantt Chart

15 Running head: PROJECT 4: THE EVALUATION PLAN 15 Appendix C Level 1 Evaluation Thank you for using our online training tutorials! So that we can continue to provide awesome support for you, we ask that you please complete this brief survey. I completed the following online training module(s): Part 1: Creating and Importing a List of Subscribers Part 3: Creating and Sending a Campaign I would describe the online training to one of my friends like this: It was super helpful! It was helpful! It was fine. It was not as helpful as I would have liked it to be :\ It was a complete waste of time Please use the following scale: 5- Strongly agree 4- Agree 3- Neutral 2- Disagree 1- Strongly disagree The quality of the online tutorials was professional: The videos were enjoyable to watch and kept my interest: The timing of the videos was appropriate for the topics covered: The narration of the video was clear and easy to understand I feel confident that I can create and send a campaign to a list of subscribers The videos could be improved in the following ways: What I liked best about the training videos is: Thank you for providing honest feedback!

16 PROJECT 4: THE EVALUATION PLAN 16 Appendix D Level 1 Evaluation: Continuous Feedback

17 PROJECT 4: THE EVALUATION PLAN 17 Appendix E Level 2 Evaluation Pre-test and Post-test Please select the option that best describes you: I have NOT watched the MailChimp Getting Started training videos I HAVE watched the MailChimp Getting Started training videos MailChimp terminology for content is a: MailChimp terminology for a list of addresses as a: Place a checkmark beside the two requirements, as mandated by law, that any coming from MailChimp must have: An easy way for subscribers to remove themselves from your list A subject line explaining what the is about The business address from the business that is sending the A plain-text only version available To ensure that your has the highest number of opens and click-throughs, it is a good idea to allow your subscribers an opportunity to view the in different formats. True False Creating different groups can be useful when: When importing a list, you can use any kind you want, including 3 rd party lists. True False Select all the options available to you when creating your list: Copy/past from Excel Upload from a file Upload list from Constant Contact Type everything in manually Import from a number of external services such as Salesforce All of the above Select the four types of campaigns that MailChimp offers: Regular ol campaigns Plain-text campaigns Fun campaigns A/B split campaigns Professional/direct campaigns RSS-driven campaigns By selecting a segment of an existing list, you can target subscribers for a particular campaign. True False

18 PROJECT 4: THE EVALUATION PLAN 18 By selecting to use Google Analytics, you can see if your campaign is driving traffic back to Google. True False Choose the template options available to you when designing the content for a campaign: Create a basic template Create a designer template Code your own template Choose from a previously saved template Import a previously created template from an external source All of the above I can make the campaign look the way I want by uploading pictures and changing the color, font, and style of it. True False Campaigns can be personalized to each recipient by using the function inside the Style Editor. There is an option to send a campaign right away or schedule it to be sent on a different day and/or time. True False Thank you so much for participating! You now have one FREE campaign added to your account!

19 Running head: PROJECT 4: THE EVALUATION PLAN 19 Appendix F Level 3 Evaluation

20 PROJECT 4: THE EVALUATION PLAN 20 Appendix G Level 4 Evaluation Has the average number of campaigns sent per user increased? Yes No If yes, please describe any events or activities that can explain this (unrelated to training): Has the average number of lists uploaded per user increased? Yes No If yes, please describe any events or activities that can explain this (unrelated to training): Has the average new customers per week increased? Yes No If yes, please explain any events or activities (such as a new promotion campaign) that can be attributed to this result (unrelated to training): Has the revenue per user account increased? Yes No If yes, please explain anything that can be attributed to the increase in revenue (unrelated to training):