How to Design Skill Certifications Like a Pro

Transcription

1 How to Design Skill Certifications Like a Pro Liberty J. Munson, PhD. Principal Psychometrician Microsoft Worldwide Learning Brought to you by

2 Shreya Sarkar-Barney, PhD. CEO & Founder Liberty J. Munson, PhD. Principal Psychometrician 2

3 We help organizations achieve better outcomes through talent using science, analytics, and empathy. Leadership Excellence US India Talent Management Excellence Evidence-based solutions for measurable and sustained improvements in leadership Drive business impact through timely talent actions 3

4 The Zoom Web Conferencing Platform To test/adjust computer audio Chat for Tech support or to comment Raise Hand to comment by voice If you require HCG or HRCI recertification credits for your participation, please be sure to join with your full name. 4

5

6 a. Yes, they are required to perform certain roles b. Certifications are preferred but not required c. Certifications are not required for any of the roles 40.74

7 a. Employees are encouraged to earn industry recognized external certifications b. We develop role/skill specific certifications in-house c. We use external and internally developed certifications 7.41 d. We have no formal requirements around certifications 37.04

8 a. Employees become certified after demonstrating they can perform their role b. Employees become certified after completing required training c. Employees must pass a test to become certified d. Employees must pass a test and demonstrate their skills on the job to become certified 37.04

9 a. Employees become certified after demonstrating they can perform their role b. Employees become certified after completing required training c. Employees must pass a test to become certified d. Employees must pass a test and demonstrate their skills on the job to become certified 37.04

10 a. Technical roles (e.g. programmers, IT, cyber security) b. Management roles (e.g. Project management) c. Finance (e.g. risk management) 3.70 d. Sales (e.g. Product knowledge, sales skills) e. Functional roles (e.g. HR, Legal, marketing) 33.33

11 a. Options for developing a certification program and the tools required b. The strategic value of assessments and certifications in a learning culture c. The relationship between certifications and role readiness d. How to build and maintain an assessment program to promote a learning culture 33.33

12

13

14 Learner Obsessed: Ensure validity, reliability, psychometric soundness of all skills validation processes (exams, assessments, etc.) Ensure that we are creating quality content Maintaining and sustaining cloud-based exams Simplify: Developing repeatable and scalable processes to minimize internal resources to develop and sustain exams Invest in Innovation: MOOC to cert pilot Adaptive testing/learning pilot Investigating how AI, ML, and bots could fundamentally change exams and certification Innovating to develop content more quickly to support cloudbased exams and internal needs Data Driven: Participate in hacks to better understand learner and test taker behaviors Evaluate the effectiveness of initiatives to improve learning Work with teams to better surveys that will results in clear recommendations Driving Impact: Reduced the costs of exam development by more than 50% Reduced development time by 30% Defined process to keep cloudbased exams up to date Regularly monitor exam quality (psychometrics) and passing rates to quickly identify issues before they affect candidates, partners, and trainers Identified approaches to drive closer alignment between exams and training Ensure exams are valid and reliable drives overall value of certification program Thought leader in certification industry driving changes in the way industry thinks about certification

15 Overview of Microsoft s Certification Program

16 Enabling the next generation of innovators, entrepreneurs, and developers Creative Coding through Games and Apps Harvard/MSFT CS50.AP C# for Absolute Beginners Hour of Code Minecraft Machine Learning Microsoft Azure Fundamentals Programming in C# MOC On Demand Learning Credits Badges Microsoft Professional Degree Total reach New/Competitive Developers reached Skilled technologists building on Microsoft platforms and services Preference for Microsoft platforms & services Spark Excitement Drive Deep Engagement Enable Capability Mastery Affinity Adoption Advocacy Empower Students & Educators Activate Cloud & Productivity Partners Enable Customers & Partners at Scale Drive Usage, Deployment & Consumption

17 Drive adoption of Microsoft technologies through training and certification Similar paths for Office: Microsoft Office Specialist Microsoft Office Expert Microsoft Office Master

18 Microsoft Certification Structure

19 410: Installing and Configuring Windows Server : Administering Windows Server : Configuring Advanced Windows Server 2012 Services MCSA Windows Server 2012 CLOUD PLATFORM & INFRASTRUCTURE 740: Installation, Storage, and Compute with Windows Server : Networking with Windows Server : Identity with Windows Server 2016 MCSA Windows Server 2016 Elective MCSE Cloud Platform & Infrastructure Earned: : Managing Microsoft Azure Infrastructure Solutions LFCS: Linux Foundation Certified System Administrator MCSA Linux on Azure Choose two from: 532: Developing Microsoft Azure Solutions 533: Managing Microsoft Azure Infrastructure Solutions 534: Architecting Microsoft Azure Solutions 473: Designing and Implementing Cloud Data Platform Solutions 475: Designing and Implementing Big Data Analytics Solutions MCSA Cloud Platform

20 Exam 1 Exam 2 MCSA MCSA Elective Exam MCSE/D MCSE/D Different Elective or Retake Exam MCSE/D + Year Renewed

21 Renewing Certifications & Retaking Exams Certifications include achievement date Signifies investment in staying up to date on the technology Every year, they can re-earn their MCSE certification by passing an additional exam from the list of electives Option to retake exams where technology is rapidly changing either to renew their MCSE or to simply show currency in the technology

22 Computer administered Test center deliveries or online proctoring An Overview of MS Certification Exams Global distribution Ongoing delivery Variety of item types (e.g., multiple choice, drag and drop, active screen, hot area, case studies, labs, text entry, code analysis, etc.) 22

23 Development Process

24 Research Candidate Objective Domain Blueprint Manager Item Development Item Selection Field Testing (a.k.a. Beta Exam) Alpha Review & Item Revision Standard Setting Forms Assembly Publish Exam Localized Exam Ongoing Psychometrics

25 Keys to Successful Evaluation Focus on application level questions rather than remembering or understanding level questions

26 Key Attributes of Well Designed Measurement Application of knowledge, skills, and abilities, not recall, understanding, etc. Present a scenario describing a business problem or situation that requires application of skills in ways that are not directly covered in the training Ask about a real-world application of a concept covered in the training Require the learner to: Analyze and provide a solution for a complex problem Combine several concepts, ideas, or skills to solve the problem Identify and analyze patterns based on a limited amount of information or data Design a solution by drawing on multiple skills learned during the training, and/or Evaluate outcomes that may result from actions taken Assess ability to apply knowledge to real-world problems, and/or Ask about concepts, actions, and outcomes that require extrapolating beyond what is specifically covered in the training or linking information from multiple sources

27 Stem = Question Congress A. John Hanson B. John Adams C. Thomas Jefferson D. George Washington Options = Key + Distracters Continental Key = Correct Answer Distracters = Wrong Answers

28 Stem Format Open The year that John Adams was elected President: Closed (Preferred) In what year was John Adams elected President? Best Practices Succinct remove unwanted language Relevant and important Non-trivial Stem is NOT teaching Avoid using Not and Except Avoid using definitions in stem Avoid two questions at once Quality Check Can you cover the options and answer the question? What is the capital of France? A. Lyon B. Paris C. Normandy D. Orlean

29 Key: Best Practices Don t be different from the other options such as the longest and technical Don t use words that are in the stem and repeated only in the key = cueing Distractors: Best Practices Incorrect Plausible Common misconceptions No overlap

30 Formatting All options begin with the same part of speech (e.g., gerund, infinitive) Example - A) Jumping B)Ran C)To flee D) marathons Ordering Alpha-numerical Example - A. 5.0% *B. 25.0% C. 6.5% D. 3.0% Overlap A) 10-30lbs B) 20-60lbs C) 55-95lbs Proper spelling, grammar and punctuation Use third person Don t use you Example If you want to distribute shares, the statue you would be most concerned with is:

31 Avoid Specific Determiners Do NOT use: always, never, only, appropriate, proper, typically, usually, and might Avoid Specific Modifiers Unless you are asking best answer questions with clear documentation supporting what is best, do not use best, most, and least because they are subjective Do NOT use All or None of the Above Acronyms Spell out acronyms first, unless they are understood by the professional community

32 Be Succinct Write using the fewest number of words to answer the question Example The best part of junk food is how good it tastes. According to the FDA, how much sugar should be consumed daily? Eliminate Repetition Place all common wording in the stem that appears in all the options A. Health Care Food Act B. Dietary Act C. American Health Eating Act D. Anti Junk Food Act Ensure Test Fairness Avoid specific cultural, gender, ethnic, religious, or personal pronoun references Example A Japanese female decides to calculate how many calories a Christian eats. She needs to determine the square feet immediately after the 9/11 tragedy for the entire Word Trade Center including the chapel and mosque? What is the square feet of the Word Trade Center that she needs to calculate if the dimensions are 10 x 10 feet?

33 How Can This Apply in Your Organization?

34 Microsoft s Technical Credentialing Program

35 Creating a Learning Culture How? Establish clear links between learning and performance Reward appropriate risk taking Ensure that learning is applied to the job Make employees more accountable for their own learning path Built on a growth mindset What is it? Set of organizational values, processes, and practices that encourage employees to increase knowledge & competence Belief that potential is not fixed, that failure is on the path to mastery, willing to take risks See gaps in their knowledge as opportunities to learn See learning as a journey

36 1 2 3 Overview Ensuring these employees build and validate their technical knowledge and skills on an ongoing basis and in conjunction with broad readiness goals Implementing technical credentialing requirements on a consistent, predictable, annual rhythm Presenting role-based requirements to the field and their managers that are clear and actionable

37 Very Effective Effective Somewhat effective Not at all Effective

38 ,200+ technical sales professionals required to meet technical credentialing requirement to demonstrate readiness 76% find certification useful to job effectiveness Azure certification Dynamics certification Office and productivity certification Windows and mobility certification Since launch of first Azure initiative in FY15 over 20,000 certifications achieved All MSFT FY17 target audience

39 Interesting: Those who selected exams are more likely to believe that the tech requirement has improved job effectiveness

40 Timeline An Example Sept 1 FY18 May 31 FY19 Q1 Q2 Q3 Q4 Q1

41 Key Changes Exam requirement replaced with a combination of exams and courses Pass 1 exam or 3 courses to fulfill your requirement Additional options (new exams, courses, and activities) may be added as they are made available Same as FY17 Where: Requirement appears in My Learning Who: Applicable to those in role by September 1 Recommended for those who enter role after that date When: Complete by May 31 Role Business Intelligence TSP Collaboration TSP Course Options Complete any three of relevant Open EdX courses 6 available to date - Excel, Power BI, O365 Additional courses/activities TBA throughout FY18 OR Complete any three of relevant Open EdX courses 23 available to date - Exchange, SharePoint, O365 Additional courses/activities TBA throughout FY18 OR Exam Options Pass any one of the following exams: : Managing Office 365 Identities and Requirements* TBA Power BI exam (late FY18) NOTE: Practice exams and prep resources will likely not be available for some time after release! Pass any one of the following exams: : Managing Microsoft SharePoint : Designing and Deploying Microsoft Exchange Server : Managing Office 365 Identities and Requirements* : Enabling Office 365 Services*

42 Characteristic Assessment Certification Primary purpose Purpose and scope of evaluation process Provide instruction/training to aid in the acquisition of knowledge/skills/competencies (learning through instruction) Evaluate accomplishment of intended learning outcomes of a specific education/training program Assess knowledge/skills/ competencies that have already been acquired (validation through testing) Confirm mastery of the knowledge/skills/competencies required to effectively perform a job function or occupational/ professional role Duration of program Ends when certificate is awarded Ongoing; requirements must be met on a routine basis to maintain credential (recertification) Recognition Appears on transcript; may be related to badging but no official acronym follows learner s name Recipient uses an acronym or letters after his/her name to highlight certified status

43 Formative vs. Summative Assessment: Driving a Learning Culture Formative assessment: Allows the learner to attempt problems multiple times, often providing hints, so s/he can identify areas of weakness for ongoing learning Summative assessment: Evaluation if test taker has the skills, learned the course objectives, etc.

44 Combining Learning with Evaluation: Moving Beyond Certificate to Certification Formative assessments built in the course through knowledge checks, homework, etc. that allow for multiple attempts Summative assessment is end of course exam requires application of skills attained during the course; it is MORE than an evaluation of learning objective attainment

45 The Certification Journey in Your Organization Could Be Course & Mini Exam #1 Course & Mini Exam #3 Course & Mini Exam #2 Skill set 1 validated Course & Mini Exam #4 Skill set 2 validated Work Project #1 Work Project #2

46

47

48 Wednesday, April 25 th,

49 Thank You! For more information contact Shreya Sarkar-Barney, Ph.D. Follow us