Cloud-Based Certification. Gary Fluitt - Google

Transcription

1 Cloud-Based Certification Gary Fluitt - Google

2 Google Admin in natural habitat Goal OS independent Learns online Google s for answers Free No IT certifications Mobile

3 Google s Goals Internet Scale Google Cloud Platform Great User Experience Automated Low Cost

4 <launch_demo>

5 What I learned 1. Performance based exams require lots of QA 2. Google Cloud Platform as an application host rocks ($11 beta) 3. WebRTC is a no-install platform-independent proctor option 4. Integration with legacy certification systems is rough 5. Use your partners to help you develop and write

6 May June July Aug Sept Oct Nov Dec Jan Analysis Blue Print Customer Interviews SME Interviews Content Review BRD 2-Page Business Requirements Doc Legal Item Writing Prototype App Dev Internal Testers Alpha Trusted Partners 2nd Alpha Customers Beta Standard Setting

7 Next Steps 1. Automate the integrations (SalesForce, Kryterion) 2. Localization into 4 languages 3. More forms 4. Linear problem delivery 5. Expand platform to other programs

8 Cloud-Based Certification Gary Fluitt - Google

9 Google Admin in natural habitat Goal OS independent Learns online Google s for answers Free Mobile No IT certifications When Google began the effort of designing training and certification for Google Apps Admins we found that the Google Apps Admin candidate profile is different than our typical IT professional of the recent years. These admins are often not from a hard core IT background (accidental admin), they don t hold multiple certifications from existing platform vendors. In fact they commonly don t have any other certifications. They almost never attend ILT training. They use search for their answers and they spend a lot of time in forums and communities. They don t remember the last time they cracked a manual open to look up an answer. They are very mobile, doing some of their admin tasks via mobile phone. They are somewhat operating system agnostic (their applications are in the cloud). They are accustomed to getting software for free but paying for service.

10 Google s Goals Internet Scale Google Cloud Platform Great User Experience Automated Low Cost After completing a job task analysis and an exam blueprint, we decided that a hands-on performance exam would lend itself well to the exam objectives. Requirements included; scale to many thousands of exams per year, had to be almost fully automated and hands-off, it had to be have a low cost of per-exam delivery, and it had to be a great user experience. Running the exam on Google s Cloud Platform was also a goal, though not a requirement.

11 <launch_demo> Demo the Examinator app. The result was an application hosted on the Google App Engine that automates the creation of the testing domains, controls the delivery of the exam, watches the user take the exam using WebRTC, scores the exam by using API calls to the Apps domain, and reports results to a records mgt. system that handles certificate fulfillment. What the app doesn t do is fee collection and certification logic. The application took about 6 weeks to develop to a prototype stage, another 8 weeks of iteration through beta. One contract developer (not a Googler) was used for the app development. A detailed BRD described the user experience, process, and reporting requirements.

12 What I learned 1. Performance based exams require lots of QA 2. Google Cloud Platform as an application host rocks ($11 beta) 3. WebRTC is a no-install platform-independent proctor option 4. Integration with legacy certification systems is rough 5. Use your partners to help you develop and write A performance exam of this type (non-sim) is a software development project. Make sure include lots of QA. Hosting the app on publically available infrastructure reduces the cost of delivery. Cost to beta 200 exams was under $11 Web Real-time communications (webrtc) is a standard released in 2013 that has support from Firefox and Chrome on all OS s. Works great for pseudoproctoring. Can be used to fully proctor including browser lockdown in Chrome. Integration of registration and records management systems is no minor task, but can be done post launch, in iterations if necessary. Hard to get the time of an internal SME. Look to your partner community. We had an awesome response from our partners to help write the exam scenarios and test out the prototypes.

13 May June July Aug Sept Oct Nov Dec Jan Analysis Blue Print Customer Interviews SME Interviews Content Review BRD 2-Page Business Requirements Doc Legal Item Writing Prototype App Dev Internal Testers Alpha Trusted Partners 2nd Alpha Customers Beta Standard Setting This is the rough schedule we arrived at. Note that analysis was the longest running activity. Recommend more time for beta deliveries and iteration.

14 Next Steps 1. Automate the integrations (SalesForce, Kryterion) 2. Localization into 4 languages 3. More forms 4. Linear problem delivery 5. Expand platform to other programs Next steps for the program are to integrate the data flow with other applications (e.g. Webassessor, Salesforce, etc.) Localize the UI and scenarios, 4 languages to start. Develop additional forms of the exam, goal of ten forms, with quarterly updates Deliver problems in a linear display with some UI to allow for back and forth Expand the application for use with other certifications