Enterprise Knowledge Platform Training History Import/Export
Document Information Document ID: EN145 Document title: EKP Training History Import/Export Version: 1.7 Document date: 14 April 2009 This document may be revised from time to time. Please check NetDimensions Support site at www.netdimensions.com/support for updates to this and other documents or send an e-mail to support@netdimensions.com to request the most recent version. Please report any errors or feedback with this document by sending an e-mail to support@netdimensions.com. Copyright Information Copyright 2000-2009 by NetDimensions Ltd. All Rights Reserved. Information in this document is subject to change without notice. The software described herein is furnished under a license agreement, and it may be copied only in accordance with the terms of that agreement. No part of this publication may be reproduced, transmitted, or translated in any form or by any means without the prior written permission of NetDimensions Ltd. All company and product names used herein may be trademarks or registered trademarks of their respective companies unless stated otherwise. How to Contact NetDimensions Support +852 2122 4588 1 866 206 6698 US toll-free number +852 2122 4588 support@netdimensions.com www.netdimensions.com/support General Enquiries +852 2122 4500 +852 2122 4588 info@netdimensions.com www.netdimensions.com
Table of Contents Description...1 About This Guide... 1 If You Need Help... 1 Scheduling Import/Export Tasks...2 Web Services... 2 Scheduled Events... 3 Import Option 1: Import Data as External Training History (Utility)...5 Overview... 5 Data Format... 6 Import Option 2: Import Data as EKP Training History (CSV)...8 Overview... 8 Data Format... 8 Export Option 1: Export Training History (XML)... 11 Overview... 11 Data Format... 11 Transcript Historical Data Loading Speed and Option Improvements... 10
Description About This Guide This document describes the mechanism by which training history data may be imported from another system into EKP, as well as EKP training history data exported from EKP for use by another system. These actions may be invoked in one of two way: on demand via a Web Services API, or automatically using the EKP scheduler these two methods are explained in a later section The principal data exchange format for EKP is generally XML, but in some cases alternative methods allow flat data files (or CSV, Comma Separated Values) to be used where it is simpler to manipulate data in that form. Where there are choices, they are listed in the configuration options described later in this document. Scheduling the automatic, periodic import and export of data into EKP is managed via the EKP Event Scheduler, which routinely runs tasks at hourly, daily, weekly, and monthly intervals. Training History Import Choices Training history from non-ekp systems may be imported into EKP using one of two methods. Which one is best depends upon the intended long-term use of the training history information, and management objectives for capturing this data. The two methods are: Import as External Training History Import as EKP defined history (available in EKP Gold only) These two methods are explained in this document. If You Need Help If you cannot resolve a configuration problem using this guide or the online help, or if you should have any queries related to the technology employed within EKP, your first line of contact should be as described in your Technical Support Contract. For other queries, or if you are not sure whom to contact, NetDimensions Ltd. may be contacted at info@netdimensions.com. Please also refer to the support section of the NetDimensions web site at www.netdimensions.com for the latest information regarding various services. 1
Scheduling Import/Export Tasks Web Services EKP provides a number of web services that can be used by other applications to query EKP for data. These web services are designed according to the REST (Representational State Transfer) architecture 1. In each case, a data-oriented document (typically XML) is sent in response to an HTTP request. Note The Web Services APIs are available in EKP Gold only. This document focuses on the manipulation of training history data, but for more information about other import/export functions, please refer to the document EKP APIs and Web Services Overview (EN049_EKP_APIs_and_Web_Services_Overview.pdf). You may use this API on an on-demand basis, as well as part of machine-to-machine communications that are automated via scripts or other processes. To make use of these services: Step 1 Authentication Setup Configure a password in ekp.properties like this: authentication.key=mysecretpassword This is not linked to a standard user account, since this is designed primarily for integration with other systems, not for direct access by users. When one of the APIs is invoked manually, the system will prompt for the authentication password to be entered. Step 2 Invoke required URI To retrieve a document via the web services API, you would enter a request containing the URI defined below, relative to the site in question. So, for example, if EKP was running on a host named ekp.example.com, using the default port for HTTP (port 80), with the default context path of /ekp, the URI to retrieve an XML representation of the student with user ID joestudent would be as shown below. http://ekp.example.com/ekp/contentgenerator/userxml?userid=joestudent An example to retrieve training history data: http://mysite.com/ekp/contentgenerator/traininghistoryxml where traininghistoryxml is listed in the chart below as the URI for training history data.. 1 See http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm. 2
For a full list of the services provided, please refer to the EKP APIs and Web Services Overview document. Scheduled Events EKP has an event manager that schedules background tasks at predefined points, typically: Hourly; Daily; Weekly; Monthly; The execution time for the daily, weekly, monthly tasks are determined by the following ekp.properties setting: system.handlerstarttime=3 This setting indicates that the daily processes are run at 3:00am, which is also the EKP default task startup time. Weekly and monthly tasks are run after the daily tasks at the same time, with weekly events run on Sunday, and Monthly events run on the 1 st of the month. To invoke the import/export tasks for training history, there are a series of configuration files in WEB-INF/conf for the different intervals: imports-hourly.opml imports-daily.opml imports-weekly.opml imports-monthly.opml exports-hourly.opml exports-daily.opml exports-weekly.opml exports-monthly.opml Note Scheduled imports and exports are available in EKP Gold only. The contents of an opml configuration file appear as follows: <opml version="1.0"> <head/> <body> <outline type="traininghistorycsv" xmlurl="file:/c:/path/to/file.csv" /> </body> 3
</opml> The URL can point to a directory instead of a specific file when the file: scheme is used. In this case, the importer will scan the directory for all files that do not have the file name extension.imported; every matching file will be processed and then renamed to add the.imported extension to the file name. Target location would be specified as a URL, with the following schemes supported: - file: (exported data is written to a file on the local file system) - ftp: (exported data is uploaded to a file on an FTP server) - http:/https: (exported data is submitted using HTTP POST) A sample file might look like this: <opml version="1.0"> <head/> <body> <outline type="users200510xml" xmlurl="file:/c:/temp/out/users.xml"/> <outline type="traininghistoryxml" xmlurl="ftp://uname:pwd@ftp.abc.com/data/traininghistory.xml"/> <outline type="csuserscsv" xmlurl="https://competencysuite.abc.com/cgi/userimporter.asp"/> </body> </opml> It is also possible to trigger imports on-demand using a web services API. Both the scheduler mechanism and the web services APIs are described in more detail in the document EKP APIs and Web Services Overview (EN049_EKP_APIs_and_Web_Services_Overview.pdf). 4
Import Option 1: Import Data as External Training History (Utility) Overview Each user s transcript shows information for both EKP-defined courses as well as for externally defined historical records. The difference between these two is that the externally defined history has no corresponding course or tracking data defined in EKP, so only basic information from the external history may be captured and reported. This is the cleanest and simplest way to store historical data from another system. This external training history for a specific user may be reviewed in their training history, or a report on many users may be created using the Report Wizard. Important Note This import technique currently based on an external database utility mechanism, and so is not available as an EKP Web Service or scheduled task. Summary of Use Within EKP Only the external_training database table is updated; Imported data appears to users and management in the Training History screen labeled as External Training History ; Reporting must be done using the Report Wizard, as most canned reports are focused on EKP tracking details and so do not reference the external training history data. 5
Data Format To perform a bulk import of external training data, a file containing the following fields is prepared and then imported using the relevant external data loader associated with the database (for Microsoft and Sybase this is bcp, hence this is used in the example below). RecordID nvarchar(15) not null UserID nvarchar(85) not null Title nvarchar(255) not null Startdate datetime null Enddate datetime null Grade nvarchar(30) null Score nvarchar(30) null Comments nvarchar(1024) null training_hours float default 0, not null course_type nvarchar(100) null subject nvarchar(100) null duration float default 0, not null duration_unit nvarchar(50) null language int default 0, not null cost numeric(15,2) default 0, not null currency nvarchar(15) null vendor_info1 nvarchar(255) null vendor_info2 nvarchar(255) null other_info nvarchar(255) null venue nvarchar(512) null studentstatus int default 0, not null Example Command Statement for Microsoft SQL Server: bcp external_training in c:\history.dat c t; -Undadmin -Pndadmin where: - external_training is the name of the database table; - in implies import (as opposed to out for export) - c:\history.dat is the name of the data file; - -c implies that this is an ascii data file; - -t; specifies a semi-colon as the delimiter - -Undadmin is the userid (which should point to the EKP database) - -Pndadmin is the password to the database Please refer to the database utility reference manual for additional details regarding the use of the utility, as you have some flexibility as to the options to present on the command line. The example command as shown is adequate for most scenarios. 6
Example Import Data File The following file illustrates how this information may be packaged in a file: 1; ray;project Management Workshop;2008-03-04;2008-03- 04;;Passed;Completed;0.0;;;0.0;;1;0.00;;;;;;4 2; rob;java Security;2008-06-16;2008-06-16;;Passed;Completed;0.0;;;0.0;;1;0.00;;;;;;4 3; cathy;courseware Standards;2008-07-18;2008-07-18;;PASS;Completed;0.0;;;0.0;;1;0.00;;;;;;4 Note 1. It is necessary to enter values for the "default 0 not null" and "not null" columns - at least 0 for int,0.0 for float and 0.00, for the numeric (15,2) values. Empty fields lead to an error during the import, at least with the bcp utility.when running the data import utility, you may specify the delimiter to be used between the fields (e.g. tab, comma or semi-colon). 2. In this file, the recordid in the first column must be a unique identifier not already in the external_training table. This is a requirement of the bcp process- 7
Import Option 2: Import Data as EKP Training History (CSV) Overview A training history importer is built into EKP that is also capable of storing imported data as an EKP transcript and course data. The advantage of doing this is that more reporting options are available to administrators. The disadvantage is that the Catalog Editor now contains more dummy courses that are tied to the users new transcript entries. Note The training history importer described in this section is available in EKP Gold only. Summary of Use Within EKP Multiple tables are updated; these include transcript, learningobject, course, and cschedule. This import process creates course entries similar to that done directly by an administrator using the Catalog Editor. Data appears to user and management in the Training History screen labeled as normal EKP data; nothing identifies it as historical data different from standard EKP training history data. Reporting may be done using both the Report Wizard and canned reports; This import process may be scheduled using the EKP Event Scheduler, or invoked on-demand using a web services API these mechanisms are described in detail in the document EKP APIs and Web Services Overview (EN049_EKP_APIs_and_Web_Services_Overview.pdf). Configuration includes the location of the data file to be imported; this could be the path of a file on the local file system, or a URL (allowing import from FTP servers and web services for example). The schedule for import is also configurable and could be hourly, daily, weekly or monthly. In addition, the import can be triggered on-demand using a web services API. Data Format The importer reads data in CSV format. The first row in the CSV file will be a header row; each subsequent row will represent one training record (or, if no user ID is specified, a course session with no associated training records). Each non-header row in the CSV file will contain the columns specified in the table below. (N.B. The order of the columns is not significant; the order can be changed provided the order of the column headers is changed accordingly.) 8
CSV Column Name USERID LEARNINGID LOTITLE STARTDATE ENDDATE STUDENTSTATUS CREATESESSION Remarks Corresponds to EKP s User ID; if blank, a classroom and session will be created with no associated training record. Corresponds with EKP s Learning ID; a classroom and session will be created even if there is no USERID tagged to this code Corresponds with EKP s Learning Module Title; a classroom and session will be created even there is no USERID tagged to this name The format of this field in the CSV file will be DD-MM-YYYY The format of this field in the CSV file will be DD-MM-YYYY If the value is Pass, the Overall Status of the training record will be set to COMPLETED; otherwise, the Overall Status of the training record will be set to INCOMPLETE Determines what happens if there is no existing session for the specified module with the specified start and end dates. If the value is true then a new session will be added, otherwise any existing session(s) will have their start and end dates updated. If this column is omitted, the default behavior is to update the start and end dates of existing sessions. Note that if the module currently has no sessions, then a new session is always created regardless of the value of this column. This setting applies only to modules of types Classroom, Virtual Classroom and Workshop/Seminar. Notes: If there is no existing learning module with the specified Learning ID, a new learning module of type Classroom will be created with the specified ID and title; otherwise, the title field of the existing learning module will be updated. 9
If the learning module created or updated in the step above has no existing sessions, a new session will be created with the specified start and end dates; otherwise, the behavior will depend on the value of the CREATESESSION column. - If the value of the CREATESESSION column is true, a new session will be added. - Otherwise, the existing session(s) will have their start and end dates updated. (N.B. This is the default behavior if the CREATESESSION column is omitted.) The importer will then check whether the student with the specified ID currently has any training records associated with the learning module created or updated in the first step. If so, these records will be updated with the specified start date, end date and Overall Status (COMPLETED if the STUDENTSTATUS field contains the value Pass ; INCOMPLETE otherwise). If not, a new training record will be created with the specified start date, end date and Overall Status. Example Import Data File The following file illustrates how this information may be packaged in a file: USERID,LEARNINGID,LOTITLE,STARTDATE,ENDDATE,STUDENTSTATUS,CREATESESSION ray,prj101,project Management Workshop,20-01-2004,23-01-2004,Pass,true rob,jav188,java Security,31-06-2003,02-07-2003,Pass,true cathy,crs77,courseware Standards,01-12-20003,01-12-2003,Fail,true in this file, the first line identifies the fields contained in the file. Transcript Historical Data Loading Speed and Option Improvements In the OPML batch scheduler, a new loader type of turbotraininghistory has been added that enables training records from an external system to be loaded into EKP. This loader has the same file format as the previous traininghistory loader (which is still supported), with the following two improvements: A new LOTYPE column allows an entry to be designated ONLINE or CLASSROOM to indicate the type of record represented. The load data rate is now many, many times better (approximately 8000 records per minute on test system). 10
Export Option 1: Export Training History (XML) Overview Training History data may be extracted from the EKP database and written to an external file in XML format. This data may then be fed into other systems, facilitating data warehouse loading or HR systems personnel reporting. Note The Training History data export mechanisms described in this section are available in EKP Gold only. To export all student records: Web Services URI: Scheduler OPML type label: contentgenerator/traininghistoryxml traininghistoryxml To export student records for completed activities only: Web Services URI: Scheduler OPML type label: contentgenerator/completedtrainingrecordsxml completedtrainingrecordsxml Data Format The data fields included in each <trainingrecord> entry of the training history export are: Id - an EKP internal identifier for the record. UserId - The learner ID; LearningModule id - Course ID; title - Course title; enrolldate - Learner enroll date; overallstatus - Final status; 11
lastattempdate - Last attempt date; attemptcount -# times launched; isselfenrolled - true/false; A sample exported file fragment is: <?xml version="1.0" encoding="utf-8" standalone="yes"?> <traininghistory xmlns="http://www.netdimensions.com/2005/10/traininghistory"> <datecreated>2005-10-21t19:34:07.371+08:00</datecreated> <trainingrecord id="ekp000073058"> <userid>looslesr</userid> <learningmodule> <id>ek-2163</id> <title>it Project + : A CompTIA Certification</title> </learningmodule> <enrollmentdate>2005-10-21t13:40:32.240+08:00</enrollmentdate> <overallstatus>enrolled</overallstatus> <lastattemptdate>2005-10-21t13:40:32.240+08:00</lastattemptdate> <attemptcount>0</attemptcount> <isselfenrolled>true</isselfenrolled> </trainingrecord> </traininghistory> The XSD file which describes this format is: <?xml version="1.0" encoding="utf-8"?> <xsd:schema targetnamespace="http://www.netdimensions.com/2005/10/traininghistory" xmlns:xsd="http://www.w3.org/2001/xmlschema" xmlns="http://www.netdimensions.com/2005/10/traininghistory" elementformdefault="qualified"> <xsd:element name="traininghistory"> <xsd:complextype> <xsd:sequence> <xsd:element name="datecreated" type="xsd:datetime"/> <xsd:element name="trainingrecord" type="trainingrecord" minoccurs="0" maxoccurs="unbounded"/> </xsd:sequence> </xsd:complextype> </xsd:element> <xsd:complextype name="trainingrecord"> <xsd:sequence> <xsd:element name="userid" type="xsd:string"/> <xsd:element name="learningmodule" type="learningmodule"/> <xsd:element name="enrollmentdate" type="xsd:datetime" minoccurs="0"/> <xsd:element name="startdate" type="xsd:datetime" 12
minoccurs="0"/> <xsd:element name="enddate" type="xsd:datetime" minoccurs="0"/> xsd:element name="overallstatus" type="overallstatus"/> <xsd:element name="score" type="xsd:decimal" minoccurs="0"/> <xsd:element name="lastattemptdate" type="xsd:datetime" minoccurs="0"/> <xsd:element name="attemptcount" type="xsd:nonnegativeinteger"/> <xsd:element name="isselfenrolled" type="xsd:boolean"/> </xsd:sequence> <xsd:attribute name="id" use="required" type="xsd:id"/> </xsd:complextype> <xsd:complextype name="learningmodule"> <xsd:sequence> <xsd:element name="id" type="xsd:string"/> <xsd:element name="title" type="xsd:string"/> </xsd:sequence> </xsd:complextype> <xsd:simpletype name="overallstatus"> <xsd:restriction base="xsd:ncname"> <xsd:enumeration value="administrativeentry"/> <xsd:enumeration value="cancelled"/> <xsd:enumeration value="completed"/> <xsd:enumeration value="enrolled"/> <xsd:enumeration value="finishedusing"/> <xsd:enumeration value="incomplete"/> <xsd:enumeration value="pendingapproval"/> <xsd:enumeration value="resource"/> <xsd:enumeration value="shared"/> <xsd:enumeration value="unknown"/> <xsd:enumeration value="waitlisted"/> <xsd:enumeration value="withdrawn"/> </xsd:restriction> </xsd:simpletype> </xsd:schema> 13