Extract from Project Report

Transcription

1 AAA Continuation Activity Durchführung von E-Assessment mit ILIAS an der Universität Bern Auszug aus dem Projektbericht E-Assessment ilub Extract from Project Report A Comparison of E-Exams with EvaExam and ILIAS ilub-team: Timon Amstutz, Hansjörg Lauener, Fabian Schmid, Yvonne Seiler Universität Bern Vizerektorat Lehre, ilub Fabrikstrasse 8 CH-3012 Bern

2 Contents 1. Vergleich: E-Klausuren mit EvaExam oder ILIAS Klausurenerstellung und Auswertung mit EvaExam Klausurenserstellung, Durchführung und Auswertung mit ILIAS E-Assessment Vergleich EvaExam und ILIAS: Funktionalitäten Vergleich der Kosten für E-Klausurensystem EvaExam vs. E-Klausurensystem mit ILIAS Fehler! Textmarke nicht definiert. This extract from the internal report compares the two systems EvaExam and ILIAS in tables.

3 1. A Comparison of E-Exams with EvaExam and ILIAS Besides ILIAS, there are support systems for e-exams such as EvaExam at the University of Bern. These software tools provide an alternative to conducting exams with ILIAS. 1.1 creating and evaluating exams with EvaExam. Technical systems such as EvaExam are electronic testing solutions that support the automatic assessment of multiple-choice exams. In the workflow from test creation to archiving, the following steps are required in support of EvaExam: 1. The entry for the lecturers or examiners occurs through individual faculty logins. 2. Lecturers or examiners create the exam paper in EvaExam (online) either directly or through existing exam questions. To achieve this, three preliminary steps in the online tool EvaExam are taken: 1) creating the exam folder, 2) entering test names and 3) creating the exam paper. The actual exam questions are then entered and organized in groups within exam paper that has been created. 3. The exam paper is printed. 4. The exam is conducted traditionally in lecture halls or classrooms with paper and pen. It is important that students meet certain requirements, so that the tests can be easily evaluated (e.g., dark pen, clear responses). 5. After the exam, the papers are scanned and TIFF documents are read into EvaExam. 6. Visual correction (= verification) is in any case necessary, even if only in part, because the examiner must decide on unclear cases. 7. EvaExam automatically creates the results and provides the option of exporting them. The test result is then evaluated and the usual further steps are taken. 8. The completed exam paper is archived as previously. Seite 3/9

4 1.2 Creating, Conducting and Evaluating exams with ILIAS e-assessment Technical systems such as ILIAS and e-assessment tools are electronic testing solutions which support the creation, conduct and evaluation of e-exams. In the workflow from test creation to archiving, the following steps are necessary to support ILIAS: 1. The entry for lecturers and examiners is at an ILIAS installation for e-exams. 2. Lecturers and examiners attend an online-exam course and create test pools in ILIAS (online) either directly or from existing exam questions. 3. The questions from the pools are collated in tests in ILIAS (with the ILIAS Test Tool). 4. The course settings and exam settings are configured and tested. 5. The exam is conducted in an exam room (a pool room with enough computers online). 6. After the exam has been conducted, it is closed. 7. Multiple choice questionss and other closed question types are scored automatically. 8. Free text responses are measured manually. The advantage of the correction of online exams is the readability of the answers. 9. ILIAS automatically compiles the results and provides the option of exporting them. The test results are evaluated and the usual other steps are carried out. 10. The exam paper can be printed and archived as previously. Seite 4/9

5 1.3 A Comparison of EvaExam and ILIAS: Functionalities Software EvaExam E-Assessment EvaExam software is a system of the Electric Paper evaluation systems company, which also sells EvaSys (for online evaluations). EvaExam supports the automated collection and analysis of paper surveys and also provides e-exams ILIAS and SafeExamBrowser E-Assessment ILIAS is an open source system which belongs to the ILIAS Association and is used by the Unibe for the support of teaching. ILIAS has been designed for many uses. Skills acquired during the creation of e-exams can also be used for creating formative tests The Safe Exam Browser (SEB) is a secure browser application for the reliable conduct of online exams. SEB puts a computer in a kiosk mode and temporarily regulates access to system functions, other websites and programs and thus prevents the use of unauthorized resources during an exam. Building of central question libraries Yes Yes Number of question types Limited (single choice, multiple choice, open question, matching question) Largest (single choice, multiple choice, select error/words, ImageMap, Cloze question, Numerical question, define terms, arrangement question, matching question, free text question, file upload, Flash and Java applet) Seite 5/9

6 Adding more question types (developed or commissioned) Not possible. Core developments (in collaboration with ILIAS OS) or plug-in in-house developments. Adjust or optimize question types Not possible. Core developments (in collaboration with ILIAS OS) or plug-in in-house developments. Adjust or optimize test system Not possible. Core developments (in collaboration with ILIAS OS) or plug-in in-house developments. Hardware equipment for software Server space for EvaExam installation. ILIAS server for installation (including backup) and maintenance/updates. hardware equipment for conducting exams None Option a : PC pool room with enough computer workstations Option b : Laptop pool for e- exams Hardware equipment follow-up Scanning stations and virtualized scanning stations. Nothing additional Workflow preparation online, printing the individual examination paper, can be written in lecture halls or classrooms exam, scan exams, vision correction, issuing the results as a PDF report preparation online, PC pool room to organize for exam (Option a ) or lecture theatres and classrooms with equipped laptops (Option b), can write exam online, issuing of results. Seite 6/9

7 IT services resources (estimated for 100 tests) Server Maintenance - 10% Server Maintenance - 25% ilub resources(support, consulting and organization) (estimated for 100 tests) 20% 100% +20% auxiliary assistant Learning curve for lecturers medium. medium. Integrated item analysis (calculation of difficulty and degree of division) to validate appropriate exam questions Yes Must be developed or export the exam and analysis of raw data in Excel, SPSS, etc.. Efficiency gain in creation of questions Initially higher, handling is difficult. Initially higher, optimized by ilub handling. Total efficiency gains for lecturers A slightly higher effort in preparing, printing and scanning - Cost savings through automated correction, especially with mass multiple-choice tests - However, correction of unclear marks to be allowed for A slightly higher effort when creating exam. - Automated correction - Free text answers are legible. - No further corrections of multiple-choice questions. - Extension of tests with other types of question format 1.4 A Comparison of costs for the EvaExam and ILIAS E-Exam Systems Licensing EvaExam E-Assessment Proprietary: annually recurring costs ILIAS and Safe Exam Browser E-Assessment Open Source - Available for free Seite 7/9

8 Usage scenario After introduction lecturers and examiners use the software themselves. Neben Einführungen für Dozierende und Prüfungsverantwortliche sind auch viele zusätzliche Tätigkeiten vor Ort notwendig. Die Dozierenden erhalten Unterstützung vom VRL und den ID. Following introduction, many essential additional activities face the lecturers and examiners. The lecturers receive support from Vice-Rectorate Teaching and ID ilub: Global Support a support service ( helpline ) is necessary. A consulting service (developing good multiple-choice questions and tests) is desirable. A support service (helpline, local technical support) is necessary. A consulting service (developing good multiple-choice questions and tests, selection of the questions from question pools and implementing the test) is necessary. IT services Technical support on hardware/server level Technical support on hardware/server level Setting up exam room, maintenance, updating computers Cost of computer science services (hosting ILIAS E- Assessment) and support services ilub (service, support, consulting services and technical support on site.) Seite 8/9

9 Hardware requirements Server Server Examination room equipped with computers. Question of space - Currently, the University of Bern has no large rooms that can be converted for exam rooms equipped with computers. Hardware costs? Server: 30k? Resources IT services (estimated for 100 exams) Resources ilub (support, consulting and organization) (estimated for 100 tests) Hardware 180 Computers: 200k Operations and technology: 100k Server maintenance - 10% Server maintenance - 10% support, consulting 40% pool room maintenance 10% 100% +20% auxiliary assistant Seite 9/9