Heuristic Evaluation of a Learning Management System

Transcription

1 Heuristic Evaluation of a Learning Management System atrah Abdullah 1, and oor Hafizah Che Ali 2 1 Universiti Teknologi Mara, Shah Alam, Malaysia, natrah@tmsk.uitm.edu.my 2 Universiti Teknologi Mara, Shah Alam, Malaysia, hafizah.cheali@gmail.com ABSTRACT Learning Management System (LMS) is a tool that uses computer's application for teaching and learning in education field. The aim for the system is to offer an efficient, user friendly e-learning environment with online assessment creation tools, collaboration tools, as well as sophisticated tracking and reporting tools for monitoring user activities and progress. However, preliminary study shows that there might be usability problem in the system and respondents from the study reported that they do not familiar with the LMS interface and face few problems to do certain tasks. This study aims to evaluate the usability of the system. This research is conducted using two techniques to collect data which are Heuristic Evaluation and questionnaire survey. Findings revealed that although the LMS is effective, efficient, comprehensive, satisfy and safe, majority of users have problems with the system including performance, access and login issues, and course contents. Recommendations for LMS are derived based on the qualitative and quantitative data from the survey. Keywords: learning management system, heuristic evaluation. I I TRODUCTIO LMS has become popular and widely being accepted as the main educational technology used in universities and colleges around the world (Blecken, Bruggemann, & Marx, 2010). It offers various functions from student registration to managing learning materials. Besides that, it also allows communication with both lecturers and other students through online. These functions offer several benefits: For instance, it helps lecturers to provide their students with learning materials as well as tracking and reporting on students activities and progress. They do not have to meet in person since they can communicate within LMS. However, the active communication activities will not guarantee users acceptance and satisfaction. Without acceptance, users will perform in an inefficient manner and time wastage if too many problems faced. Consequently, users will not be satisfied by using the system. Considering these circumstances, a usability evaluation for LMS had to be conducted in order to find out about users opinions and problems with the offered functionalities. Based on the findings in the evaluation, recommendations for improvement have to be derived to ensure the quality in the future. These suggestions for refinement are also of interest for other e-learning platforms as they are generally applicable to learning management systems. The approach presented in this paper was based on practices at Kolej Profesional MARA (KPM) established in May Initially, KPM only have 100 students for the first batch with 4 lecturers. Currently, there are 6 KPM campuses in Malaysia with 350 lecturers and 4310 students. On 2010, MARA Higher Education Division has ordered all KPM campuses to implement Learning Management System (LMS) at their colleges. The objective of this order is to improve teaching and learning style among lecturers and students at KPM. At this moment, users access the LMS through The users are lecturers and students of KPM. There are 15 features in the LMS. The features are course description, documents, learning path, links, tests, gradebook, announcements, glossary, agenda, forums, user, chat, assignments, surveys and wiki. II LITERATURE REVIEW There are a number of guidelines and principles in previous studies and each of them described in different set of attributes. Most of the guidelines highlight on efficiency, consistency and memorable. Researchers may adopt or adapt any guideline that suit to this research. Integrated Model has been chosen as heuristic or guideline in this research. This model is very generic and can be easily applied to all kinds of software systems because it combines and presents all the attributes and aspects based on the usability attributes suggested by various researchers (Dubey, Gulati, & Rana, 2012). The detail attributes discussed in this model will help researcher in designing questions for survey. There are many definitions used by previous researchers to define the meaning of heuristic technique. Table 1 includes a few definitions from earlier studies. 66

2 Table 1. Heuristic Technique Definition from Different Researchers. Definition HT is a type of usability evaluation where 2 to 3 usability experts inspect an application design using a list of established usability heuristics. HT consists of a small set of expert evaluators, guided by a set of usability guidelines known as heuristics, to determine whether a system follows the guidelines in order to identify certain usability problems in the application. HT is a usability evaluation method and it is guided by a list of heuristic principles. A heuristic is considered as a general rule of thumb, which can be used to guide a decision in designing or to comment on a design that has already been prepared. Heuristic Evaluation (HE) is an inspection evaluation technique that is used by professional to find usability problem in any product or application. HT involves inspector to evaluate the interface of a product to identify the good and bad of its features. Rules or usability guidelines are used as guidance. Researcher Almarashdeh, Sahari & Mat Zin (2011) Ssemugabi & de Villiers (2007) Yen & Bakken (2009) Swindler, Randall & Beisner (2006) Mohamed, Yusoff & Jaafar (2012) Mahrin, Strooper & Carrington (2009) Based on the above definitions, heuristic technique can be summarized as an evaluation or inspection techniques to determine the usability problems in an application using a list of heuristic principle. Normally this technique will be done by evaluators and they might be the users of the application or person who expert in usability field. From the definition given by Almarashdeh et al. (2011), they had stated that 2 to 3 experts should be involved in an evaluation. Therefore, this study follows their definition and will involve suggested number of experts to evaluate the LMS at Kolej Profesional MARA. Table 2 shows the different techniques used in previous study. These studies had focused on Learning Management System. Based on three studies done by previous researchers, they used to apply System Usability Scale (SUS) to evaluate their LMS. SUS is provided with free of charge and proposed as a method specifically for system s quantitative usability requirement, but it is not able to provide reasons that explain the user s perception about the quality of user experience towards LMS. Table 2. Technique Used Technique Used in Previous Studies. Researcher System Usability Scale (SUS) Katsanos, Tselios & Xenos (2012) Heuristic Evaluation Almarashdeh et al. (2011) Thinking Aloud and System Usability Scale (SUS) Blecken et al. (2010) Heuristic Technique also can be applied to evaluate usability on LMS (Almarashdeh et al., 2011). Therefore, the study can be used as the main reference in this research. Quantitative (Heuristic Evaluation data and results) Figure 1. Triangulation Mixed-methods design by Creswell (2008). As shown in Figure 1, the evaluation technique used for the qualitative approach is heuristic technique and it is supported by questionnaires which represent the quantitative approach. III Interpretation Qualitative (Questionnaire Data & Results) DATA A ALYSIS A D RESULT A. Heuristic Evaluation Session The heuristic evaluation session was conducted face to face with HCI experts. Each session only involved one expert. The experts were given a user manual of the task that they need to perform on the LMS. They need to open an Internet browser and go to URL In this evaluation, the experts will play the role as lecturer. There are 15 features or modules in this LMS. The main feature that is always used by lecturer of Kolej Profesional MARA is Document Module. Therefore, the experts only test on the module. In order to evaluate the LMS, experts need to perform five tasks. The tasks are login as lecturer, upload a document, view or download a document, delete a document and logout from lecturer account. 67

3 B. Usability Problem Identified from Heuristic Evaluation. An evaluation usually will presents a list of usability problems identified in an interface (Aitta, Kaleva, & Kortelainen, 2008). After the Heuristic Evaluation has been done with two experts, it was found that the Learning Management System (LMS) which focus on Document Module has five user interface usability issues. The first issue is inappropriate wording and labelling. The wording and icons that are inappropriate to be used are Edit on the table heading, (move icon), (visibility icon) and Standby that represents waiting a file to be uploaded. The second issue is inappropriate contents in the homepage. The contents such is Latest Jadual Bertugas Felo SEPT 2013 is not suitable to be published on the homepage of LMS. The design also not encourage user to use the LMS. The following figure shows the contents in the homepage of LMS.The third issue identified is inappropriate positioning of icons and feedback message. The position of these icons are not suitable because it s position is quite far from the filename. User might confuse the pairing of filename and it s icons. Besides that, the feedback message which inform the successful of uploaded file also need to be repositioned because it is located out of user sight. The fourth problem is redundant function. There are two delete functions that have been identified. The first delete function using icon and the other one using. Users might confuse which delete function they should use if they want to delete a file. Lastly, there is less important item that has been published in LMS. The item that should not be given priority to be displayed in the Document Module page is file size. It consumed the space to display icons. C. Recommended Modification from Heuristic Evaluation. In order to improve the usability of the Learning Management System, some modification should be made. The recommended modifications are summarized according to the LMS page. Table 3. Page Homepage Course Modules Document Module Upload Document Page Recommended Modification by Experts. Recommended Modification Change the contents of homepage. The contents should related to learning. Refer to other LMS to make the homepage more attractive. The visibility icon ( ) should be changed. Use a familiar icon or replace with text. The visibility icon ( ) and move icon ( ) should be changed. Use a familiar icon or replace with text. The position of these icons should be near to filename. Use only one delete function instead of two. Remove or hide the file size. The feedback message should be positioned in sight. Usually at the center of a page. Change the Standby which presents waiting file to be uploaded to Uploading Table 3 shows summarization of expert recommendation towards LMS. The pages that need to be improved are homepage, course modules, document module and upload document page. Most of the recommendations are related to icons. The experts felt that the some icons used in LMS are not familiar and should be change to other icons or text. The positioning of icon and feedback message also not suitable and should be re-position to suitable location. Figure 2. Problematic Homepage Contents. D. Survey Evaluation The population for survey method included all students and instructors from Kolej Profesional MARA. There were 4310 students and 350 lecturers. The data collection period was from October 18, 2013, to November 18, 2013, for a total collection time of four weeks. There were

4 students and 185 lecturers who responded the survey. Of 372 students, 350 completed the survey for a response rate of percent. For instructors, all respondents completed the survey for a response rate of 100 percent. Although two sets of survey designed for student and lecturer, the difference between the surveys only on demographic information. Therefore, this research presents separate demographic information for student and lecturer. While for the rest questions, this research will combine the findings of students and lecturers. The total question that has been asked in both surveys is 45 questions. The survey has divided to two section which are demographic section and usability section. The 11 questions are asked in the demographic section while 34 questions on the usability part. For the usability part, it has divided to another five elements that has been introduced by Dubey (2012) in Integrated Model Guideline. The elements include effectiveness, efficiency, satisfaction, comprehensibility and safety. 9 questions are asked in the effectiveness and satisfaction part, 8 questions comprehensibility, 4 questions on efficiency and 2 questions on safety. The other 2 questions are open ended question that are asked regarding comment and recommendation for the improvement of LMS. Likert Scale of five point scale is used to allow the participant to express how much they agree or disagree with a particular statement on usability section. E. Findings on Student Demographic Information The validity and reliability have been tested to 32 usability questions. The test has been done using Cronbach Alpha. Cronbach Alpha is used to measure the consistency of a test or scale. It is represented as a number between 0 and 1 (Tavakol & Dennick, 2011). Table 4. Reliability Statistic Using Cronbach s Alpha. Element of Cronbach s Items Alpha Effectiveness Efficiency Satisfaction Comprehensibility Safety The Cronbach s Alpha value were considered to represent an acceptable level of internal consistency if value within 0.5 to 0.7 and good level if the Cronbach s Alpha value more than 0.7 (Yusoff, 2012). Referring to Table 4, all element results the value within 0.5 and above except for the efficiency element while the satisfaction element achieved good level of Cronbach s Alpha. Cronbach s Alpha value is also affected by reverse score items. For example researcher had two items that were agreed by respondents while two items were not agreed by respondents. The item which not agreed by respondents are The LMS gives a good performance e.g. fast respond and The LMS page is successfully downloaded within 10 seconds. Most of respondents rated these questions as Strongly Disagree and Disagree because they did not agree with the items. In order to get high Cronbach s Alpha, the item should be agreed as Question 1 and Question 2. These items should be phrased reversely to be rated agree. These reverse phrased items are important to decrease response bias. In reliability analysis, these reverse scored items will affected it s result: in the extreme, it may lead to negative Cronbach s Alpha (StatSoft, 2011). F. Descriptive Analysis. The following table, Table 5 shows the statistical result calculated in SPSS. The analysis only focuses on effective, efficiency, comprehensibility, satisfaction and safety items in the survey. These items are evaluated in 5-point scale ranging Strongly Disagree, Disagree, Neutral, Agree and Strongly Agree. For data recorded in SPSS, the Strongly Disagree recorded as 1, Disagree recorded as 2, Neutral recorded as 3, Agree recorded as 4 and Strongly Agree recorded as 5. The total data recorded from respondents are 535 (350 students and 185 lecturers). The following table shows the result of descriptive statistics for the above sample. Table 5. Descriptive Statistics. Element Mean Std. Deviation Effectiveness Efficiency Satisfaction Comprehensibility Safety By referring to Table 5, the average effectiveness rated by respondents was 3.10 (SD=0.38), average of efficiency was 3.46 (SD=0.38), average of satisfaction 3.57 (SD=0.39), average of comprehensibility 3.49 (SD=0.28) and average of safety 4.00 (SD=0.48). From the results, it shows 69

5 that most of respondents agree with the items in the safety element while for other elements, they almost agree with the items. Table 6. Coefficient and P-Value. Element Unstandardized Sig. (P- Coefficient Value) Effectiveness Efficiency Satisfaction Comprehensibility Safety.06 Unable to be computed because the fit is perfect In order to analyse the effectiveness, efficiency, comprehensibility, satisfaction and safety of the LMS, Linear Regression Analysis is used to get the result. From the analysis, statistical significance or p-value will indicate the degree to which the result is true. The p-value between 0.01 P 0.05 consider as statistically significant and below than is highly significance. By referring to Table 5 and Table 6, researcher had found that the model of effectiveness, efficiency, comprehensibility and satisfaction are highly significant because its p-value is which is smaller than However, based on the sample safety model was unable to be computed because the fit is perfect. According to respondent, fit is perfect means that the model is very significant and contain no error on the data. Element Learning Manageme nt System Table 7. Findings On The Effectiveness, Efficiency, Comprehensibility, Satisfaction And Safety Of LMS. Efficient Comprehensibility Effective Satisfaction Safety In the Table 7, it shows that the Learning Management System used at Kolej Profesional MARA is effective, efficient, comprehensive, satisfy and safe. G. Recommendation from User The last two questions in the survey ask regarding the comment, recommendation and features to be improved in the LMS. Table 8. Comment, Recommendation and Features to be Improved in the LMS. Comment / Recommendation Upgrade the performance Resolve Internet connection to improve access or login issues Do not host the system in the college's server Make the LMS can be access without redirect through KPMIM website. Features to be Improved Performance, server Login or access issue Performance & connectivity connectivity The above Table 8 shows data regarding the comment, recommendation and features to be improved in LMS. Most of respondents are not the satisfied with the performance in LMS. In demographic information, the performance issue also has been rated in high percentages which are 100% from lecturers and 83.14% from students. Therefore, this issue shouldn t be ignored by Administrator to make the LMS more usable. IV CO CLUSIO In this study, two research objectives had been addressed. The first objective is to evaluate the usability of the Learning Management System (LMS) at Kolej Profesional MARA using Heuristic Evaluation. Another objective is to suggest the improvements for Learning Management System (LMS). From the usability evaluation conducted through Heuristic Evaluation sessions, five usability problems were identified including inappropriate wording and labelling, inappropriate contents in the homepage, inappropriate positioning of icons and feedback message, redundant function and less important item has been published. While in the survey findings, the most identified issue in the LMS is performance, followed by access and login issues and course contents. Thus, for the improvement of the system, it is recommended to the developer to replace the contents of homepage. The contents should relate to learning. The developer may refer to other LMS as a guideline. The visibility icon ( ) also should be replaced. Experts suggested using a familiar icon or replacing with text. Besides that, the position of these icons should be placed besides the filename. Developer also should remove one delete function in Document Module page. Another element that needs to remove or hide is the file size in Document Module page. The feedback message 70

6 should be positioned at the center of a page. The developer need to replace the Standby which represents waiting file to be uploaded to Uploading. From the expert suggestion the header label Edit need to be replaced to other meaningful or related word. From user s opinion, they recommended to improve the performance of LMS (in terms of speed), find other alternative to host the LMS and resolve Internet connection to improve access or login issues. From the Regression Analysis that has been done, researcher found that the LMS is effective, efficient, comprehensive, satisfy and safe. REFERE CES Aitta, M.-R., Kaleva, S., & Kortelainen, T. (2008). Heuristic evaluation applied to library web services. ew Library World, 109(1/2), Almarashdeh, I. A., Sahari, N., & Mat Zin, N. A. (2011). Heuristic evaluation of distance learning management system interface. Paper presented at the Electrical Engineering and Informatics (ICEEI), 2011 International Conference on Electrical Engineering and Informatics. Blecken, A., Bruggemann, D., & Marx, W. (2010). Usability Evaluation of a Learning Management System. Paper presented at the Proceedings of the rd Hawaii International Conference on System Sciences. Dubey, S. K., Gulati, A., & Rana, A. (2012). Integrated Model for Software Usability. International Journal on Computer Science & Engineering, 4(3), Katsanos, C., Tselios, N., & Xenos, M. (2012). Perceived Usability Evaluation of Learning Management Systems: A First Step towards Standardization of the System Usability Scale in Greek. Paper presented at the Informatics (PCI), th Panhellenic Conference on. Mahrin, M. N. r., Strooper, P., & Carrington, D. (2009). Selecting Usability Evaluation Methods for Software Process Descriptions. Paper presented at the Software Engineering Conference, APSEC'09. Asia-Pacific. Mohamed-Omar, H., Yusoff, R., & Jaafar, A. (2012). Quantitive analysis in a heuristic evaluation for usability of Educational Computer Game (UsaECG). Paper presented at the Information Retrieval & Knowledge Management (CAMP), 2012 International Conference on. Ssemugabi, S., & de Villiers, R. (2007). A comparative study of two usability evaluation methods using a web-based e-learning application. Paper presented at the Proceedings of the 2007 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countries. StatSoft Inc. (2011). Electronic Statistics Textbook. from WEB: Swindler, S. D., Randall, C. E., & Beisner Ii, G. (2006). An Application of Heuristic Techniques for Evaluation of Collaborative System Interfaces. Paper presented at the CTS. Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach's alpha. International Jurnal of Medical Education, 2, Yen, P.-Y., & Bakken, S. (2009). A comparison of usability evaluation methods: heuristic evaluation versus end-user think-aloud protocolâ an example from a web-based communication tool for nurse scheduling. Paper presented at the AMIA Annual Symposium Proceedings. Yusoff, M. S. B. (2012). Stability of DREEM in a Sample of Medical Students: A Prospective Study. Education Research International,