Lecture Encoding of Distance Education by Multimedia Integration

Transcription

1 Journal of Engineering Education Research Vol. 17, No. 3, pp. 34~41, May 2014 Lecture Encoding of Distance Education by Multimedia Integration Wouseok Jou *, *Dept. of Computer Engineering, Myongji University 멀티미디어통합에의한원격교육강의녹화주우석 *, * 명지대학교컴퓨터공학과 ABSTRACT In distance education, use of proper software tools can greatly enhance student s attention and learning efficiency. In such software tools, offering diverse multimedia information is one of the most critical factors. However, integration and synchronization of the various media types has been relatively difficult parts of implementation. This paper proposes a prototype system that uses a metafile and event handling mechanism for the uniform treatment of various media types. This event-level integration and synchronization of multimedia makes the implementation relatively simple. With this approach, instructor s behaviors are automatically recorded, and the instructors can freely choose and show any type of multimedia contents while lecturing. Current commercial or non-commercial lecture management systems could incorporate this approach, so that the distance education market could be expanded with richer multimedia contents. Keywords: remote education, e-learning, distance education, multimedia synchronization, event handling, Lecture Management System I. INTRODUCTION 1) The purpose of this paper is to propose a prototype software that provides a convenient multimedia integration tool for distance education. In general, the most distinctive feature of the distance education is that it allows us to study anywhere and anytime. While allowing ample opportunities to study, the distance education is also concerned with increasing the quality of the study(1). Due to the high-speed networks currently available, diverse multimedia contents can be used to support distance education or e-learning. As a result, various research efforts have been made on using multimedia in distance education(1, 2). It is said that humans concentration lasts only for a short period of time. In this respect, communications based on the multimedia can continuously awaken human concentration, Received 17 April, 2014; Revised 17 April, 2014 Accepted 28 May, 2014 Corresponding Author: red@mju.ac.kr and thus it is widely used in distance education to increase learning efficiency. For instance, in distance lecture, usage of both animations and texts greatly enhances learning efficiency compared with that of texts only(3, 4). As a result, the usage of multimedia can be said to be mandatory in distance education(5). Among multimedia, the video recording of instructor's lecture scene requires large quantity of data. For this reason, most commercial software uses video streaming to reduce download time. Since the compression schemes are improving and processor speeds are climbing, current streaming software is getting better than ever. Windows Media Encoder is one of the most popular streaming tools(6). However, it is only one part of the multimedia lecture. Other kinds of media must be integrated to the video stream. Combining these media with the video stream in a uniform file format is inherently difficult. Media integration requires precise control of the timing between different types of media(7, 8). As a result, current commercial authoring tools simply show 34

2 Lecture Encoding of Distance Education by Multimedia Integration only the streamed video, or they show slides embedded with animation or pictures which are separate from and asynchronous with the streamed video(9, 10). The SMIL(Synchronized Multimedia Integrated Language) is an XML-based language recommended by W3C(11). It synchronizes multimedia by allowing us to link the location of each media file through metafile tags. Application of SMIL to multimedia user interface can be found in(12, 13), and(14). In these systems, however, individual media element must go through entire production process. In other words, instructors must capture, edit, compress, catalog the media clips, and edit a SMIL presentation. In this paper, we will show that by recording multimedia in event level, individual media element can be integrated and treated uniformly. With this approach, we can produce completely synchronized multimedia that requires no production process. The instructors can just keep lecturing, generating any kind of events requiring multimedia. The sequence of events is automatically recorded into a file called metafile. Accordingly, decoder can easily interpret the metafile and reproduce the recorded sequential events with complete media synchronization. screen. If we ignore the top menu bar for the moment, the encoding window is partitioned into four viewport areas; display viewport, data viewport, slide viewport, and text viewport. Table 2 shows media contents that fill each viewport area. The display viewport is mainly used to show the instructor's lecture scene. It is important since the instructor's facial expression or gesture can greatly help student's understanding of the lecture. The data viewport is composed of three Table 1 Supported Media Media Name Media Type File Type Lecture Video Video Windows Media File Slides Text and Graphic Windows MetaFile Scripts Graphic Stroke Binary File Texts Text Text File Links URL Text File References Application Execution File II. THE ENCODER INTERFACE 1. Media Types Table 1 summarizes the kinds of media types supported by the proposed prototype system. The lecture video is the audio-visual recording of instructor. The slides are the power-point slides composed of texts and graphics. The scripts mean handwritten images produced by a drawing tablet. They are overlaid upon the slides. If additional texts are too long to be written by the scripts, the instructor can directly type in the texts using keyboards. Moreover, the instructor can navigate web sites using the links while lecturing. Most importantly, the instructor can run application programs or execution files while lecturing. He can run any of these media resources at any time during the entire lecturing or encoding stage. The prototype system automatically synchronizes and integrates individual resources. Fig. 1 shows the location of media type on the instructor's Fig. 1 Lecturer's Screen during Encoding Table 2 Viewport Description Viewport Media Name Contents Texts Lecture Title Display View Lecture Video Instructor s Image Texts Elapsed Time Slides Current Slide Number Data View References Demo Programs Links Web Links Slide View Slides Lecture Slides Scripts Handwritten Images Text View Texts Supplementary Texts Journal of Engineering Education Research, 17(3),

3 Jou, Wouseok tabs. The slides tab shows the slide number we are showing. The references tab allows the instructor to choose and demonstrate application programs or execution files. Similarly, the link tab allows the instructor to navigate through web sites during the lecture. However, to click the tabs and to demonstrate them, the instructor must catalog the reference files or URLs under a predefined directory before the lecture begins. The slide viewport shows the lecture slide with the handwritten scripts overlaid on it. The text viewport occupies the bottom of the screen. The scroll bars and buttons in it are used to view and save the typed-in texts while lecturing. Fig. 2 Reference Tabs and Link Tabs. 2. Launch Mode versus Capture Mode While encoding the lecture, instructors can run execution files to enhance the understanding the lecture contents. Also they can run application programs like Excel to demonstrate how to use it. These operations are enabled by clicking references tab during lecturing as shown in Fig. 2. In addition, instructors can navigate and explain the web contents by clicking the links tab. However, in using these tabs, they must select the video mode by clicking the Launch or Capture submenu as shown in Fig. 3. The launch mode is useful for student's self-exercise. It assumes that the program already resides in the student's computer. Common application programs such as web browser, word processor, spreadsheet, and presentation software are usually already installed in the student's site. Therefore, in this mode, the encoder simply records the opening time and name of the application, leaving the decoder to actually drive the application in the student's computer. In this way, the data exchange and communication load between the encoder and decoder can be minimized. In the launch mode, however, the application is running by itself on the student s site, and the control of the application is switched to the student rather than the instructor. For instance, the instructor might let the students to exercise the Excel sheets for themselves since the instructor has already finished explaining how the Excel sheets work. In this mode, the instructor limits the duration of the application, and the application stops at a preset time to return the control to the instructor. Fig. 3 Mode Selection. The capture mode is useful when the instructor wants to explain while running the application. Contrary to the launch mode, it captures the video sequence of the application run. The encoder records it as a simple movie file. It runs the movie file in a separate window, but the movie is overlapped and synchronized with the lecture video stream. Moreover, since the control belongs to the instructor, he can stop the application at any time and switch back to the lecture. Similarly, web navigation can be encoded in two ways. In the launch mode, the web browser runs independently in the student s site to encourage the student's self-study. In the capture mode, the instructor's navigation sequence is recorded together with the instructor's explanation. 36 공학교육연구제 17 권제 3 호, 2014

4 Lecture Encoding of Distance Education by Multimedia Integration III. EVENT HANDLING MECHANISM 1. Event Handling The main reason why the prototype system can efficiently integrate hybrid media resources is that it uses an event handling mechanism. Driving a media element during the encoding is treated as an event, and the event is recorded into an event file that records the type of the event and the name of relevant resource data. In this sense, the event file can be called a metafile in that it does not directly combine all the different low level media resource data having independent formats. Suppose that the instructor needs to show an animation. The file may be in student s site already or it can be predownloaded when the decoder starts. Moreover, when the instructor clicks the animation file in the references tab during the encoding, the metafile records the name of the file and the starting time. It does not include the animation file itself. In other words, the prototype system does not combine different types of media files in a unified single file format. Instead, it leaves individual media types to retain its own format, and it only controls and drives them on the metafile level. Proper actions in response to the event are taken by the event handling routine in the decoder. The decoder invokes callback functions depending on the type of the event, and the callback function invokes corresponding media data files. In this way, different media types can be integrated and synchronized without any possible conflict. Fig. 4 shows the event handling mechanism of the prototype. Upon start, the decoder, which is implemented by an activex object, replays the lecture video. At the same time, it starts reading the metafile which is composed of event records tagged with time stamps showing when the events have occurred. Inside the event handler, OnTimer( ) function continuously sends GetCurrentTime( ) message to the lecture video stream and the video stream returns the elapsed time in milliseconds since the start of the replay. On the other hand, OnTimer( ) function sends GetEvent( ) message to the event objects reconstructed by scanning the metafile. The events are sorted by the time stamp value so that they can be recovered sequentially Fig. 4 Event Handling Routine. based on the value. Event.Time is compared with the Time returned from the lecture video. If it is larger than or equal to the Time, then the corresponding event callback is immediately executed. OnTimer( ) function loops through until finally the lecture video finishes. In this way, the metafile and event handling mechanism can control and synchronize the different media elements uniformly. 2. Event Types and Event Records To implement above event handling, event type and metafile format must be determined. Fig. 5 shows the encoder toolbar shown in Fig. 1. Each icon in the toolbar also represents an event type. Table 3 lists the name of the event type and corresponding callback operations. BUTTON_DOWN, BUTTON_ UP, DRAW events are used for script operations. While lecturing, the instructor can overwrite manual scripts on top of the slide. If the instructor has too many things to write, he can make an empty slide instead of overwriting on the existing slide. WHITEBOARD event is used for that purpose. The five icons in SLIDE event control the slide movement so that the instructor can move back and forth. When finished with the WHITEBOARD, the instructor can go back to the previous slide by clicking the return icon. When current slide is filled with too many scripts, he can alternatively erase the scripts portion of the slide by generating the ERASEALL event. The COLOR and WIDTH Journal of Engineering Education Research, 17(3),

5 Jou, Wouseok event is used to select color and width of the tablet pen. TEXT event allows us to enter supplementary text at the bottom of the encoder window. LAUNCH and CAPTURE events are enabled by selecting the submenu shown in Fig. 3. They are disabled by controlling windows. Fig. 6 shows separate view of the some of the events. As soon as an event occurs, the encoder records related data into the metafile. Depending on the type of events, some event requires another data file to be written. Fig. 7 a) shows the format of an event object. It is composed of Event ID, Time Stamp, and Event Record field. Event ID is one byte field storing the type of the event such as Fig. 5 Event Type of the Toolbar in Fig. 1 Table 3 Event Types and Operations Event Type Name User Interface Operations START Start Encoding STOP Stop Encoding PAUSE Pause Encoding BUTTON_DOWN Click Tablet Pen Begin Script Stroke BUTTON_UP Release Tablet Pen End Script Stroke DRAW Drag Tablet Pen Continue Stroke SLIDE or BUTTON_DOWN. Four-byte time stamp records the time when the event occurred. It is initialized to zero when the encoder starts, and records elapsed time in milliseconds since then. As shown in Fig. 7 a), the event record field is of variable length depending on the type of the event. For instance, STOP event requires no event record as shown in Fig. 7 b), whereas SLIDE event requires the target slide number as in Fig. 7 c). Fig. 7 d) shows the BUTTON_DOWN event which marks the two-byte starting coordinates of a script stroke. Since the coordinate values occupy major portion of the encoding data, we store them in a separate binary file to reduce the file size. Once the type is determined, these types of events are of fixed length in the metafile. Fig. 7 e) shows the LAUNCH_START event. Compared with other types of event, this type of event requires variable-length event record field. Application file name should be of any length, and character strings in TEXT event or a URL require variablelength field. The prototype system uses a separate text string file to record the variable-length strings. In this case, the event record shows only the line number of the corresponding string in the string file. SLIDE Control Slides WHITEBOARD COLOR WIDTH ERASEALL TEXT Make Empty Slides Adjust Pen Color Adjust Pen Width Erase Current Scripts Store Text LAUNCH_START Select Launch Start Launch Mode LAUNCH_STOP Close Launch Window Stop Launch Mode CAPTURE_START Start Capture Mode Select Capture CAPTURE_STOP Minimize Capture Window Stop Capture Mode Fig. 6 Event Types in Action. 38 공학교육연구제 17 권제 3 호, 2014

6 Lecture Encoding of Distance Education by Multimedia Integration BUTTON_UP, and DRAW occur. By default, these files are downloaded right before the actual decoding begins in the student's site. If the capture mode or the launch mode is used, additional movie files or execution files are downloaded if they are not already in the student's site. As soon as the lecture begins in the student's site, the lecture video is streamed and the event control loop starts by the decoder. Fig. 7 Event Object Formats in the Metafile IV. RESULTS AND DISCUSSIONS Fig. 8 Files and Flow of Media Information Fig. 8 shows the flow of information in the prototype system. As it shows, the system can integrate diverse media elements such as video streams, graphic strokes, texts, power-point slides in windows metafile format, application programs, and execution files. Among them, the execution files includes widest range of multimedia contents, since any user-written program in execution file format can be used to help student's understanding. This is possible since the metafile in the prototype system treats individual media only in the event level while making it unnecessary to design a single unified file format to cover all various media file formats. The encoder produces three kinds of files on finishing encoding. First, the metafile containing lists of time-stamped event objects are produced. Second, the window media file containing original recoding of the lecture scene is produced. Third, depending on the type of the events, different data files can be produced. For instance, the string files are produced if TEXT or LAUNCH or CAPTURE events occur. Binary files are produced if such script events as BUTTON_DOWN, Often times, the instructor wants to show related web pages while lecturing. Fig. 9 shows a scene linking a web site while the instructor encodes the lecture. The scene appears in a separate window so that it can be overlaid on top of existing lecture scene. Since the web page is running in capture mode, the instructor has control of it. In other words, he can open and close the window and return back to the lecture at any time. Moreover, he can follow related links by clicking a hypertext inside the web page while narrating about the contents. All these instructor behaviors are automatically recorded into the metafile as simultaneous events so that the additional production process to integrate and synchronize the media is unnecessary. Fig. 10 is an example of a launch mode viewed at student's site. It is assumed that the instructor has just finished lecture on how to use Excel. Now he lets the student to exercise it for some time by enabling the launch mode. Right after the launch mode starts, the lecture presentation stops, and the control of the application program is passed to the student. It allows the student to learn first by the lecture on theory, and then by self-exercise. This mode finishes and returns to the lecture if preset time is reached. In this case, the Excel program itself need not be transmitted from the encoder. The encoder sends only the name of the application program through the metafile. Laboratory teaching on how to use software can be facilitated by this approach. Fig. 11 is an example of a capture mode running an execution program. In this case, the instructor has been lecturing with slides. As a result, the manual scripts can be seen on the background slide. The execution program window is placed on top of the lecture slide. Since it is in capture mode, the instructor has control of the execution program, and Journal of Engineering Education Research, 17(3),

7 Jou, Wouseok the entire process of the instructor's demonstration is captured frame by frame with narrations. The instructor's action of closing the window is recorded into the metafile and the metafile allows the decoder to repeat exactly the same sequence of demonstration as the instructor did while encoding. Fig. 9 Capture Mode Linking a Web Site Table 5 compares the prototype system with current major lecture encoding tools. MAXMEDIA supports relatively limited kinds of media. The e-stream supports more diverse media. But it requires the production process of editing and preinserting individual media elements into slides before the actual lecture encoding begins, since it basically supports only the lecture video and slides. For this reason, instructors must make extra efforts besides lecturing itself. Requiring a separate production process for media integration is a time-consuming work in view of the instructors whether it is a pre-processing or post-processing. In case of the preprocessing, media composition and media presentation sequence is relatively fixed so that the instructors have no freedom of varying the media while encoding. SMIL is a useful language for synchronization of multimedia elements, but it requires a post-processing with HTML-like tags to specify such information as media types and screen regions. As a result, it lacks in real-time interaction of media with the lecture itself while encoding. In view of the instructors, these types of user interfaces can be said to be relatively restrictive. Currently, the prototype system supports almost all kinds of multimedia elements. Moreover, it does not require the pre-processing or post-processing in integrating media elements. The integration is handled by the metafile, and the synchronization is handled by the event handler. Most importantly, it allows automatic integration and synchronization Fig. 10 Launch Mode Running an Excel Program Fig. 11 Capture Mode Running an Execution File Table 4 Comparison with other software Tools Media Support Main Features MAXMEDIA estream SMIL Prototype System Lecture Video, Captured Video, Slides, Scripts Lecture Video, Captured Video, Slides, Scripts, Images, Sounds, Animations Lecture Video, Captured Video, Slides, Images, Sounds, Animations Lecture Video, Captured Video, Slides, Images, Sounds, Animations, Scripts, Supplementary Texts, Application Files, Execution Files, URLs Slides and captured video run separate from lecture video Pre-insert images, sounds, and animations into slides Insert individual media by programming Automatically records and synchronizes as the lecture encoding proceeds. Supports different media types. 40 공학교육연구제 17 권제 3 호, 2014

8 Lecture Encoding of Distance Education by Multimedia Integration while encoding. The instructors can keep lecturing just as they do in the classroom. In the meantime, they can produce and demonstrate scripts, text, whiteboards, and applications whenever required, moving back and forth from the lecture mode. All these instructor s actions are recorded as the events, and corresponding callback functions are invoked by the decoder. V. CONCLUSIONS This paper designed and implemented a prototype multimedia lecture control system for distance education. The system uses an event control mechanism based on a metafile. By controlling media on the event level, diverse media could be integrated. In addition, individual behavior of the instructors could be automatically recorded into a metafile in the form of events as they happen during the lecture. Moreover, by synchronizing media using the event handler, the media elements could be synchronized. The diversity of media contents and ease of use are essential parts of increasing the learning efficiency of distance education. The prototype system has been running for some of the courses in this department, and currently our lab is working on extending the prototype system with SMIL. REFERENCES 1. L. Neal and D. Miller(2005). The Basics of E-learning: an Excerpt from Handbook of Human Factors in Web Design, ACM elearn Magn, vol. 2005, issue 8, p J. Rosbottom, J. Crellin, and D. Fysh(2000). A Generic Model for On-line Learning, ACM SIGCSE Bulletin, vol. 32, issue 3, pp T. Okamoto et al.(2000). Towards Intelligent Media-oriented Distance Learning and Education Environments, 8th IEEE Conf. Computers in Education ICCE 2000, pp A.W. Bates(2005). Technology, E-learning, and Distance Education, London U.K.: Routledge. 5. A.W. Bate(2008). Transforming Distance Education through New Technologies, in The International Handbook of Distance Education, T. Evans, M. Haughey, and D. Murphy, Ed. Bingley, UK: Emerald Press, pp Microsoft Corporation(1999), Inside Windows Media, Que Publishing. 7. D.P. Anderson, G. Homsy(1991). A Continuous Media I/O Server and Its Synchronization Mechanism, IEEE Computer, vol. 24, no. 10, pp W. Jou, et al.(2002). Combining Hybrid Media Tools for Web-based Education, LNCS, vol. 2532, pp Realtimetech Corporation, MAXMEDIA Product Description, Available: Xinics Corporation, estream Product Description, Available: heep://xinics.com. 11. W3C, Synchronized Multimedia Integration Language (SMIL 3.0), Available: SMIL J.V. Ossenburggen et al.(2001). Towards Second and Third Generation Web-based Multimedia, Proc. 10th International World Wide Web Conf., pp T.J. Bond, and M. Walpole(2006). Streaming Audio with Synchronized Transcripts Utilizing SMIL, Library Hi Tech, vol. 23, no. 3, pp M. Milrad et al.(2005). Implementing an Educational Digital Video Library Using MPEG-4, SMIL, and Web Technologies, Educational Technology and Society, vol. 8, no. 4, pp 주우석 (Jou, Wouseok) 1983년 : 서울대학교전자공학학사 1987년 : Univ. of Florida 컴퓨터공학석사 1991년 : Univ. of Florida 컴퓨터공학박사 1992-현재 : 명지대컴퓨터공학과교수 red@mju.ac.kr 관심분야 : 원격교육시스템, 컴퓨터그래픽스, 게임프로그래밍 Journal of Engineering Education Research, 17(3),