Multimedia Systems Design. Klara Nahrstedt. University of Illinois. curriculum because digital audio/video information

Transcription

1 Multimedia Systems Design Teaching Students How to Design and Build Multimedia Systems Klara Nahrstedt University of Illinois Abstract \Multimedia Systems Design (MSD)" teaches graduate students principles and practice of large-scale multimedia system design. The biggest challenge to teach students how to design and build multimedia system is the integration aspect because multimedia systems integrate various computer science areas such as audio/video processing, operating system concepts, high-speed network protocols, user interfaces, and others. Our approach is to apply concepts and examples from these areas to construct a signicant project, namely a digital distributed VCR system. The course focus is on a large-scale, longlived software system of a size requiring group eort. We therefore address multimedia systems design, integrated implementation, maintenance, testing, documentation, code readability and group dynamics. The results which we have seen are (1) a large set of useful software which the student carries with them after the course is over; (2) a learning of new concepts in the area of audio/video processing and communications; (3) an understanding of integration importance when designing and building multimedia systems; (4) an appreciation of software design and maintenance, in particular group dynamics and communication; and (5) a greater understanding of advanced CS topics such as programming language design and implementation, user interfaces, compression algorithms, communications protocols, high-speed networks, and operating systems. We show how the student's knowledge, capabilities and understanding of lectured material together with engineering and scientic principles are related to the programming assignments of the VCR project. We also present the course methodology, and eval- This work was supported by the National Science Foundation Career Award, and Oce of Vice President for Academic Aairs, UIUC. uate project management, group structure and dynamics. Such a course is essential to Computer Science curriculum because digital audio/video information becomes a permanent part of our computing and communication environment. This course enables a student's transformation from a programmer with a few pieces of abstractions into a functioning computer scientist capable of designing and building future advanced multimedia systems. 1 Introduction There is a need for extension in the Computer Science curriculum towards multimedia system course. Digital audio/video information are becoming part of ubiquitous computational and communications environment. It means that the multimedia systems including this type of information are coming out of research laboratories and entering our educational organizations such as universities. The educational organizations use multimedia systems as tools to improve the education of traditional courses, and they apply this technology to increase the eectiveness of course preparation and teaching [SB95]. However, we also need courses which will teach the future computer scientists the basic concepts and principles how to design and build integrated multimedia systems. It is necessary but not sucient to have the knowledge of traditional courses such as programming languages, operating system, distributed systems, real time systems or networks. These courses typically introduce individual fundamentals (e.g., scheduling, protocols, distributed algorithms). The assignments are smaller: they demonstrate language features or build toy applications and data structures. They are, naturally, individual assignments. The multimedia systems are complex systems and require a non-trivial integration of individ- 1

2 2 ual traditional components with new concepts and approaches. We have developed a course Multimedia Systems Design which we believe ts the extension in the CS curriculum. It introduces new concepts, services, protocols, algorithms and integration of all of them to design and build future advanced multimedia systems. Furthermore, the focus is on the application of principles and concepts, and on learning from a single extended experiment. The learning is directed towards construction of signicant long-lived, large-scale multimedia systems, as opposed to construction of throwaway examples. A number of observations helped to shape the course: Multimedia systems are complex systems and integration of individual system components is a non-trivial task. Students mostly bring knowledge and experience from individual areas such as operating system, audio/video processing, networking, programming languages and/or performance analysis. Thus, our course should provide bridges and glues between these areas as well as teach new concepts where traditional concepts do not apply. Due to the complexity of the multimedia systems design and implementation software engineering techniques must be applied. Signicant software engineering tasks have a long lifetime, characterized by a design phase, an implementation phase, and a long maintenance phase. In real systems, the maintenance phase accounts for most of the money spend, and thus there is typically signicant eort spent in the design phase to ease maintenance. One diculty with long-lived systems is that environments change and new features are required. Thus, one must design for maintenance, coupled with the notion of software reuse. A course should structure assignments in such a way that previous work should be reused in phases. At each phase, the previous code is used, or the instructor's code is used (necessitating reading and understanding a system which is more complex), as a platform. Due to the complexity of multimedia systems, design and testing strategies must be introduced. The students must understand the trade-os between their design choice and the impact on the performance of the system. In many cases, design activities are independent of implementation, but all implementations request testing. The choices of test cases, and the choice of testers is crucial to eective testing. Due to the complexity of multimedia systems a team builds the system to shorten the design, development and testing phases. For team communication documentation is essential because many involved in the design and engineering of the system do not want to read the precise statement of problem solution embodied in the code in order to obtain adequate understanding for their role. They want to understand precisely only the interfaces required for performance of their own tasks. The documentation among the team members can take the following form: embedded commentary, associated text les in form of Latex, pointers to relevant literature included in the program text. All these forms contribute to better human understanding because big systems need more than a few people on the same wawelentgh. It is important to read code written by dierent members of a team. Therefore, code must be written in such a way as to be read. Coding standards of interfaces, and style sheets help in program readability. Due to the complexity of multimedia systems a team building the system must work together. Group size, individual skills within the group and adequate group communication are important aspects for the success of the project. The course has accomplished successfully its goals. The students leave the course with a thorough understanding of various areas in multimedia systems, especially their integration aspect. Furthermore, they learn a tool-rich programming environment for building complex audio-visual systems. More important, they have worked on a project of signicant scope, built signicant software application, and understand the group nature of systems building. There are not many multimedia systems courses available currently. There exist several courses which emphasize individual parts of multimedia systems, such as courses on video compression [Smi95], multimedia communications [Aus95, om95], multimedia applications [Ber95], or multimedia databases [Aus95]. Our course similarly to the other courses

3 3 includes the basics of the individual parts of a multimedia system. The major dierence we see is in the integration aspect of the multimedia systems parts and the choice of the project which stresses the importance of the integration (nahrstedt.cs.uiuc.edu:5000/pub/htdocs/home.html and nahrstedt.cs.uiuc.edu:5000/pub/cs397mm.html). The paper is organized as follows: in Section 2 we present a detailed description of course material. Readers not interested in the details are encouraged to use the summary in Table 1 at the end of Section 2. Section 3 discusses the course management issues. Section 4 reports on results and achievements of this course as well as impact and byproducts of this course. 2 Course Detail The course presentation was designed so that covered material would emphasize the integration aspects of the multimedia systems and would properly map to the project design, implementation and testing stages. The lectured material and the project are approached so that they would not become obsolete; the students would get the knowledge and experience with the newest technology, however, would understand the tradeo among the design choices if certain technology is not in place. The students would be constantly working towards both the development of a project and overall understanding of the multimedia system issues, which would serve them well in both this course, and later courses. The following books comprise the reading list: \MULTIMEDIA: Computing, Communications and Applications" [SN95], chosen because it covers the basics of individual parts in multimedia systems, and provides strong integration between them as well. \UNIX Network Programming" [Ste90], chosen because it illustrates use of UNIX networking tools and libraries with realistic examples. "An Introduction to TCL/TK" [Ous94], chosen because it teaches use of a programming language TCL/TK to develop fast complex user interfaces. In addition, the following books and documentations are recommended reading for students unfamiliar with some operating and networking system concepts \Operating System Concepts" [SG94], \Computer Networks" [Tan88]. To work on the multimedia programming project \Video Development Environment" and \Audio Development Toolkit" reference guides [HP95, HP92] are provided. In subsection 2.1 to 2.7 we present the assignments that were given and their intended role in the student's experience. All assignments were specied in form of a design document. 2.1 Project Design The rst assignment towards building the multimedia VCR system was to provide a high-level specication and design document of a digital VCR. The students got from the instructor a high level system architecture specication as shown in Figure 1. It in- UI Client IPC Scheduler Sync TCP/IP UDP/IP Server Scheduler Figure 1: High-level Architecture of the VCR System dicates that the nal version of the built system will consist of client and server end-points. The server side will operate (store/retrieve) on one video/audio le (movie clip). The client side will include the user interface with appropriate buttons for initiating functions PLAY, RECORD, FAST FORWARD, REWIND, STOP, and PAUSE. The user interface will be a separate process communicating through InterProcess Communication (IPC) mechanisms with the main process called Scheduler at the client side. 2.2 Audio and Video Basic Functionalities Since the students were not expected to be familiar with digital audio and video, this assignment gave them the opportunity to start to work with audio and video devices. Using the manuals for au-

4 4 dio and video API (Application Programming Interfaces), they were asked to implement a program vcr which takes arguments from a UNIX command line. The arguments are function, type, length, le. The function argument represents the `vcr' functional values such as PLAY, RECORD, FAST FOR- WARD or REWIND, type species media types such as the audio or video type, length species number of frames, which determines how many seconds the student will record/play a audio-visual clip, le includes the le name such as le.au (audio le) or le.mjpg (motion JPEG le). An example of such a UNIX command line is vcr RECORD audio 5 file.au which means that the program should read from the audio input device for 5 seconds and write the audio data to the local le le.au. 2.3 Audio/Video Scheduling Until now the students treated the processing of audio/video information independently. However, when playing a movie, audio and video information must play concurrently. If both media types share the same processor, which we assume, they must be scheduled together in a careful manner. Therefore, the following assignment directed the students to think about CPU resource their audio and video streams consume. The students were required to implement (1) admission control for CPU allocation 1 to check if there is enough CPU bandwidth available and (2) rate-monotonic scheduler which schedules audio and video tasks together. The students reused and modied the function vcr further. An example of the extended functionality vcr PLAY 5 file.au file.mjpg indicates the tasks in this assignment. The program should provide admission control of audio/video PLAY tasks, specication of a feasible schedule for PLAY functionality, retrieval of audio/video samples and frames from the le.au, le.mjpg and write them to audio headphone device, and display on the screen according to the rate-monotonic schedule for duration of 5 time units. 1 This means to perform a schedulability test for ratemonotonic scheduling [LL73]. Also note that in order to go through the schedulability test, the students need to measure the processing times (e) of the write/read audio and video tasks processing samples/frames and store them somewhere so that the admission service nds the information for admission. 2.4 Client/Server Multimedia Communication At this point of the multimedia system (vcr) design, the students became familiar with local processing of the digital audio and video information. However, in current distributed computer environment, few of the multimedia systems operate locally due to large size of multimedia information. Digital audio and video information is and will be stored on video servers, hence the students need to learn about client and server multimedia communication. This goal was achieved in the client/server multimedia communication assignment. The vcr program was extended and the high-end video/audio devices separated from the disk device through a local area network. The client side of the vcr program included ratemonotonic scheduler with admission, implemented in previous assignment, and processed information coming to/from audio microphone/headphone, video parallax board /display [PG95] and the network. The server side of the vcr program on a dierent machine scheduled tasks according to the rate-monotonic scheduler and processed information coming from/to network and disk. The multimedia call between the client and the server vcr programs had several network connections. One connection (control connection) carried the control information between the client and the server (separation principle of control and transfer functions in Broadband ISDN architecture)[sn95]. The second connection (transfer connection), from client to server, carried audio/video during recording. The third connection (transfer connection), from the server to the client, carried audio/video during playback. For implementation of the control connection the students used the TCP/IP protocol, for the transfer connections the students had an option to use either a UDP/IP or TCP/IP protocols. Taking one of the design choices they had to understand the tradeos of their decision. 2.5 Lip Synchronization At this point of the vcr multimedia system design, the students realized that the network introduced unpredictable delays and the audio/video information for PLAY function may have arrived having large jitter 2 2 Jitter measures the variance of dierences between intermediate arrivals of packets belonging to the same stream.

5 5 and skew 3 delays which contributed to the out-of-sync situation between audio and video information. The students improved the out-of-sync situation by synchronizing audio with video during PLAY function. To achieve this they had to take the following steps: 1. During the RECORD function at the client side they captured the relation between the audio and video frames. For example, Audio sample 1 <=> Video Frames All these information created a synchronization information. 2. After they captured the synchronization information during the RECORD function, they sent it to the server and stored it in a separate le. 2.6 User Interface for Multimedia Applications The user-friendly access to the vcr program in form of a user interface was part of the multimedia system design. Hence, the students were asked to implement user interface, which consists of buttons STOP, PAUSE, PLAY, RECORD, REWIND, FAST- FORWARD, using TCL/TK [Ous94] and eld for text input of the movie name. An example of a user interface is shown in Figure 2. MOVIE NAME Digital VCR System - Graphical User Interface 3. During the PLAY function they utilized the synchronization information by retrieval and display operations. During retrieval, the synchronization information helped to retrieve information which belong together. During display, the synchronization information was made available to the play-audio and display-video threads, so that the group belonging together was played in one particular interval. 4. During the PLAY function the network introduced unpredictable delays and jitters into the audio/video transmission which made the synchronization during the display very dicult at the client side. One possibility to work around this problem was to use smoothing buers at the client side and adaptive feedback. Students introduced between the receive threads and display threads a ring buer. The students learn about pointer management to control overow and starvation of the smoothing buers. For resolving both critical situations, they used adaptive feedback sent from the client to the server to decrease the sending rate if receiving was faster than displaying/playing video/audio or to increase the sending rate if receiving was slower than displaying/playing g video/audio information. 3 Skew measures the dierence between intermediate arrivals of correlated packets belonging to two dierent streams. RECORD PLAY PAUSE EXIT STOP REWIND Figure 2: Possible Graphical User Interface of the VCR System This assignment emphasized the user-friendly design and consistent implementation of the buttons representing a nite state machine. Especially, the students took into account what knowledge a user brings about buttons and their underlying functionality from the analogue TV vcr system environment. 2.7 Integration In the nal assignment, the students integrated the user interface with the underlying vcr system which included lip synchronized client/server multimedia communication. Integration was the hardest part because the individual modules had to t together. As an example of the integration we will show the integrating protocol between the GUI and some underlying `vcr' functionalities for PLAY function. FF

6 6 Segment Description 1. Project Design Design GUI and process structure 2. Multimedia `record', `play' Functions `rewind', `fast forward' 3. Multimedia CPU admission Scheduling and RM scheduler 4. Client/Server Client/server Communication protocols for audio/video 5. Lip Synchronization Synchronization structures, and streams 6. User Interface User-friendly GUI 7. Integration Integrate client/server communication with GUI Table 1: Summary of Course Phases When user clicks on the PLAY button, the GUI process sends a message to the main process through IPC. The main process does all the functions between the client and server to establish a multimedia call for PLAY and send/play movie in synchronized fashion. The students had to be aware that the scheduler at the client side should also run a control task which will check the IPC for incoming user requests. For example, the user might press the PAUSE button or STOP button during playback. In this case the control task should get the message and pause/stop the client threads which receive/play movie data as well as send a control message to the server to pause/stop the server tasks to retrieve/send movie data. 2.8 Overview Table 1 gives a summary of the course phases. All phases are group eorts. The gure 3 shows the building blocks of the project and the re-usability between implementation phases. 3 Discussion of the Course There are several components we see in this type of a course, namely (1) learning the state of the art in multimedia systems area, (2) experiments, (3) replication of experiments, and (4) observation and deduction. The course met for two lectures per week. There were also interactive lab session meetings with the instructor and the teaching assistant at least twice a week either during oce hours or arranged times in Figure 3: Reuse between implementation phases areas with terminals. The lecture material emphasized newest mechanisms, algorithms and approaches - the state of the art - in multimedia system area. The machine problem assignments as described in Section 2 provided a broad range for experimentation. Student experimentation was of three types. First, given that the students were working with a digital audio and video, they could apply and utilize in `reverse engineering' some of the existing multimedia software. The deduction and observations from these experiments were incorporated into the design of the student projects. Students were enthusiastic about experimenting especially because they could see and hear results of their experiments. Second, the students experimented with new concepts by writing smaller programs and gradually building up a new application. Each assignment was designed so that the students could clearly see and hear if their task was accomplished. Third, the students experimented with design choices and trade-o decisions for multimedia systems. Each assignment left enough freedom for the group to make their own decision on data structures, and basic algorithms for audio and video processing and communication. For example, the students could decide if they want to send video and audio over separate connections or multiplex them into one connection. Experimental data was replicated within a group for several reasons. First, working in groups meant to divide a workload, and at the end integrate individual modules. Hence. uniform experimental data helped to test individual modules as well as the integrated software. Second, common experiments led to a discussion among students in the group, and better group communication which improved debugging and

7 7 testing of the integrated software. We used electronic communication which allowed the student to obtain answers across the week, rather than a few preset times. An online bulletin board mechanism allowed posting of sources, interesting questions, interesting answers, and the details of the assignments. WWW provided help in asynchronous learning. The students could access lecture material on the web in form of a hypermedia document consisting of text (html), gif les, short audio and video clips, taped during lecture, digitized, edited and posted. (viewed with vosaic [CTCL95] software). The html document with text and gif les was available before lectures, digital audio and video were added to the hypermedia document later. The choice of the VCR application had four positive eects: 1. Students knew this application from their homes, hence the functionality of the application and the goals where clear. 2. The students were not forced to complete the design process before they were ready. Their design process consisted of analyzing new features in the context of their existing software, designing appropriate data structures and algorithms, and implementing the features. 3. The completed VCR application allowed the students to become familiar with most of the aspects of digital video and audio processing and communication. Functional details of compression, scheduling and Internet communication taught students not only the individual areas but also the integration aspect when building a complex distributed multimedia application. 4. Experimentation with digital video and audio within their VCR application was a very worthwhile experience. Especially, because they could mostly hear or see how their design performs. Grading of all programming assignments previous to the project completion relied on an even split between code quality and execution testing. The execution testing was done in form of short demonstrations. The nal project was graded wholly by success or failure on a set of individual experiments. One dif- culty which seems to always occur in group work is unequal contributions. This was actually resolved by assigning all group members the same grade for the project. It was interesting that weak members of a group didn't do very well on exams neither, therefore, the overall grade for this group member was lower. Overall, the course had the following principal accomplishments: The student developed a non-trivial multimedia application, consisting of both techniques and developed skills with digital audio and video. The focus on one signicant project brings out the point of integration, which is so important in distributed multimedia applications. Forcing integration of new features with previous work also demands that attention be paid to design. Of course, building on previous work shows the value of re-use as illustrated in Figure 3. The course is a system course which brings, similarly to operating systems course, together concepts, approaches, algorithms and structures with real implementation and experimentation on real application. 4 Conclusion I hear and I forget, I see and I remember, I do and I understand. (Chinese Proverb) We believe that the construction of signicant multimedia systems is a necessary part of the multimedia education in Computer Science. While the impact of education is dicult to evaluate, we will attempt a brief discussion. Did the course fulll its goals? Discussions with instructors of more advanced system courses in the CS curriculum and with advisors of students per-suing multimedia system research have noted that students from `Multimedia Systems Design' class perform better than those who had not taken the course. Discussions with several companies indicated that students who took this class are well prepared to design new challenging multimedia systems. Some requested the names of top few students. They then recruited them for their company. Are the graduates satised? Several students dropped the course due to the extensive programming required. However, of the 36 graduates last semester, several requested recommendation for industrial positions, some students extended

8 8 their project and their knowledge in this area towards master thesis. Several students continue towards PhD in the area of multimedia systems. [Smi95] B. Smith. Multimedia systems. 95/CS631, Fall References [Aus95] University of Texas Austin. Multimedia communication and databases. Course; [Ste90] [Ber95] University of California Berkeley. Multimedia systems and applications. Course; Fall [CTCL95] Z. Chen, S-M. Tan, R.H. Campbell, and Y. Li. Real Time Video and Audio in the World Wide Web. In WWW 95, December [SN95] R. Steinmetz and K. Nahrstedt. Multimedia:Computing, Communications, and Applications. Prentice Hall, Inc., W. R. Stevens. UNIX Network Programming. Prentice Hall, [Tan88] A. S. Tanenbaum. Computer Networks. Prentice-Hall, Englewood Clis, NJ, 2 edition, [HP92] [HP95] Corporation HP. Using the Audio Application Program Interface. Hewlett Packard Corporation, September Corporation HP. Video Development Environemnt. Hewlett Packard Corporation, [LL73] C. L. Liu and J. W. Layland. Scheduling Algorithms for Multiprogramming in a Hard Real-Time Environment. Journal of the ACM, 20(1):46{61, January [om95] University of Massachussetts. Multimedia networking. Course; [Ous94] J.K. Ousterhout. An Introduction to TCL/TK. Addison-Wesley Publishing Inc., [PG95] Inc. Parallax Graphics. Video Development Environment, Reference Guide for Sun Solaris, HP/UX and IBM AIX. Parallax Graphics, [SB95] M. Swaord and D. Brown. Mallard. September [SG94] A. Silberschatz and P.B. Galvin. Operating System Concepts. Addison-Wesley, 1994.