Ch. 0: Course Overview Multimedia Systems Prof. Ben Lee School of Electrical Engineering and Computer Science Oregon State University Outline What is multimedia? Why multimedia? Multimedia applications Technical challenges Course outline Course logistics 2 1
What is Multimedia? (1) Development, integration, and delivery of any combination of text, graphics, animations, sound, and video through a computer. 3 What is Multimedia? (2) When different people mention the term multimedia, they often have quite different, or even opposing, viewpoints: A PC vendor: A PC that has sound capability, a DVD-ROM drive, and perhaps the superiority of multimedia-enabled microprocessor that understand additional multimedia instructions, e.g., MMX, AltiVec, SSE, 3D Now! etc. A consumer entertainment vendor: Interactive cable TV with hundreds of digital channels available, or a cable TVlike service delivered over a high-speed Internet connection, VoD. A Computer Science & Engineering student: Applications that use multiple modalities, including text, images, drawings (graphics), animation, video, sound including speech, and interactivity. 4 2
Why Multimedia? (1) Information can often be better represented using audio/video/animation rather than using text, images and graphics alone. Collaboration and virtual environments. Potential for improving our lives (e.g., learning, entertainment, and work). High Market Demand! Major driver of Computer Technology 5 Why Multimedia? (2) Convergence of computers, communications, and TV. Growth in computational capacity: Dramatic increase in CPU processing power. Dedicated compression engines for audio, video, etc. Rise in storage capacity: Large capacity disks. Increase in storage bandwidth, e.g., disk array technology and SSD. Surge in available network bandwidth: High speed networks - gigabit networks. Fast packet switching technology. 6 3
Reality The word multimedia has been overused to the point that it does not mean anything. Multimedia is pervasive in our society! 7 Mobile Multimedia Devices Enabling Technologies Samsung Galaxy S6 Application Processor Text ZigBee Video Mobile OS Image Bluetooth WLAN Satellite GSM/CDMA Audio WiMAX/WiBro Apple iphone 6 Garmin GPS HP Chromebook Apple ipad Air 8 4
Games Enabling Technologies Desktops Application Processor Text ZigBee Video Mobile OS Image Bluetooth WLAN Satellite GSM/CDMA Audio WiMAX/WiBro Sony PlayStation PS4 Microsoft Xbox One Nintendo WII U 9 Wearable Computers Enabling Technologies Smart Clothing Application Processor Text ZigBee Video Mobile OS Image Bluetooth WLAN Satellite GSM/CDMA Audio WiMAX/WiBro Apple Watch Google Glass GoPro Hero Oculus VR 10 5
Residential Applications Baby Monitoring Enabling Technologies Application Processor Text ZigBee Video Mobile OS Image Bluetooth WLAN Satellite GSM/CDMA Audio WiMAX/WiBro Multimedia Home Shopping + Virtual Reality Smart Homes Video Conferencing Peer-to-Peer Video Streaming 11 Business Applications Enabling Technologies Distance Education Application Processor Text ZiBee Video Mobile OS Image Bluetooth WLAN Satellite GSM/CDMA Audio WiMAX/WiBro Cooperate Training Video on Demand Multimedia Conferencing 12 6
Remote Presence Systems The ability to project yourself to another location and to move, see hear and talk as though you were actually there. InTouch Health RP-7 Robot 13 Immersive Environment An artificial, interactive, computer created scene, or world within which a user can immerse themselves. Space Shuttle Aerodynamics (Los Alamos National Lab) Virtual Store (Kimberly Clark Corporation) Immersive Command & Control (US Army) 3D Design Software Projected Grid (Northrop Grumman) (Los Alamos National Lab) 14 14 7
Technical Challenges Need to manage high volumes of data in a timely manner. Need to process traditional media (text, images) as well as continuous media (audio/video). Need to communicate multimedia data across networks that makes no promise about End-to-end delay, Variation of packet delay within a packet stream (i.e., jitters), or Bandwidth. This puts a strain on all aspects of computer and network technologies: computation, communication, and storage. 15 High Data Volume Characteristics BW Required Speech CD Audio Satellite Imagery NTSC Video HD video 8000 samples/s, 1 byte/sample 44,100 samples/s, 2 bytes/sample, stereo 180 180 km 2 30 m 2 resolution 30 fps, 640 480 pixels, 3 bytes/pixel 60 fps, 1920 1080 pixels, 3 bytes/pixel 8 Kbytes/s or 64 Kbps 176.4 Kbytes/s or 1.41 Mbps 600 Mbytes/image (60 MB, compressed) 27.6 Mbytes/s or 221.2 Mbps (2-8 Mbps, compressed) 373.2 Mbytes/s or 2.99 Gbps (15-30 Mbps, compressed) 16 8
Course Outline Ch. 0: Course Overview Ch. 1: Audio/Image/Video Representation Ch. 2: Basic Coding & Compression Techniques Ch. 3: Image compression Ch. 4: Video compression Ch. 5: Audio compression Ch. 6: Multimedia Networking Ch. 7: Mobile Application and Multimedia Processors 17 Ch.1: Audio/Image/Video Representation Computer Representation of Audio: Quantization Sampling Digital Image Representation: Color system Chrominance Subsampling Digital Video Representation Hardware Requirements 18 9
Ch. 2: Basic Coding & Compression Techniques Why compression? Classification Entropy and Information Theory Lossless compression techniques: Run Length Encoding Variable Length Coding Dictionary-Based Coding Lossy encoding 19 Ch. 3: Image compression Introduction JPEG Standard: YUV or YIQ conversion and subsampling DCT Quantization Zig-zag ordering and RLE/DPCM Variable Length Coding (entropy coding) JPEG Header Format JPEG Compression Modes 20 10
Ch. 4: Video compression Introduction MPEG overview MPEG encoding: Motion compensation Frame encoding MPEG video bit stream MPEG video standards H.264/AVC 21 Ch. 5: Audio Compression Sampling techniques (PCM): Linear Non-linear Generic Coding Techniques: DPCM ADPCM Psychoacoustic Coding MPEG 22 11
Ch. 6: Multimedia Networking Introduction Wireless LANs: What is the Internet? 802.11 Dealing with packet jitters and losses: Issues Reducing Delay Jitters Recovering from packet loss Case Study: Error Resiliency Feature of H.264 23 Ch. 7: Mobile Application and Multimedia Processor Introduction Application Processors The ARM Architecture Intel Atom Processor 24 12
Other Topics Not Covered OS support for multimedia Real-time scheduling, buffer handling, I/O subsystems, OS structure/architecture, etc. Multimedia applications and services: Content authoring Content query Distributed collaboration Multimedia synchronization Mobile multimedia 25 Course Logistics (1) Class web site http://www.eecs.orst.edu/~benl/courses/ece477_sp18.html Textbook: Fundamentals of Multimedia, by Li and Drew, 2004, Prentice Hall.<http://www.cs.sfu.ca/mmbook/> Materials from archive journal, conference, and magazine papers. See the course web page. Reference books: Computer Networking: A Top-Down Approach Featuring the Internet, 4 th Edition, by Kurose and Ross, 2008, Addison Wesley. Internetworking Multimedia, by Crowcroft, Handley, and Wakeman, 1999, Morgan Kaufmann Publishers 26 13
Course Logistics (2) Office Hours: MW 1:00-2:00 and by appointment. Grading Policy: Assignments (20%) 2 exams (50%) Final project and presentation (30%) 27 Final Project and Presentation Work in groups. Start early! Doing background work is more than half the work. 30% of the grade! Types of projects: Survey Implementation/Demo Project Proposal & Progress (mid) Report Presentation will be done in the last week(s) of class. All group members must be involved in the presentation. Final report due during finals week. 28 14
Some Project Topics Look at course web site for projects done in the past Video/Audio compression/processing OS support for multimedia Synchronization Multimedia applications Middleware for multimedia Hardware architecture Content authoring VoD VoIP QoS Security 29 Questions? 30 15