Multimedia Systems Giorgio Leonardi A.A.2014-2015 Lecture 1: General information and Introduction
Overview Course page (D.I.R.): https://disit.dir.unipmn.it/course/view.php?id=639 Consulting: Office hours by appointment: giorgio.leonardi@mfn.unipmn.it Office #182 (in front of Sala Seminari ) Email me any time
Overview of this course Multimedia Systems cover a very large number of topics
Objectives of this course Understand the basic principles of multimedia: Signal representation and sampling Digital signals properties, elaboration and compression Mathematical Transforms Compression Algorithms Image representation and common image formats: GIF JPEG Image processing and filtering Sound representation and common sound formats: PCM MP3 Sound processing and filtering
Out of the scope of this course Audio and Image editing Software You re Computer Scientists, this is not a Photoshop course!!! Media authoring: Flash, HTML, Multimedia retrieval
Prerequisites None in particular, but Recommended: Basic math Basic programming skills
Organization of the course 48 hours total/24 lectures 18 face-to-face lectures: theory 6 lectures: work in lab: Exercises about image loading and processing
Schedule 09:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 Lunedì Martedì Mercoledì Giovedì Venerdì Dalle 11:00 alle 13:00 SISTEMI MULTIMEDIALI Leonardi Giorgio DiSIT aula (206?) Dalle 16:00 alle 18:00 SISTEMI MULTIMEDIALI Leonardi Giorgio DiSIT aula (206?)
Final exam Exam is split in two parts: 1. Short project: Implementation of a filter or of a processing image algorithm grade1 2. Written examination about theory and some exercise grade2 Final grade: weighted median of grade1 and grade2:
Finalm exam Part 1 (lab): Assigned during the last lecture in lab Take your time (not months!) to complete it and send it to me at any time We will discuss it briefly within a week from posting Meanwhile, contact me for any problem Part 2 (theory): Performed during the proper exam sessions No mid-term exams The two parts are taken separately and in any order, but must be both passed: grade1 18 and grade2 18 If only one of the two parts fails, the other grade can be maintained within one year.
Reference books Suggested Textbook Jennifer Burg The Science of Digital Media Prentice-Hall, 2009. [http://www.pearsonhighered.com/educator/product/fundamentals-of-multimedia/9780130618726.page] Optional Textbook (in UPO s library) Ze-Nian Li and Mark S. Drew Fundamentals of Multimedia Prentice-Hall, 2004. [http://www.pearsonhighered.com/educator/product/fundamentals-of-multimedia/9780130618726.page]
Reference books Additional Books Nigel Chapman Digital Multimedia, 3rd Edition John Wiley & Sons, 2009 Fred Halsall (in UPO s library) Multimedia Communications: Applications, Networks, Protocols and Standards Addison-Wesley, 2001 [Out-of-Print]
A brief introduction to multimedia systems
Multimedia Multimedia is an overused term because It touches different disciplines/industries Engineering, Computer Science Telecommunications Industry TV & Radio Broadcasting Industry Consumer Electronics Industry
Multimedia Multimedia is an overused term because It means different things to different people PC vendor: PC that has great sound capability, a Blu-Ray drive, a 3D graphics card, Consumer entertainment vendor: interactive cable TV or video-on-demand TV-like services
What is Multimedia? Multimedia can have many definitions. These include (from the computer system perspective): Multimedia means that computer information can be represented through audio, video, and animation in addition to traditional media (i.e., text, graphics/drawings, images).
Multimedia It is not a new concept: Magazines: text + still images TV: audio + video Radio: audio + text (RDS) Contemporary opera: speech + video + sound WWW page: text + audio + video + images
General Definition A good general working definition: Multimedia is the field concerned with the computer controlled integration of text, graphics, drawings, still and moving images (Video), animation, audio, and any other media where every type of information can be represented, stored, transmitted and processed digitally.
Multimedia Application Definition of Multimedia Application: A Multimedia Application is a computer application which uses a collection of multiple media sources e.g. text, graphics, images, sound/audio, animation and/or video.
Multimedia application We are interested in digital multimedia, where media are digital (or digitized) documents Bits represent text, sound, pictures, Can be manipulated by computer programs Can be transmitted over computer networks Users can interact with media elements in novel ways For instance, by manipulating objects directly using the mouse
Multimedia Applications Examples of Multimedia Applications include: World Wide Web Multimedia Authoring, e.g. Adobe/Macromedia Director Hypermedia courseware Video-on-demand Interactive TV Computer Games Virtual reality Digital video editing and production systems Multimedia Database systems
Types of Media Media (pl. of medium) are means for communicating information It is used both in abstract and concrete terms Abstract: ways to communicate information Text, audio, Concrete: all of the various technologies we use to record information and transmit it to others Videotape and DVD-RW are recording media TV and radio broadcasting are transmission media
Types of Media
Types of Media: Static/Time-based Two different types of media: Static media: does not change with time Text Still images Time-based media: exhibits some change over time Moving images (animations) Video Sound
Types of Media: Linear/Nonlinear Media may be divided into linear and non-linear categories Linearity: information progresses sequentially from the beginning to the end The contents are arranged in a logical order of presentation from beginning to end Refers to the media creator s intention and cannot be changed or modified by the reader Nonlinearity: the flow of information is changed by user s interaction
Linear/Nonlinear Example: a book Fixed arrangement of text pages Book: physical arrangement of text and pages implies a linear reading order
Linear/Nonlinear Example: a film A fixed sorted array of frames Film: fixed order of frames defines a single playback sequence
Example: a hypertext E.g., Web pages Linear/Nonlinear Hypertext: Information flow can be changed by means of references (hyperlinks, aka anchors). Links between pages permit multiple arbitrary reading order
Media: Interactivity Interactive media is the ability of media to respond to the user s actions System is able to select media and information on the basis of the user s requests Web pages can automatically display information coming from other sources or that has been retrieved from a database Interactivity is enabled by nonlinearity
Interactive media Media: Interactivity
Characteristics of a Multimedia System A Multimedia system has four basic characteristics: 1. Multimedia systems must be computer controlled. 2. Multimedia systems are integrated. 3. The information they handle must be represented digitally. 4. The interface to the final presentation of media is usually interactive.
Key Issues for Multimedia Systems Key issues multimedia systems need to deal with: How to represent and store temporal information. How to strictly maintain the temporal relationships on play back/retrieval Data has to represented digitally: Analog Digital Conversion, Sampling etc. Large Data Requirements: bandwidth, storage, Data compression is usually mandatory
Features of a MMS Integration: MMS must be able to integrate contents of different media Digital movie with subtitles: audio + video + text Synchronization: MMS must be able to synchronize contents of different media Videoconference: audio + video
Architecture of a MMS
Architecture of a MMS The usage perspective studies: The type and design of multimedia applications, and The interface between users, multimedia applications and systems
Application Layer Applications Interactive Entertainment Audio / Video Teleconferencing E-learning Telemedicine Cooperative Work Environments Social Media
Architecture of a MMS Services perspective Services and protocols in multimedia networks and communication to send multimedia streams Synchronization of media streams Various system and service components to achieve the best end-to-end perceptual quality for the user
Example of service Group Communication Computer Supported Cooperative Work (CSCW) Videoconferencing Desktop sharing Architecture of a Group Communication System:
CSCW Service Rendezvous components describe processes for the organization of meetings and to set up group sessions Synchronous methods: expliclit invitation (e.g., multicast) Asynchronous methods: implicit invitation (e.g., by email)
CSCW Service Conference components offer services to allow participants to communicate using multiple media streams and to support group communication Defines: establishment of a conference, add new participants, leave the conference,
CSCW Service Cooperation components describe techniques for concurrent replication of information among a large number of participants Application sharing: the resulting changes performed on a shared object are distributed to all participants
Architecture of a MMS Systems perspective This group of multimedia fields relates system areas such as processing, storage, and communication, and their relevant interfaces
Systems perspective Operating system Process management A time-critical process must never be subject to priority inversion A non-critical process should not suffer from starvation due to critical processes Hard real-time requirements: all requirements must be guaranteed Video process within a tele-surgery application Soft real-time requirements: some requirement can be violated Movie on-demand for entertainment purpose Soft real-time scheduling Memory and buffer management Provide access to data with a guaranteed timing delay end efficient manipulation functions
Systems perspective Networks Allow the exchange of media data between devices generating and processing them Require appropriate services and protocols Data have to be processed within a certain delay or during a defined time interval Small end-to-end delay E.g., ATM services (CBR, VBR, )
Systems perspective Communication User and Application requirements Data throughput: stream-like behavior of audio/video demands high data throughput Multicasting: to share resources at end-systems Processing and Protocol constraints Synchronization Error correction/detection E.g.: Real-time Transport Protocol (RTP) Sequencing: packets are re-ordered in real-time Intra-media synchronization: synchronization for a specific type of media Inter-media synchronization: synchronization between different type of media within the same session Frame identification: identify media logical units
Architecture of a MMS Basics perspective Definition of media-specific data definition and strategies for their efficient representation and storage
Basics Perspective Image representation and compression formats Dimensions, ppi, bit depth, GIF, PNG, JPEG, Audio representation and compression formats Bitrate, sampling frequency, quantization, Vorbis (OGG), MP3, AAC, Video compression formats Theora (OGV), ITU-T H.264, VP8 (WebM),
We focus on
We focus on This is a basic course, therefore we concentrate on: Basics level: Image and audio representation and storage formats Compression algorithms Systems level: Programming: implementation of image manipulation algotithms
Let s start! Good work to all of you!!!