Multimedia Systems Giorgio Leonardi A.A Lecture 1: General information and Introduction

Transcription

1 Multimedia Systems Giorgio Leonardi A.A Lecture 1: General information and Introduction

2 Overview Course page (D.I.R.): Consulting: Office hours by appointment: Office #182 (in front of Sala Seminari ) me any time

3 Overview of this course Multimedia Systems cover a very large number of topics

4 Objectives of this course Understand the basic principles of multimedia: Signal representation and sampling Digital signals properties, elaboration and compression Mathematical Transforms Compression Algorithms Image representation and common image formats: GIF JPEG Image processing and filtering Sound representation and common sound formats: PCM MP3 Sound processing and filtering

5 Out of the scope of this course Audio and Image editing Software You re Computer Scientists, this is not a Photoshop course!!! Media authoring: Flash, HTML, Multimedia retrieval

6 Prerequisites None in particular, but Recommended: Basic math Basic programming skills

7 Organization of the course 48 hours total/24 lectures 18 face-to-face lectures: theory 6 lectures: work in lab: Exercises about image loading and processing

8 Schedule 09:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 Lunedì Martedì Mercoledì Giovedì Venerdì Dalle 11:00 alle 13:00 SISTEMI MULTIMEDIALI Leonardi Giorgio DiSIT aula (206?) Dalle 16:00 alle 18:00 SISTEMI MULTIMEDIALI Leonardi Giorgio DiSIT aula (206?)

9 Final exam Exam is split in two parts: 1. Short project: Implementation of a filter or of a processing image algorithm grade1 2. Written examination about theory and some exercise grade2 Final grade: weighted median of grade1 and grade2:

10 Finalm exam Part 1 (lab): Assigned during the last lecture in lab Take your time (not months!) to complete it and send it to me at any time We will discuss it briefly within a week from posting Meanwhile, contact me for any problem Part 2 (theory): Performed during the proper exam sessions No mid-term exams The two parts are taken separately and in any order, but must be both passed: grade1 18 and grade2 18 If only one of the two parts fails, the other grade can be maintained within one year.

11 Reference books Suggested Textbook Jennifer Burg The Science of Digital Media Prentice-Hall, [ Optional Textbook (in UPO s library) Ze-Nian Li and Mark S. Drew Fundamentals of Multimedia Prentice-Hall, [

12 Reference books Additional Books Nigel Chapman Digital Multimedia, 3rd Edition John Wiley & Sons, 2009 Fred Halsall (in UPO s library) Multimedia Communications: Applications, Networks, Protocols and Standards Addison-Wesley, 2001 [Out-of-Print]

13 A brief introduction to multimedia systems

14 Multimedia Multimedia is an overused term because It touches different disciplines/industries Engineering, Computer Science Telecommunications Industry TV & Radio Broadcasting Industry Consumer Electronics Industry

15 Multimedia Multimedia is an overused term because It means different things to different people PC vendor: PC that has great sound capability, a Blu-Ray drive, a 3D graphics card, Consumer entertainment vendor: interactive cable TV or video-on-demand TV-like services

16 What is Multimedia? Multimedia can have many definitions. These include (from the computer system perspective): Multimedia means that computer information can be represented through audio, video, and animation in addition to traditional media (i.e., text, graphics/drawings, images).

17 Multimedia It is not a new concept: Magazines: text + still images TV: audio + video Radio: audio + text (RDS) Contemporary opera: speech + video + sound WWW page: text + audio + video + images

18 General Definition A good general working definition: Multimedia is the field concerned with the computer controlled integration of text, graphics, drawings, still and moving images (Video), animation, audio, and any other media where every type of information can be represented, stored, transmitted and processed digitally.

19 Multimedia Application Definition of Multimedia Application: A Multimedia Application is a computer application which uses a collection of multiple media sources e.g. text, graphics, images, sound/audio, animation and/or video.

20 Multimedia application We are interested in digital multimedia, where media are digital (or digitized) documents Bits represent text, sound, pictures, Can be manipulated by computer programs Can be transmitted over computer networks Users can interact with media elements in novel ways For instance, by manipulating objects directly using the mouse

21 Multimedia Applications Examples of Multimedia Applications include: World Wide Web Multimedia Authoring, e.g. Adobe/Macromedia Director Hypermedia courseware Video-on-demand Interactive TV Computer Games Virtual reality Digital video editing and production systems Multimedia Database systems

22 Types of Media Media (pl. of medium) are means for communicating information It is used both in abstract and concrete terms Abstract: ways to communicate information Text, audio, Concrete: all of the various technologies we use to record information and transmit it to others Videotape and DVD-RW are recording media TV and radio broadcasting are transmission media

23 Types of Media

24 Types of Media: Static/Time-based Two different types of media: Static media: does not change with time Text Still images Time-based media: exhibits some change over time Moving images (animations) Video Sound

25 Types of Media: Linear/Nonlinear Media may be divided into linear and non-linear categories Linearity: information progresses sequentially from the beginning to the end The contents are arranged in a logical order of presentation from beginning to end Refers to the media creator s intention and cannot be changed or modified by the reader Nonlinearity: the flow of information is changed by user s interaction

26 Linear/Nonlinear Example: a book Fixed arrangement of text pages Book: physical arrangement of text and pages implies a linear reading order

27 Linear/Nonlinear Example: a film A fixed sorted array of frames Film: fixed order of frames defines a single playback sequence

28 Example: a hypertext E.g., Web pages Linear/Nonlinear Hypertext: Information flow can be changed by means of references (hyperlinks, aka anchors). Links between pages permit multiple arbitrary reading order

29 Media: Interactivity Interactive media is the ability of media to respond to the user s actions System is able to select media and information on the basis of the user s requests Web pages can automatically display information coming from other sources or that has been retrieved from a database Interactivity is enabled by nonlinearity

30 Interactive media Media: Interactivity

31 Characteristics of a Multimedia System A Multimedia system has four basic characteristics: 1. Multimedia systems must be computer controlled. 2. Multimedia systems are integrated. 3. The information they handle must be represented digitally. 4. The interface to the final presentation of media is usually interactive.

32 Key Issues for Multimedia Systems Key issues multimedia systems need to deal with: How to represent and store temporal information. How to strictly maintain the temporal relationships on play back/retrieval Data has to represented digitally: Analog Digital Conversion, Sampling etc. Large Data Requirements: bandwidth, storage, Data compression is usually mandatory

33 Features of a MMS Integration: MMS must be able to integrate contents of different media Digital movie with subtitles: audio + video + text Synchronization: MMS must be able to synchronize contents of different media Videoconference: audio + video

34 Architecture of a MMS

35 Architecture of a MMS The usage perspective studies: The type and design of multimedia applications, and The interface between users, multimedia applications and systems

36 Application Layer Applications Interactive Entertainment Audio / Video Teleconferencing E-learning Telemedicine Cooperative Work Environments Social Media

37 Architecture of a MMS Services perspective Services and protocols in multimedia networks and communication to send multimedia streams Synchronization of media streams Various system and service components to achieve the best end-to-end perceptual quality for the user

38 Example of service Group Communication Computer Supported Cooperative Work (CSCW) Videoconferencing Desktop sharing Architecture of a Group Communication System:

39 CSCW Service Rendezvous components describe processes for the organization of meetings and to set up group sessions Synchronous methods: expliclit invitation (e.g., multicast) Asynchronous methods: implicit invitation (e.g., by )

40 CSCW Service Conference components offer services to allow participants to communicate using multiple media streams and to support group communication Defines: establishment of a conference, add new participants, leave the conference,

41 CSCW Service Cooperation components describe techniques for concurrent replication of information among a large number of participants Application sharing: the resulting changes performed on a shared object are distributed to all participants

42 Architecture of a MMS Systems perspective This group of multimedia fields relates system areas such as processing, storage, and communication, and their relevant interfaces

43 Systems perspective Operating system Process management A time-critical process must never be subject to priority inversion A non-critical process should not suffer from starvation due to critical processes Hard real-time requirements: all requirements must be guaranteed Video process within a tele-surgery application Soft real-time requirements: some requirement can be violated Movie on-demand for entertainment purpose Soft real-time scheduling Memory and buffer management Provide access to data with a guaranteed timing delay end efficient manipulation functions

44 Systems perspective Networks Allow the exchange of media data between devices generating and processing them Require appropriate services and protocols Data have to be processed within a certain delay or during a defined time interval Small end-to-end delay E.g., ATM services (CBR, VBR, )

45 Systems perspective Communication User and Application requirements Data throughput: stream-like behavior of audio/video demands high data throughput Multicasting: to share resources at end-systems Processing and Protocol constraints Synchronization Error correction/detection E.g.: Real-time Transport Protocol (RTP) Sequencing: packets are re-ordered in real-time Intra-media synchronization: synchronization for a specific type of media Inter-media synchronization: synchronization between different type of media within the same session Frame identification: identify media logical units

46 Architecture of a MMS Basics perspective Definition of media-specific data definition and strategies for their efficient representation and storage

47 Basics Perspective Image representation and compression formats Dimensions, ppi, bit depth, GIF, PNG, JPEG, Audio representation and compression formats Bitrate, sampling frequency, quantization, Vorbis (OGG), MP3, AAC, Video compression formats Theora (OGV), ITU-T H.264, VP8 (WebM),

48 We focus on

49 We focus on This is a basic course, therefore we concentrate on: Basics level: Image and audio representation and storage formats Compression algorithms Systems level: Programming: implementation of image manipulation algotithms

50 Let s start! Good work to all of you!!!