Multimedia Authoring Chapter 1 1IT342
Outlines What is Multimedia? Components of Multimedia Multimedia Research Topics and Projects Multimedia and Hypermedia Multimedia Authoring metaphors Multimedia Production Multimedia Presentation Some Technical Design Issues Automatic Authoring 2IT342
What is Multimedia? When different people mention the term multimedia, they often have quite different, or even opposing, viewpoints. A PC vendor: a PC that has sound capability, a DVD-ROM drive, and perhaps the superiority of multimedia-enabled microprocessors that understand additional multimedia instructions. A consumer entertainment vendor: interactive cable TV with hundreds of digital channels available, or a cable TV-like service delivered over a high-speed Internet connection. A Computer Science (CS) student: applications that use multiple modalities, including text, images, drawings (graphics), animation, video, sound including speech, and interactivity. Multimedia and Computer Science: Graphics, HCI, visualization, computer vision, data compression, graph theory, networking, database systems. Multimedia and Hypermedia 3IT342
What is Multimedia? A good general definition is: Multimedia is a combination of text, graphic, sound, animation, and video that is delivered interactively to the user by electronic or digitally manipulated 4IT342
Why to use Multimedia? According to Dr. Albert Mehrabian, a specialist in interpersonal communication at the University of California, says: People recall 20% of what they see 40% of what they see & hear 70% of what they see, hear & do Multimedia allows for seeing, hearing and doing 5IT342
Components of Multimedia Text Animation Video Multimedia Components Audio Images 6IT342
Components of Multimedia Multimedia involves multiple modalities of text, audio, images, drawings, animation, video and Interactivity. Examples of how these modalities are put to use: Video teleconferencing Distributed lectures for higher education Using voicerecognition to build an interactive environment Tele-medicine Searching in (very) large video and image databases for target visual objects Making multimedia components editable Augmented reality: placing real-appearing computer graphics and video objects into scenes. ex 7IT342
Multimedia Research Topics and Projects To the computer science researcher, multimedia consists of a wide variety of topics: 1. Multimedia processing and coding: multimedia content analysis, content-based multimedia retrieval, multimedia security, audio/image/video processing, compression, etc. 2. Multimedia system support and networking: network protocols, Internet, operating systems, servers and clients, quality of service (QoS), and databases. 3. Multimedia tools, end-systems and applications: hypermedia systems, user interfaces, authoring systems. 4. Multi-modal interaction and integration: ubiquity webeverywhere devices, multimedia education including Computer Supported Collaborative Learning, and design and applications of virtual environments. 8IT342
Current Multimedia Projects Many exciting research projects are currently underway. Here are a few of them: Camera-based object tracking technology: Tracking of the control objects provides user control of the process. 3D motion capture: Used for multiple actor capture so that multiple real actors in a virtual studio can be used to automatically produce realistic animated models with natural movement. Multiple views: Allowing photo-realistic (video-quality) synthesis of virtual actors from several cameras or from a single camera under differing lighting. Specific multimedia applications: Aimed at handicapped persons with low vision capability and the elderly a rich field of endeavor. 9IT342
Current Multimedia Projects Many exciting research projects are currently underway. Here are a few of them: Digital fashion: Aims to develop smart clothing that can communicate with other such enhanced clothing using wireless communication, so as to artificially enhance human interaction in a social setting. Electronic House call system: An initiative for providing interactive health monitoring services to patients in their homes Augmented Interaction applications: Used to develop interfaces between real and virtual humans for tasks such as augmented storytelling. 10
Multimedia and Hypermedia History of Multimedia: Newspaper: perhaps the first mass communication medium, uses text, graphics, and images. Motion pictures: conceived of in 1830 s in order to observe motion too rapid for perception by the human eye. Wireless radio transmission: Guglielmo Marconi, at Pontecchio, Italy, in 1895. Television: the new medium for the 20th century, established video as a commonly available medium and has since changed the world of mass communications. The connection between computers and ideas about multimedia covers what is actually only a short period 11
Hypermedia and Multimedia A hypertext system: meant to be read nonlinearly, by following links that point to other parts of the document, or to other documents (Fig. 1.1) HyperMedia: not constrained to be text-based, can include other media, e.g., graphics, images, and especially the continuous media sound and video. The World Wide Web (WWW) the best example of a hypermedia application. Multimedia means that computer information can be represented through audio, graphics, images, video, and animation in addition to traditional media. 12
Hypermedia and Multimedia 13
Fig 1.1: Hypertext is nonlinear 14
Examples of typical present multimedia applications include: Digital video editing and production systems. Electronic newspapers/magazines. World Wide Web. On-line reference works: e.g. encyclopedia, games, etc. Home shopping. Interactive TV. Multimedia courseware. Video conferencing. Video-on-demand. Interactive movies. 15
Multimedia Software Tools Music Sequencing and Notation Cakewalk, Cubase, Macromedia Soundedit Digital Audio Cool Edit, Sound Forge, Pro Tools Graphics and Image Editing Adobe Illustrator, Adobe Photoshop, Macromedia Fireworks, Macromedia Freehand Video Editing Adobe Premiere, Adobe After Effects, Final Cut Pro Animation Multimedia APIs: Java3D, DirectX, OpenGL Rendering Tools: 3D Studio Max, Softimage XSI, Maya, RenderMan Multimedia Authoring Macromedia Flash, Macromedia Director, Authorware, Quest 16
Multimedia Authoring Authoring is the process of creating multimedia applications Metaphors are methodologies employed by the authoring system to accomplish its task (create multimedia applications). Some common authoring metaphors are: 1. Scripting language metaphor 2. Slide show metaphor 3. Hierarchical metaphor 4. Iconic/flow-control metaphor 5. Frames metaphor 6. Card/scripting metaphor 7. Cast/score/scripting metaphor 17
Multimedia Authoring Metaphors 1. Scripting Language Metaphor: use a special language to enable interactivity (buttons, mouse, etc.), and to allow conditionals, jumps, loops, functions/macros etc. -- load an MPEG file extfilename of MediaPlayer "thempegpath" = "c:\windows\media\home33.mpg"; -- play extplaycount of MediaPlayer "thempegpath" = 1; -- put the MediaPlayer in frames mode (not time mode) extdisplaymode of MediaPlayer "thempegpath" = 1; -- if want to start and end at specific frames: extselectionstart of MediaPlayer "thempegpath" = 103; extselectionend of MediaPlayer "thempegpath" = 1997; -- start playback get extplay() of MediaPlayer "thempegpath"; 18
Multimedia Authoring Metaphors 2. Slide Show Metaphor: A linear presentation by default, although tools exist to perform jumps in slide shows, EX. PowerPoint or ImageQ. 2. Hierarchical Metaphor: User-controllable elements are organized into a tree structure often used in menu-driven applications. 19
Multimedia Authoring Metaphors 4. Iconic/Flow-control Metaphor: Graphical icons are available in a toolbox, and authoring proceeds by creating a flow chart with icons attached (Fig. 2.1), EX. Authorware by Macromedia. As well as simple flowchart elements, such as an IF statement, a CASE statement, group of elements using MAP (subroutine). Also, simple animation is possible. 20
Multimedia Authoring Metaphors 5. Frames Metaphor: Like Iconic/Flow-control Metaphor; however links between icons are more conceptual, rather than representing the actual flow of the program (Fig. 2.2), EX. Quest by Allen Communication. Flowchart consists of modules composed of frames. Frames are constructed from objects, such as text, graphics, audio, animations and video, all of which can respond to events. 21
Multimedia Authoring Metaphors 6. Card/Scripting Metaphor: Uses a simple index-card structure easy route to producing applications that use hypertext or hypermedia; used in schools. e.g., SuperCard, HyperCard by Apple 22
Multimedia Authoring Metaphors 7. Cast/Score/Scripting Metaphor: Time is shown horizontally; like a spreadsheet: rows, or tracks, represent instantiations of characters in a multimedia production. Multimedia elements are drawn from a cast of characters, and scripts are basically event-procedures or procedures that are triggered by timer events. Director, by Macromedia, is the chief example of this metaphor. Director uses the Lingo scripting language, an object-oriented event-driven language. 23
Multimedia Authoring Metaphors 7. Cast/Score/Scripting Metaphor: Cast Score 24
Multimedia Production People produce multimedia involve an art director, graphic designer, production artist, producer, project manager, writer, user interface designer, sound designer, videographer, and 3D and 2D animators, as well as programmer. Multimedia production design phase consists of storyboarding, flowcharting, prototyping and user testing as well as a parallel production of media. A storyboard depicts the initial idea content of a multimedia concepts in a series of sketches (keyframes). A flowchart organizes the storyboards by inserting navigation information (multimedia concept s structure and user interaction). A prototype includes development of detailed functional specification (walk-through, screen action, user interface). User testing is extremely important before final development phase (tools may be used). 25
Multimedia Presentation Features that affect presenting multimedia content : 1. Graphics Styles: Human visual dynamics are considered in regard to how such presentations must be constructed. 1. Color principles and guidelines: Some color schemes and art styles are best combined with a certain theme or style. A general hint is to not use too many colors, as this can be distracting. A color contrast program can be used based on the concept: If the text color is some triple(r,g,b), a legible color for the background is that color subtracted from the maximum (here assuming max=1): (R, G, B) (1 R, 1 G, 1 B) This is called the principal complementary color, for color values in the range 0 to 1 2. Fonts: For effective visual communication in a presentation, it is best to use large fonts (i.e., 18 to 36 points), and no more than 6 to 8 lines per screen (fewer than on this screen!). 26
Li & Drew Fig. 2.4: Colours and fonts [from Ron Vetter]. 27
Fig. 2.5: Program to investigate colours and readability. 28
Fig. 2.6 shows a colour wheel, with opposite colours equal to (1-R, 1-G, 1-B) 29
Multimedia Presentation 2. Sprite Animation: sprites are special graphic objects that can move independently of the screen background. based on, Suppose we have an animation figure, as in Fig. 1 (a). Now create a 1-bit mask M, as in Fig. 1 (b), black on white, and accompanying sprite S, as in Fig. 1 (c). Fig. 1: Sprite creation: Original, mask image M, and sprite S ( Duke figure courtesy of Sun Microsystems.) 30
Multimedia Presentation We can overlay the sprite on a colored background B, as in Fig. 2 (a) by first ANDing B and M, and then ORing the result with S, with final result as in Fig. 2 (e). Fig. 2: Sprite animation: (a): Background B. (b): Mask M. (c): B AND M. (d): Sprite S. (e): B AND M OR S 31
Multimedia Presentation 32
33
Multimedia Presentation 3. Video transitions: is an effective way to indicate a change to the next section. It is a semantic means to signal scene changes and often carry semantic meaning. Many different types of transitions: Cut: an abrupt change of image contents formed by abutting two video frames consecutively. This is the simplest and most frequently used video transition. 34
Multimedia Presentation Wipe: a replacement of the pixels in a region of the viewport with those from another video. Wipes can be left-to-right, right-to-left, vertical, horizontal, like an iris opening, swept out like the hands of a clock, etc. 35
Multimedia Presentation Dissolve: replaces every pixel with a mixture over time of the two videos, gradually replacing the first by the second. Most dissolves can be classified as two types: Cross dissolve (Type I) Dither dissolve (Type II) 36
Multimedia Presentation Cross dissolve (Type I): Blends the first clip into the second clip. Every pixel is affected gradually. 37
Multimedia Presentation Dither dissolve (Type II) Determined by α(t), increasingly more and more pixels in video A will abruptly (instead of gradually as in Type I) change to video B. 38
Multimedia Presentation Fade-in and fade-out are special types of Type I dissolve: video A or B is black (or white). Wipes are special forms of Type II dissolve in which changing pixels follow a particular geometric pattern. 39 http://www.mediacollege.com/video/editing/transition/types.html
Some Technical Design Issues Computer Platform: Much software is ostensibly portable but cross-platform software relies on run-time modules which may not work well across systems. Video format and resolution: The most popular video formats NTSC, PAL, and SECAM are not compatible, so a conversion is required before a video can be played on a player supporting a different format. Memory and Disk Space Requirement: At least 128 MB of RAM and 20 GB of hard-disk space should be available for acceptable performance and storage for multimedia programs. Delivery Methods: CD-ROMs: may be not enough storage to hold a multimedia presentation. As well, access time for CD-ROM drives is longer than for hard-disk drives. Electronic delivery is an option, but depends on network bandwidth at the user side (and at server). A streaming option may be available, depending on the presentation. 40
Automatic Authoring Facilitating automatic authoring is either by creating new multimedia presentations or by automatic creation of more useful multimedia documents from existing resources. Hypermedia documents: Generally, three steps are to produce documents to be viewed nonlinearly: 1. Capture of media: From text or using an audio digitizer or video frame-grabber; is highly developed and well automated. 2. Authoring: How best to structure the data in order to support multiple views of the available data, rather than a single, static view. 3. Publication: i.e. Presentation, is the objective of the multimedia tools we have been considering. 41
Automatic Authoring Externalization versus linearization: (a) the essential problem involved in communicating ideas without using a hypermedia mechanism (i.e., linearization). 42
(b) In contrast, hyperlinks allow us the freedom to partially mimic the author s thought process (i.e., externalization). 43
Using, e.g., Microsoft Word, creates a hypertext version of a document by following the layout already set up in chapters, headings, and so on. But problems arise when we actually need to automatically extract semantic content and find links and anchors (even considering just text and not images etc.) Fig. 2.13 displays the problem. 44
Fig. 2.13: Complex information space [from David Lowe]. Once a dataset becomes large we should employ database methods. The issues become focused on scalability (to a large dataset), maintainability, addition of material, and reusability. 45
Automatic Authoring Semi-automatic migration of hypertext The structure of hyperlinks for text information is simple: nodes represent semantic information and these are anchors for links to other pages. Fig. 2.14: Nodes and anchors in hypertext [from David Lowe]. 46
Automatic Authoring Hyperimages We need an automated method to help us produce true hypermedia: Fig. 2.15: Structure of hypermedia [from David Lowe]. 47
Can manually delineate syntactic image elements by masking image areas. Fig. 2.16 shows a hyperimage, with image areas identified and automatically linked to other parts of a document: Fig. 2.16: Hyperimage [from David Lowe]. 48
Summary Automatic Authoring: an advanced helper for creating new multimedia presentations or a mechanism to facilitate automatic creation of more useful multimedia documents from existing sources Multimedia Authoring Metaphors: for easier understanding of the methodology employed to create multimedia applications. Multimedia Production: how to produce multimedia Multimedia Presentation: effects for presenting multimedia content as well as guidelines for content design 49