Development of Multimedia WebApp on Tizen Platform

Size: px

Start display at page:

Download "Development of Multimedia WebApp on Tizen Platform"

Bethany Poole
6 years ago
Views:

1 Development of Multimedia WebApp on Tizen Platform 1. HTML Multimedia 2. Multimedia Playing with HTML5 Tags (1) HTML5 Video (2) HTML5 Audio (3) HTML Pulg-ins (4) HTML YouTube (5) Accessing Media Streams and Playing (6) Multimedia Contents Mgmt (7) Capturing Images 3. Multimedia Processing Web Device API -1-

2 1. HTML Multimedia What is Multimedia? Multimedia comes in many different formats. It can be almost anything you can hear or see. Examples: Pictures, music, sound, videos, records, films, animations, and more. Web pages often contain multimedia elements of different types and formats. Multimedia Formats Multimedia elements (like sounds or videos) are stored in media files. The most common way to discover the type of a file, is to look at the file extension. When a browser sees the file extension.htm or.html, it will treat the file as an HTML file. The.xml extension indicates an XML file, and the.css extension indicates a style sheet file. Pictures are recognized by extensions like.gif,.png and.jpg. Multimedia files also have their own formats and different extensions like:.swf,.wav,.mp3,.mp4,.mpg,.wmv, and.avi. -2-

3 2. Multimedia Playing with HTML5 Tags (1) HTML5 Video Some of the popular video container formats include the following: Audio Video Interleave (.avi) Flash Video (.flv) MPEG 4 (.mp4) Matroska (.mkv) Ogg (.ogv) Browser Support -3-

4 Common Video Format Format File Description MPEG AVI WMV QuickTime RealVideo Flash.mpg.mpeg.avi.wmv.mov.rm.ram.swf.flv MPEG. Developed by the Moving Pictures Expert Group. The first popular video format on the web. Used to be supported by all browsers, but it is not supported in HTML5 (See MP4). AVI (Audio Video Interleave). Developed by Microsoft. Commonly used in video cameras and TV hardware. Plays well on Windows computers, but not in web browsers. WMV (Windows Media Video). Developed by Microsoft. Commonly used in video cameras and TV hardware. Plays well on Windows computers, but not in web browsers. QuickTime. Developed by Apple. Commonly used in video cameras and TV hardware. Plays well on Apple computers, but not in web browsers. (See MP4) RealVideo. Developed by Real Media to allow video streaming with low bandwidths. It is still used for online video and Internet TV, but does not play in web browsers. Flash. Developed by Macromedia. Often requires an extra component (plug-in) to play in web browsers. Ogg.ogg Theora Ogg. Developed by the Xiph.Org Foundation. Supported by HTML5. WebM MPEG-4 or MP4.web m.mp4 WebM. Developed by the web giants, Mozilla, Opera, Adobe, and Google. Supported by HTML5. MP4. Developed by the Moving Pictures Expert Group. Based on QuickTime. Commonly used in newer video cameras and TV hardware. Supported by all HTML5 browsers. Recommended by YouTube. Only MP4, WebM, and Ogg video are supported by the newest HTML5 standard. -4-

5 Video Tag Until now, there has not been a standard for showing a video/movie on a web page. Today, most videos are shown through a plug-in (like flash). However, different browsers may have different plug-ins. HTML5 defines a new element which specifies a standard way to embed a video/movie on a web page: the <video> element. The control attribute adds video controls, like play, pause, and volume. Text between the <video> and </video> tags will only display in browsers that do not support the <video> element. The <video> element allows multiple <source> elements. <source> elements can link to different video files. The browser will use the first recognized format. <video width="320" height="240" controls="controls"> <source src="movie.mp4" type="video/mp4" /> <source src="movie.ogg" type="video/ogg" /> Your browser does not support the video tag. </video> To start a video automatically use the autoplay attribute: <video width="320" height="240"autoplay> -5-

6 Example <!DOCTYPE html> <html> <body> <div style="text-align:center"> <button onclick="playpause()">play/pause</button> <button onclick="makebig()">big</button> <button onclick="makesmall()">small</button> <button onclick="makenormal()">normal</button> <br /> <video id="video1" width="420"> <source src="mov_bbb.mp4" type="video/mp4" /> <source src="mov_bbb.ogg" type="video/ogg" /> Your browser does not support HTML5 video. </video> </div> <script type="text/javascript"> var myvideo=document.getelementbyid("video1"); function playpause(){ if (myvideo.paused) myvideo.play(); else myvideo.pause(); } function makebig(){ myvideo.width=560; } function makesmall(){ myvideo.width=320; } function makenormal(){ myvideo.width=420; } </script> <p>video courtesy of <a href=" target="_blank">big Buck Bunny</a>.</p> </body> </html> -6-

7 HTML Audio and Video DOM Reference The HTML5 DOM has methods, properties, and events for the <audio> and <video> elements. These methods, properties, and events allow you to manipulate <audio> and <video> elements using JavaScript. HTML Audio/Video Methods Method addtexttrack() canplaytype() load() play() pause() Description Adds a new text track to the audio/video Checks if the browser can play the specified audio/video type Re-loads the audio/video element Starts playing the audio/video Pauses the currently playing audio/video -7-

8 HTML Audio/Video Properties Property audiotracks autoplay buffered controller controls crossorigin currentsrc currenttime defaultmuted defaultplaybackrate duration ended Description Returns an AudioTrackList object representing available audio tracks Sets or returns whether the audio/video should start playing as soon as it is loaded Returns a TimeRanges object representing the buffered parts of the audio/video Returns the MediaController object representing the current media controller of the au dio/video Sets or returns whether the audio/video should display controls (like play/pause etc.) Sets or returns the CORS settings of the audio/video Returns the URL of the current audio/video Sets or returns the current playback position in the audio/video (in seconds) Sets or returns whether the audio/video should be muted by default Sets or returns the default speed of the audio/video playback Returns the length of the current audio/video (in seconds) Returns whether the playback of the audio/video has ended or not -8-

9 Property error loop mediagroup muted networkstate paused playbackrate played preload readystate seekable seeking src startdate texttracks videotracks volume Description Returns a MediaError object representing the error state of the audio/video Sets or returns whether the audio/video should start over again when finished Sets or returns the group the audio/video belongs to (used to link multiple audio/video el ements) Sets or returns whether the audio/video is muted or not Returns the current network state of the audio/video Returns whether the audio/video is paused or not Sets or returns the speed of the audio/video playback Returns a TimeRanges object representing the played parts of the audio/video Sets or returns whether the audio/video should be loaded when the page loads Returns the current ready state of the audio/video Returns a TimeRanges object representing the seekable parts of the audio/video Returns whether the user is currently seeking in the audio/video Sets or returns the current source of the audio/video element Returns a Date object representing the current time offset Returns a TextTrackList object representing the available text tracks Returns a VideoTrackList object representing the available video tracks Sets or returns the volume of the audio/video -9-

10 HTML Audio/Video Events Event abort canplay canplaythrough durationchange emptied ended error loadeddata loadedmetadata loadstart pause play playing progress Description Fires when the loading of an audio/video is aborted Fires when the browser can start playing the audio/video Fires when the browser can play through the audio/video without stopping for b uffering Fires when the duration of the audio/video is changed Fires when the current playlist is empty Fires when the current playlist is ended Fires when an error occurred during the loading of an audio/video Fires when the browser has loaded the current frame of the audio/video Fires when the browser has loaded meta data for the audio/video Fires when the browser starts looking for the audio/video Fires when the audio/video has been paused Fires when the audio/video has been started or is no longer paused Fires when the audio/video is playing after having been paused or stopped for b uffering Fires when the browser is downloading the audio/video -10-

11 Event ratechange seeked seeking stalled suspend timeupdate volumechange waiting Description Fires when the playing speed of the audio/video is changed Fires when the user is finished moving/skipping to a new position in the audio/video Fires when the user starts moving/skipping to a new position in the audio/video Fires when the browser is trying to get media data, but data is not available Fires when the browser is intentionally not getting media data Fires when the current playback position has changed Fires when the volume has been changed Fires when the video stops because it needs to buffer the next frame -11-

12 (2) HTML5 Audio Browser Supports Browser MP3 Wav Ogg Internet Explorer YES NO NO Chrome YES YES YES Firefox YES YES YES Safari YES YES NO Opera YES YES YES <!DOCTYPE html> <html> <body> <audio controls> <source src="horse.ogg" type="audio/ogg"> <source src="horse.mp3" type="audio/mpeg"> Your browser does not support the audio element. </audio> </body> </html>

13 (3) HTML5 Plugs-in HTML Helpers (Plug-ins) Helper applications are computer programs that extend the standard functionality of a web browser. Helper applications are also called plug-ins. Examples of well-known plug-ins are Java applets. Plug-ins can be added to web pages with the <object> tag or the <embed> tag. Plug-ins can be used for many purposes: display maps, scan for viruses, verify your bank id, etc. The <object> Element The <object> element is supported by all browsers. The <object> element defines an embedded object within an HTML document. It is used to embed plug-ins (like Java applets, PDF readers, Flash Players) in web pages. <!DOCTYPE html> <html> <body> <object width="400" height="50" data="bookmark.swf"></object> </body> </html> -13-

14 The <embed> Element The <embed> element is supported in all major browsers. The <embed> element also defines an embedded object within an HTML document. Web browsers have supported the <embed> element for a long time. However, it has not been a part of the HTML specification before HTML5. The element will validate in an HTML5 page, but not in an HTML 4 page. <!DOCTYPE html> <html> <body> <embed width="400" height="50" src="bookmark.swf"> </body> </html> <!DOCTYPE html> <html> <body> <embed src="audi.jpeg"> </body> </html> -14-

15 (4) HTML5 YouTube Videos Playing a YouTube Video in HTML Upload the video to YouTube (automatic format conversion by YouTube!!) Take a note of the video id Define an <iframe> element in your web page Let the src attribute point to the video URL Use the width and height attributes to specify the dimension of the player Add any other parameters to the URL YouTube Parameters autohide Value 0: The player controls are always visible. Value 1: The player controls hides automatically when the video plays. Value 2 (default): If the player has 16:9 or 4:3 ratio, same as 1, otherwise same as 0. autoplay Value 0 (default): The video will not play automatically when the player loads. Value 1: The video will play automatically when the player loads. -15-

16 controls Value 0: Player controls does not display. The video loads immediately. Value 1 (default): Player controls display. The video loads immediately. Value 2: Player controls display, but the video does not load before the user initiates playback. Loop Value 0 (default): The video will play only once. Value 1: The video will loop (forever). playlist A comma separated list of videos to play (in addition to the original URL). <!DOCTYPE html> <html> <body> <object width="420" height="315" data=" control=1> </object> </body> </html> -16-

17 (5) Accessing Media Streams and Playing Retrieving information about media files from the device s file system is achieved through using the ContentManager. In order to do this function, the application needs the correct privileges. Adding the following URIs to the Privileges section of the config.xml file will grant both read and write access for your application: tizen.org/privilege/content.read tizen.org/privilege/content.write Discovering Multimedia Contents list all the public media directories through the getdirectories function: (simple version) var success = function(directories) { directories.foreach(function(directory){ console.log( directory.directoryuri); }); }; var error = function(e) { alert(' Error: ' + e.message); }; tizen.content.getdirectories(success, error); -17-

18 Accessing Media Streams getusermedia: Getting access to local devices that can generate <body> multimedia streams (W3C Editor's Draft 30 November 2011) <h1>getusermedia tutorial</h1> <video id="videoview" src="" autoplay></video> <br /> <img id="imgview" src=""> <canvas id="canvasview" style="display: none;" width="300" height="225"></canvas> <br> <input type="button" value="webcamstart" onclick="getvideostream();" id="btnplay"> <input type="button" value="capture" onclick="getcapture();" id="btncapture"> </body> <script> var localstream = ""; function getcapture() { if (localstream) { var canvas = document.getelementbyid("canvasview"); var context = canvas.getcontext('2d'); context.drawimage(document.getelementbyid("videoview"), 0, 0, 300, 225); document.getelementbyid("imgview").src = canvas.todataurl('image/webp'); } } function getvideostream() { navigator.webkitgetusermedia({video: true}, SuccessCallBack, ErrorCallBack); } function SuccessCallBack(stream) { var URL = window.webkiturl; document.getelementbyid("videoview").src = URL.createObjectURL(stream); localstream = stream; } function ErrorCallBack(e) { console.log(e); } </script> -18-

19 Playing Multimedia Content 1 Create the video element and buttons used to control the play and pause: <div class="media"> <video id="video src="media/video_sample.mp4"> </video> <div> <button id="v-play" type="button">play</button> <button id="v-pause" type="button" disabled>pause</button> </div> </div> 2 Define the button functions. Play and pause the media file using the play() and pause() methods of the HTMLMediaElement interface <script> var play_button = document.getelementbyid("v-play"); var pause_button = document.getelementbyid("v-pause"); play_button.addeventlistener("click", function(){video.play(); /* Play movie */}, false); pause_button.addeventlistener("click", function(){video.pause(); /* Pause movie */}, false); -19-

20 3 Apply event listeners to detect button events an react to them var video = document.getelementbyid("video"); video.addeventlistener("play", function() { play_button.disabled = true; /* Play button disabled*/ pause_button.disabled = false; /* Pause button enabled */ }, false); video.addeventlistener("pause", function() { play_button.disabled = false; /* Play button enabled */ pause_button.disabled = true; /* Pause button disabled */ }, false); video.addeventlistener("ended", function() { play_button.disabled = false; /* Play button enabled */ pause_button.disabled = true; /* Pause button disabled */ }, false); video_element_and_html5_the_audio_element/play_pause.html -20-

21 Retrieving Media Information <body> <h1>video and audio tutorial</h1> <div class="media"> <h2>play Time Example</h2> <video id="video" src="media/video_sample.mp4" controls preload="auto" width="340 height="220"> </video> <div class="log"> Current Play Time : <span id="currenttime"></span></div> <div class="log"> Total Play Time : <span id="totaltime"></span> </div> </div> </body> <script> var video = document.getelementbyid("video"); var currenttime = document.getelementbyid("currenttime"); var totaltime = document.getelementbyid("totaltime"); //realize loadedmetadata event (indicate total duration) video.addeventlistener("loadedmetadata", function () { var totalsec = video.duration; //total video duration var time = secondstotime(totalsec); //Import function to change seconds to Time(HH:MM:SS) format //Display the calculated time on screen totaltime.innerhtml = parseint(totalsec) + "sec (" + time.h + ":" + time.m + ":" + time.s + ")"; }, false); -21-

22 //realize timeupdate event (indicate play time status) video.addeventlistener("timeupdate", function () { var currsec = video.currenttime; //currernt video time var c_time = secondstotime(currsec); //seconds 를 Time( 시 : 분 : 초 ) format 으로변경하는 function 호출 //Display current play time currenttime.innerhtml = parseint(currsec) + "sec (" + c_time.h + ":" + c_time.m + ":" + c_time.s + ")"; }, false); function secondstotime(secs) { var hours = Math.floor(secs / (60 * 60)); var divisor_for_minutes = secs % (60 * 60); var minutes = Math.floor(divisor_for_minutes / 60); var divisor_for_seconds = divisor_for_minutes % 60; var seconds = Math.ceil(divisor_for_seconds); //Set as json type s object var obj = { "h": inttostring(hours), "m": inttostring(minutes), "s": inttostring(seconds) }; return obj; } //Ensure that each time is set in 2 characters according to the time format. function inttostring(time) { return (parseint(time) < 10? ("0" + time) : time); } </script> -22-

23 Displaying Progress State <body> <h1>video and audio tutorial</h1> <div class="media"> <h2>progress Example</h2> <video id="video" src="media/video_sample.mp4" width="340" height="220" controls poster="media/poster_sample.png"></video> <div>progress : <span id="state"></span></div> </div> </body> <script> var video = document.getelementbyid("video") //Realizes progress event video.addeventlistener("progress", function () { //Returns a TimeRanges object representing the buffered parts of the media var buffered = video.buffered; // returns time in seconds of the last buffered TimeRa var buffered_end = buffered.end(0); //Change to progress rate % var progress = (buffered_end / video.duration) * 100; //Display progress rate document.getelementbyid("state").innerhtml = parseint(progress) + "%" }, false); </script></html> The buffered attribute of the video object returns a TimeRanges object that represents the downloaded buffering range. Use the end property of the TimeRanges object to find out the end time (in seconds) of the buffered range: -23-

24 Moving the Timeline Position <div class="media"> <video id="video" src="media/video_sample.mp4" controls></video> <div> <button id="v-back">5 Sec Backward</button> <button id="v-forw">5 Sec Forward</button> </div> <div><button id="v-goto">go to 40 Sec</button></div> </div> <script> var video = document.getelementbyid("video"); /* Video object */ var back_button = document.getelementbyid("v-back"); /* Back button */ var forw_button = document.getelementbyid("v-forw"); /* Forward button */ var go_button = document.getelementbyid("v-goto"); /* Jump button */ /* Move 5 seconds back */ back_button.addeventlistener("click", function() { video.currenttime -= 5 }, false); /* Move 5 seconds forward */ forw_button.addeventlistener("click", function() { video.currenttime += 5 }, false); /* Move timeline to 40 seconds */ go_button.addeventlistener("click", function() { video.currenttime = 40 }, false); </script> -24-

25 (6) Multimedia Contents Management Content API Tizen enables you to search for content (images, videos, music or other) located on the local device storage. You can also perform content management tasks, such as viewing and updating content attributes. The Content API is mandatory for both Tizen mobile and wearable profiles, which means that it is supported in all mobile and wearable devices. The main features of Content API include: 1) Content Retrieval : The Content API uses the same ContentManager interface instance for all content-related functionalities. getdirectories() and find() methods of the ContentManager interface. : browse and search for content directories and content When searching for content items, you can create attribute filters, attribute range filters, and composite filters based on specific filter attributes of the ContentManager interface. You can also sort the search results. 2) Content Management : view and edit content item details (common file information and metadata attributes of the media file). find() : To view content item details update() : To update some attributes of a content item 3) Content Change Notification 4) Playlist Mgmt -25-

26 Content: Discovering Multimedia Content (Such as Images, Videos, or Music) 1 Retrieve the ContentManager interface instance using the tizen global object: var manager = tizen.content; 2 To search for the content directories in the local device, use the getdirectories() method of the ContentManager interface. The method returns an array of ContentDirectory objects. function ondirectoryarraysuccess(directories) { for (var i = 0; i < directories.length; i++) { /* Retrieve folder-specific information */ } } manager.getdirectories(ondirectoryarraysuccess); 3 To search for the content items in all directories, use the find() method of the ContentManager interface. The method returns an array of Content objects. function oncontentitemarraysuccess(contents) { for (var i=0; i < contents.length; i++) { console.log(i + ":" + contents[i].type + ":" + contents[i].title + ":" + contents[i].mimetype); } } var contenttype = "VIDEO"; var filter = new tizen.attributefilter("type", "EXACTLY", contenttype); manager.find(oncontentitemarraysuccess, null, null, filter); -26-

27 ContentManager Example A simple three pages application Lists media directories Lists contents of a selected directory Overview/preview page for a single piece of content Retrieving information about media files from the device s file system is achieved through using the ContentManager. Adding the following URIs to the Privileges section of the config.xml file will grant both read and write access for your application: tizen.org/ privilege/content.read tizen.org/ privilege/content.write -27-

28 -28-

29 -29-

30 -30-

31 -31-

32 (7) Capturing Images getusermedia API To make use of the device s camera and microphone hardware, your application can use the HTML5 Media Capture and Streams feature. However, at the time of writing, the JavaScript API in the Web Runtime uses a vendor-prefixed version of the function name (webkit). Future releases will deprecate the vendor prefix and eventually the prefix will be removed altogether from the function name. To make your code work with both the prefixed and unprefixed versions, you can assign them back to the navigator object. The following JavaScript code shows how to do this: navigator.getusermedia = navigator.getusermedia navigator.webkitgetusermedia; getusermedia API accepts three parameters, constratins (type: object) : The media types that should support the LocalMediaStream interface (used in the success callback). successcallback (type: function) : A function handler invoked when a request for a media device is successful. An instance of LocalMediaStream is passed as the first parameter. errorcallback (type : function) : A function handler invoked when a request for the media device fails. -32-

33 Creating a Simple Camera Application -33-

34 add some JavaScript that defines a new object (Camera) that first requests access to the media stream and handles the success and error callbacks: a request is made to navigator.getusermedia, which is passed references to the success and error callback functions attached to the current Camera instance. Notice that a call to bind is made to each callback function, with this passed as the only parameter. This ensures that when each of the functions is called, it retains the same context as the Camera instance. -34-

35 Success Handler Inordertooutputthestreamtoawebpage, youneedtocreateanelementthatcanhappily contain it. TheHTML5<video>tagfitsthebillperfectlyandthatiswhatisinitiallyhappeninginsidethe camerasuccesscallback. Notice that the src property is assigned the value of window.webkiturl.createobjecturl, which is a reference to the file object of the stream. Once the element has been created it is appended to the video container. Lastly, event listeners are bound to the two control buttons. -35-

36 Error Handler The error callback is passed an instance of NavigatorUserMediaError containing the type of error generated. The error is evaluated and an appropriate alert message is shown. In your production-ready applications, you ll obviously handle errors better than simply displaying an alert message. -36-

37 Take a still picture from the current video stream. creates a new < canvas > element with width and height properties matching the < video > element s dimensions. it uses the jquery function, passing a string of the element tag as the first parameter and a JavaScript object full of options as the second parameter. The return value is a jquery object containing the newly created element. Using the raw < canvas > element, the code gets the two-dimensional drawing context and calls the drawimage function to create the still image. An image is then rendered using the video element specified at the top-left corner of the rectangle, with the same dimensions as the video element itself. In order to obtain the raw image data, the function proceeds to get the base64-encoded version of the image. Then it creates a new filename in the format of IMAGE_[ TIMESTAMP]. png and continues to use the file system API to save the new file. The first thing the code does is to resolve the destination directory path to an images folder inside the current working directory. Inside the success handler for this call, a new file is created inside the directory, and then, finally, the base64-encoded image data is written to that stream. -37-

38 HTML5 Video Player Comparison ( -38-

HTML5: MULTIMEDIA. Multimedia. Multimedia Formats. Common Video Formats

HTML5: MULTIMEDIA. Multimedia. Multimedia Formats. Common Video Formats LEC. 5 College of Information Technology / Department of Information Networks.... Web Page Design/ Second Class / Second Semester HTML5: MULTIMEDIA Multimedia Multimedia comes in many different formats.