Multimedia Information Systems

Transcription

1 Multimedia Information Systems VU ( ) Vedran Sabol KTI, TU Graz Nov 09, 2015 Vedran Sabol (KTI, TU Graz) Multimedia Information Systems Nov 09, / 67

2 Outline 1 Introduction 2 Drawing in the Browser (SVG, 3D) 3 Audio and Video 4 Javascript APIs 5 JavaScript Changes Vedran Sabol (KTI, TU Graz) Multimedia Information Systems Nov 09, / 67

3 HTML5 - Part II Multimedia Information Systems VU ( ) Vedran Sabol KTI, TU Graz Nov 09, 2015 Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

4 Drawing in the Browser (SVG, 3D) SVG Scalable Vector Graphics (SVG) Developed by W3C ( Web standard for vector graphics (as opposed to canvas - raster-based) XML application (SVG DTD) SVG is supported by all current browsers Editors Inkscape and svg-edit (Web App) Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

5 Drawing in the Browser (SVG, 3D) SVG Features Basic shapes: rectangles, circles, ellipses, path, etc. Painting: filling, stroking, etc. Text Example - simple shapes Grouping of basic shapes Transformation: translation, rotation, scale, skew Example - grouping and transforms Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

6 Drawing in the Browser (SVG, 3D) SVG Features Colors: true color, transparency, gradients, etc. Clipping, masking Filter effects Interactivity: user events Scripting, i.e. JavaScript, supports DOM Animation: attributes, transforms, colors, motion (along paths) Raster images may be embedded (JPEG, GIF,...) Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

7 Drawing in the Browser (SVG, 3D) SVG Examples and Tutorials Example: stars and stripes from Wikipedia Geoinformation systems Yosemiti Further resources W3C SVG Reference MDN SVG Element Reference SVG Basics Tutorial SVG Tutorial Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

8 Drawing in the Browser (SVG, 3D) 3D in browser Many less successful approaches in the past: VRML, Java3D Separate projects by Google, Mozilla, etc., now Khronos Group: Google, Mozilla, Apple, Opera Javascript interface to OpenGL Based on OpenGL ES 2.0 (OGL subset) Since OpenGL drivers are installed everywhere browser native 3D without plugins WebGL uses canvas element for the graphics context Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

9 Drawing in the Browser (SVG, 3D) WebGL: examples WebGL examples 22 Experimental WebGL Demo Examples Three.js library: Examples Quake 2 Chrome experiments Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

10 Drawing in the Browser (SVG, 3D) WebGL Previously weak browser support Improving: support (at least partial) by all browsers in the latest version 3D applications possible using WebGL and other HTML5 technologies Further reading Learning WebGL WebGL Lessons ChemDoodle Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

11 Audio and Video Video and Audio New elements for video and audio It works only if the format is compatible and if the codecs are integrated in the browser Format wars H.264 most commonly used video format covered by patents (MPEG LA) supported by (almost) all browser-platform combinations Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

12 Audio and Video Video and audio Initially recommended format Ogg (Theora - video, Vorbis - audio) Free and open protocols (Xiph.Org foundation) not supported by Microsoft and Apple WebM avoids MPEG LA patents originaly VP8 - video, Vorbis - audio recently updated to VP9 - video, Opus - audio Royalty-free, open video and audio for HTML5 Support by Google, Opera, Mozilla Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

13 Audio and Video Video Specify multiple sources in different formats, browser picks format Not advisable: browser detection - complicated, error prone (versions, platforms)... <video poster="video.jpg" width="640" height="360" controls> <source src="video.mp4" type="video/mp4" /> <source src="video.webm" type="video/webm" /> <source src="video.ogv" type="video/ogg" /> Browser does not support HTML5 video. </video>... Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

14 Audio and Video Video: further info and examples As with any other element you can use Javascript to manipulate the element (script your own controls) You can use CSS to style videos, etc. Attributes: autoplay, controls, loop, muted, poster HTML5Video.org by Kaltura YouTube - browser support YouTube example Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

15 Audio and Video Audio... <audio> <source src="test.mp3" type="audio/mpeg" /> <source src="test.ogg" type="audio/ogg" /> </audio>... Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

16 Audio and Video Audio: further info and examples Similarly to video you can manipulate the audio element with Javascript and CSS JPlayer, JPlayer Audio Cross-platform player library for HTML5 (with Flash fallback) Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

17 HTML5 and Javascript HTML5 is not only markup but combination of markup, presentation (CSS3) and behavior (Javascript) New Javascript APIs Evolved in the last 10 years of the Web development Possibility to write Web Apps Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

18 APIs Geolocation: retrieve geographical location (subject to user approval) AppCache: specify which files to cache (available when offline) FileSystem API: access to a sandboxed file system Storage APIs: go far beyond what is possible with cookies Web Storage, Web SQL, Indexed DB Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

19 APIs Web Workers: background running scripts WebSocket: bi-directional communication protocol Drag & Drop History: API for manipulating browser history Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

20 Geolocation Collects all available location information WLAN, GSM, Internet provider Retrieves the geographical coordinates latitude, longitude, accuracy (optionally: altitude, speed, heading) Forward that info to a Geo data provider, e.g. Google Maps Applications, e.g. find local information Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

21 Geolocation Privacy concerns However, browsers typically ask for permissions Better protection of privacy then e.g. smartphones Example if (navigator.geolocation) { navigator.geolocation.getcurrentposition(function(mypos) {... } } Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

22 AppCache Standard browser cache unreliable AppCache: makes Web applications accessible offline Possibility to specify which files should (or should not) be cached Advantages Faster browsing as cached resources are served locally Reduced server and network load Offline browsing of the full site Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

23 AppCache To enable the cache for your application you need to specify manifest attribute in every HTML page you want to cache <html manifest="example.appcache">... </html> Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

24 AppCache In the manifest file you specify which files to cache CACHE MANIFEST index.html stylesheet.css images/logo.png scripts/main.js Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

25 AppCache CACHE MANIFEST CACHE: /favicon.ico index.html stylesheet.css images/logo.png scripts/main.js NETWORK: login.php /myapi FALLBACK: /main.py /static.html images/large/ images/offline.jpg *.html /offline.html Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

26 AppCache Cached content will be shown until until one of the following happens Cache is cleared (user) Manifest file is changed on the server (comment change will do) Programatically via API check update is ready, swap cache, reload page Further Reading var appcache = window.applicationcache; appcache.update();... if (appcache.status == window.applicationcache.updateready) { appcache.swapcache(); window.location.reload(); } Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

27 FileSystem API Files and directories: creating, navigating, writing, reading, renaming, moving, deleting etc. Complex, powerful API As of today full support in Chrome and recently Opera Access to a sandboxed section of the user s local file system Web apps cannot see each other files or the native file system Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

28 FileSystem API Access to a sandboxed file system: LocalFileSystem API LocalFileSystem asynchronous API (and LocalFileSystemSync synchronous API for WebWorkers) Implemented by Window (and WorkerGlobalScope) Create (TEMPORARY or PERSISTENT) storage of requested size window.requestfilesystem(type, size, oninit, onerror) Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

29 FileSystem API Navigating directory structure DirectoryReader: read the entries in a directory FileEntry, DirectoryEntry: represents a file/directory, properties, manipulation (remove, copy, move...) Support by Chrome and Opera (W3C stopped standardization) Further reading: Exploring the FileSystem APIs window.requestfilesystem(type, 1024, oninit(fs) { var dirreader = fs.root.createreader(); dirreader.readentries(function(results) {... Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

30 File API Reading and manipulating files FileList: sequence of selected Files File: represents a file and its properties retrieved from FileList, DataTransfer object (DnD), or FileEntry Blob: access to raw binary data FileReader: asynchronous reading of File/Blob For reading local files (e.g. uploding, creating thumbnails...) Creating and writing: FileWriter (FileEntry.createWriter()) For use with FileSystem API Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

31 FileReader FileReader allows you to read local files via JavaScript Support by all current browsers Single files, multiple files, Blobs (slices of binary files) All reading operations can be executed asynchronously read-methods: readasarraybuffer/asbinarystring/astext/asdataurl Callbacks: onabort, onerror, onloadstart, onload, onloadend, onprogress Further reading: Reading Files in JavaScript Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

32 FileList/File FileList returns File objects obtained by user selecting files using the input element (i.e. file upload) or from a drag and drop operation s DataTransfer object With Javascript you can add additional client-side logic Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

33 FileList/File filtering files of type image.* var files = evt.target.files; // FileList object for (var i = 0, f; f = files[i]; i++) { if (!f.type.match( image.* )) { continue; }... } Further reading: File API (W3C Specification) Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

34 Web Storage Web Storage: simple key-value local data storage window.localstorage: persistent storage (no expiration date) Session-related data: sessionstorage object Example cleared on the tab close/browser shutdown // store localstorage.setitem("mykey", "myvalue"); // store console.log(localstorage.getitem("mykey")); // retrieve localstorage.removeitem("mykey"); // delete Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

35 Web Storage Replaces cookies Advantages better API, faster, more capacity more secure (no data transfer to server) excelent browser support (from IE8) Disadvantages no transactions, no indexing and searching not suitable for large data Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

36 Web SQL Web SQL: fully fledged relational database (incl. transactions) Future uncertain: W3C stopped specification development Not supported by IE, Edge and FireFox var db = window.opendatabase("dbname", "1.0", "description", 5*1024*1024); //5MB db.transaction(function(tx) { tx.executesql("select * FROM test", [], successcallback, errorcallback); }); Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

37 Indexed DB Indexed DB is somewhere between Web SQL and Web Storage Key-value store for records (simple values and hierarchical objects) Indexing for faster searching Synchronous and asynchronous APIs Transaction support Support for larger data sets Lately very good browser support (IE without full feature set) Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

38 Indexed DB Create Object Stores Persist Javascript Objects in the store Multiple Indexes for querying (w/o SQL) Extensive and complex API (W3C Reference) Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

39 Web Workers Javascript is a single threaded environment Prevented applications of being ported to the client side Porting to client side might improve performance and scalability Web workers bring threading to Javascript Spawn background threads in your Web applications UI remains responsive while processing runs in background Support by all browsers Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

40 Web Workers Applications (Large) data processing and number crunching Code syntax highlighting or other real-time text formatting Spell checker Image filtering in <canvas> Analyzing video or audio data Updating many rows of a local web database Background data fetching Prefetching and/or caching data for later use Background I/O or polling of web services Processing large arrays or humongous JSON responses Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

41 Web Workers Spawn a worker: var worker = new Worker( dowork.js ); worker.postmessage(); // start web worker Stopping a worker: worker.terminate(); // kill worker from main page self.stop(); // from inside the worker Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

42 Web Workers Data communication between main thread and workers via messages Messages are copied by serialization (not shared): strings or JSON objects Background threads cannot modify the DOM Instead workers post messages to let their creator (main page) do this Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

43 Web Workers Worker script: task.js self.addeventlistener( message, function(e) { self.postmessage("working..."); do something... self.postmessage(e.data); }, false); Main script: var worker = new Worker( task.js ); worker.addeventlistener( message, function(e) { console.log( Worker replied:, e.data); }, false); worker.postmessage( Hello ); Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

44 Web Workers Example 1: background processing Example 2: multiple workers Tutorial: WebWorkers Basics Demos Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

45 WebSocket Request/response cycle of HTTP: client-initiated Long Polling: simulation of server-initiated connection AJAX: asynchronous HTTP requests Javascript callbacks when server initiates connection Still the request needs to come from the client Polling strategy for data updates Client have to reconnect periodically High latency problems (HTTP overhead) Low latency apps impossible (multiplayer FPS) Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

46 WebSocket WebSocket: bi-directional client-server communications Independent protocol over a TCP connection ( ws://... ) Secure connection ( wss://... ) HTTP compatible handshake process If successful: TCP connection remains open Message passing possible in both directions var wsconnection = new WebSocket( wss://echo.websocket.org );... wsconnection.close() Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

47 WebSocket wsconnection.onopen = function () { connection.send( Ping ); }; wsconnection.onerror = function (error) { console.log( WebSocket Error + error); }; wsconnection.onmessage = function (e) { console.log( Server: + e.data); }; wsconnection.onclose = function (e) { console.log( Connection closed ); }; Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

48 WebSocket Supported by all current browsers Cross origin communication possible Server side implementations available in Java, Python,.NET etc. New usage pattern with many connections - problems for some servers Upgrade from HTTP to Web Socket protocoll during handshake to pass proxies Still, dropped connections by proxies and load balancers possible Fallback to AJAX may be necessary Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

49 WebSocket Applications Multiplayer online games Chat applications Live sports ticker Realtime updating social streams Example Further Reading Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

50 Drag & Drop Native browser drag & drop We need to define elements draggable (attribute) <div class="column" draggable="true"> <header>a</header> </div> <img src="smiley.png" draggable="true"> DataTransfer object used to specify the data and type Handle events: ondragstart, ondragover, ondrop Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

51 Drag & Drop ondragstart: invoked when the drag is initiated DataTransfer.setData(mimeType, data) ondragover: specifies where the dragged data can be dropped to allow a drop prevent element default behaviour (DnD disabled): event.preventdefault() ondrop: invoked when the data is dropped append data into the target element Example Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

52 History History API for manipulating the browser history stack Extends windows.history.go()/back()/forward() History stack stores all URLs that a user visited Clicks on links that execute Javascript and update the page: history stack does not reflect this situation history.pushstate and popstate manipulate the stack and recover a previous state Example Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

53 JavaScript Changes Ecma JavaScript language is standardized by ECMA European Computer Manufacturers Association A lot of controversy about Javascript Adobe left the consortium and developed ActionScript The main reason: static types ECMA Script 5 (2011) and 6 (2015) without static typing Previous version, ECMA Script 3 since, 1999 Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

54 JavaScript Changes JavaScript Major Changes JavaScript Strict mode JSON changes Arrays manipulation Further reading: New in ECMAScript 5 Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

55 JavaScript Changes JavaScript v5 Major Changes use strict directive at the beginning of the script to enable Strict mode: better error handling, disables poorly thought of features Cannot use a variable without declaring it Cannot delete a variable, function or argument Cannot define a property more than once Cannot write a read-only property Cannot duplicatie parameter names... turn language design issues into outright errors not backwards compatible with ECMA Script 3 Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

56 JavaScript Changes JSON Parsing Previously JSON parsing with eval() function Executes any JS code within the script Slow and problems with security JSON.parse() converts a serialised JSON string to a JS object var jsonstring="{ name : Maurer }"; var obj = JSON.parse(jsonstring); Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

57 JavaScript Changes JSON Filtering and Transforming JSON.parse(text[, reviver]) reviver: transformes the value before returning it If reviver() returns: a valid value, then member value will be replaced null or undefined, then the member is deleted var cool=[ name ]; var jsonstring="{ name : Maurer, address : Inffeldfasse }"; var obj = JSON.parse(jsonstring, reviver); var reviver=function(key, value){ if(cool.indexof(key) >= 0) return value; } Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

58 JavaScript Changes JSON Serialization var jsonstring=json.stringify(obj[, replacer [, space]); Writing out a JSON string replacer can be one of the two: function(key, value): transforming values array: values indicate names of the properties that will be included space: pretty printing (indenting) Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

59 JavaScript Changes Array manipulation New functions for searching and manipulation of arrays (previously tedious) indexof() and lastindexof() for searching foreach() for iteration and performing an action every() and some() check elements on satisfying a condition Transformations: filter() returns a filtered array map() returns a modified array reduce() returns an accumulated result Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

60 JavaScript Changes Array manipulation data = [0,1,2,1,3,2]; data.indexof(2) // 2 data.lastindexof(1) // 3 data.indexof(4) // -1 var sum = 0; data.foreach(function(value) { sum += value; }); // 9 Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

61 JavaScript Changes Array manipulation data = [10,25,50,90,5]; data.every(function(x) { return x < 100; }) // true data.every(function(x) { return x % 2 === 0; }) // false data.filter(function(x) { return x < 50 }); // [10,25,5] data.map(function(x) { return 2*x; }); // [20,50,100,180,10]; data.reduce(function(x,y) { return (x>y)?x:y; }, 0) // 90 Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

62 JavaScript Changes JavaScript v6 Major Changes Support for classes, incl. constructors and inheritance Define modules and manage dependencies (import, export) Block scoping using let keyword Constants and more... Further reading: New in ECMAScript 6 Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

63 JavaScript Changes HTML5 Demos and Resources Learning W3C - HTML5 Introduction MDN - HTML5 Web Developer Guide Browser compatibility ECMAScript 5 compatibility table Demos Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

64 JavaScript Changes CSS3 Changes in selectors but also in various properties Webfonts, text-wrapping, native support for columns Text strokes, opacity, different color models (HSV) Rounded corners, gradients, shadows, background, borders Transitions, transformations, animations Example Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

65 JavaScript Changes CSS3 Many changes are browser specific E.g. -webkit-transform, -moz-transform -ms-transform Prefixes should be droped as specification matures (CR status) Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

66 JavaScript Changes MathML Language for inclusion of mathematical expressions in Web pages W3C MathML Previously only as images No interaction Interaction, presentation with Javascript and CSS Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67

67 JavaScript Changes MathML Language for inclusion of mathematical expressions in Web pages W3C MathML Previously only as images No interaction Interaction, presentation with Javascript and CSS Firefox native support Test suite Interaction example Vedran Sabol (KTI, TU Graz) HTML5 - Part II Nov 09, / 67