Applications of User Modelling INF SERV Media Storage and Distribution Systems: User Modeling 8/9 3 Encoding Formats! Exploit limited awareness of users " JPEG/MPEG video and image compression " MP3 audio compression! Based on medical and psychological models Quality Adaptation! Adapt to changing " no models - need experiments Synchronity! Exploit limited awareness of users " no models - need experiments Access Patterns! When will users access a content?! Which content will users access?! How will they interact with the content? " no models, insufficient experiments - need information from related sources Why user modeling? Multimedia approach! If you can t make it, fake it Translation! Present real-life quality! If not possible, save resources where it is not recognizable Requirement! Know content and environment! Understand limitations to user perception! If these limitations must be violated, know least disturbing saving options User Perception of Quality Changes What? User Modelling! Formalized understanding of Why? " users awareness " user behaviour! Achieve the best price/performance ratio! Understand actual resource needs " achieve higher compression using lossy compression " potential of trading resources against each other " potential of resource sharing " relax relation between media Quality Changes Quality of a single stream! Issue in -on-demand, Music-on Demand,...! Not quality of an entire multimedia application Quality Changes! Usually due to changes in " overloaded server " congested network " overloaded client 1
Kinds of Quality Changes Kinds of Quality Changes Kinds of Quality Changes Kinds of Quality Changes " no back channel " no content adaptivity " continuous severe disruption " packet loss " frame drop " alleviated by protocols and codecs Kinds of Quality Changes Kinds of Quality Changes " no back channel " no content adaptivity " continuous severe disruption " change to another encoding format " change to another quality level " requires mainly codec work " packet loss " frame drop " alleviated by protocols and codecs " scaling of data streams " appropriate choices require user model
Kinds of Quality Changes " no back channel " no content adaptivity " continuous severe disruption " change to another encoding format " change to another quality level " requires mainly codec work " packet loss " frame drop " alleviated by protocols and codecs " scaling of data streams " appropriate choices require user model Planned quality changes! Long-term changes! Short-term changes " Use scalable encoding " Reduce short-term fluctuation by prefetching and buffering Two kinds of scalable encoding schemes! Non-hierarchical " encodings are more error-resilient o fractal single image encoding! Hierarchial " encodings have better compression ratios Scalable encoding! Support for prefetching and buffering is an architecture issue! Choice of prefetched and buffered data is not Planned quality changes Planned quality changes! Lots of research in scalable audio! No specific results for distribution systems! Rule-of-thumb " Always degrade video before audio! Long-term changes! Short-term changes! Long-term changes! Short-term changes " Use scalable encoding " Reduce short-term fluctuation by prefetching and buffering Short-term fluctuations! Characterized by " frequent quality changes " small prefetching and buffering overhead! Supposed to be very disruptive See for yourself Planned quality changes Planned quality changes! Long-term changes " Use separately encoded streams " Switch between formats " Non-scalable formats compress better than scalable ones (Source: Yuriy Reznik, RealNetworks)! Short-term changes Switching between formats! Needs no user modeling! Is an architecture issue 3
Subjective Assessment A test performed by the Multimedia Communications Group at TU Darmstadt Goal! Predict the most appropriate way to change quality Approach! Create artificial drop in layered video sequences! Show pairs of video sequences to testers! Ask which sequence is more acceptable Compare two means of prediction! Peak signal-to-noise ratio (higher is better) " compares degraded and original sequences per-frame " ignores order! Spectrum of layer changes (lower is better) " takes number of layer changes into account " ignores content and order Subjective Assessment How does the spectrum correspond with the results of the subjective assessment? Comparison with the peak signal-to-noise ratio Shape S(v) of shape 1 S(v) of shape PSNR of shape 1 PSNR of shape Average of Assessment.35 A B 6.86 4 6.86 61.46 63.15 48.1 49.4 66. 9.84 49.47 73.8 5.38 5.8 6.95 63.8 64.3.55.73 1.18 1..18 -.4 According to the results of the subjective assessment the spectrum is a more suitable measure than the PSNR E 1 G H K.5.5 L Subjective Assessment Subjective Assessment Used SPEG (OGI) as layer encoded video format amplitude of layer variation frequency of layer variation Conclusions! Subjective assessment of variations in layer encoded videos! Comparison of spectrum measure vs. PSNR measure " Observing spectrum changes is easier to implement " Spectrum changes indicate user perception better than PSNR " Spectrum changes do not capture all situations Missing! Subjective assessment of longer sequences! Better heuristics " "thickness" of " order to quality changes " target layer of changes Subjective Assessment What is better? First gap first or lowest gap first? User Model for Synchronity Early or late high quality? 4
Synchronization Content Relation! se.g.: several views of the same data Spatial Relations! Layout Temporal Relations! Intra-object Synchronization " Intra-object synchronization defines the time relation between various presentation units of one time-dependent media object! Inter-object Synchronization " Inter-object synchronization defines the synchronization between media objects Relevance! Hardly relevant in current NVoD systems! Somewhat relevant in conferencing systems! Relevant in upcoming multi-object formats: MPEG-4, Quicktime Synchronization Requirements Fundamentals 1% accuracy is not required, i.e., skew is allowed Skew depends on! Media! Applications Difference between! Detection of skew! Annoyance of skew Explicit knowledge on skew! Alleviates implementation! Allows for portability Inter-object Synchronization Lip synchronization! demands for a tight coupling of audio and video streams with! a limited skew between the two media streams Slide show with audio comment Main problem of the user model! permissible skew Experimental Set-Up Experiments at IBM ENC Heidelberg to quantify synchronization requirements for! /video synchronization! /pointer synchronization Selection of material! Duration " 3s in experiments " 5s would have been sufficient! Reuse of same material for all tests Introduction of artificial skew! By media composition with professional video equipment! With frame based granularity Experiments! Large set of test candidates " Professional: cutter at TV studios " Casual: every day user! Awareness of the synchronization issues! Set of tests with different skews lasted 45 min Inter-object Synchronization A lip synchronized audio video sequence (1 and ) is followed by a replay of a recorded user interaction (RI), a slide sequence (P1 - P3) and an animation (Animation) which is partially commented using an audio sequence (). Starting the animation presentation, a multiple choice question is presented to the user (Interaction). If the user has made a selection, a final picture (P4) is shown Lip Synchronization: Major Influencing Factors! Content " Continuous (talking head) vs. discrete events (hammer and nails) " Background (no distraction)! Resolution and quality! View mode (head view, shoulder view, body view) Main problem of the user model! permissible latency " analysing object sequence allow prefetching " user interaction complicates prefetching! Content! Background noise or music! Language and articulation 5
Lip Synchronization: Level of Detection Pointer Synchronization: Level of Detection Areas! In sync QoS: +/- 8 ms! Transient! Out of sync Observations! Difficult to detect out of sync " i.e., other magnitude than lip sync! Asymmetry " According to every day experience Lip Synch.: Level of Accuracy/Annoyance Pointer Synchronization: Level of Annoyance Some observations! Asymmetry! Additional tests with long movie " +/- 8 ms: no distraction " -4 ms, +16 ms: disturbing Areas! In sync: QoS -5 ms, +75 ms! Transient! Out of sync Pointer Synchronization Fundamental CSCW shared workspace issue Quality of Service of Two Related Media Objects Expressed by a quality of service value for the skew! Acceptable skew within the involved data streams! Affordable synchronization boundaries Analysis of CSCW scenarios! Discrete pointer movement (e.g. technical sketch )! Continuous pointer movements (e.g. route on map ) Most challenging probes! Short audio! Fast pointer movement Production level synchronization! Data should be captured and recorded with no skew at all! To be used if synchronized data will be further processed Presentation level synchronization! Reasonable synchronization at the user interface! To be used if synchronized data will not be further processed 6
Quality of Service of Two Related Media Objects Modelling Media Animation Images Text Mode, application Correlated Lip synchronization Overlay No overlay Overlay No overlay QoS +/- 1 ms +/- 8 ms +/- 4 ms +/- 5 ms +/- 4 ms +/- 5 ms User behaviour! The basis for simulation and emulation " In turn allows performance tests! Separation into " Frequency of using the VoD system " Selection of a movie User Interaction! Models exist " But are not verified so far Selection of a movie! Dominated by the access probability! Should be simulated by realistic access patterns Quality of Service of Two Related Media Objects Focus on -on-demand Media Animation Image Text Pointer Mode, application Event colleration Tightly coupled (stereo) Loosely coupled (dialog mode with various participants) Loosely coupled (background music) Tightly coupled (music with notes) Loosely coupled (slide show) Text annotation related to shown item QoS +/- 8 ms +/- 11 µs +/- 1 ms +/- 5 ms +/- 5 ms +/- 5 ms +/- 4 ms -5 - +75 ms -on-demand systems! Objects are generally consumed from start to end! Repeated consumption is rare! Objects are read-only! Hierarchical distribution system is the rule Caching approach! Simple approach first! Various existing algorithms Simulation approach! No real-world systems exist! Similar real-world situations can be adopted User Model for Access Patterns Using Existing Models Use of existing access models?! Some access models exist! Most are used to investigate single server or cluster behaviour! Real-world data is necessary to verify existing models Optimistic model! Cache hit probabilities are over-estimated! Caches are under-dimensioned! Network traffic is higher than expected Pessimistic model! Cache hit probabilities are under-estimated! Cache servers are too large or not used at all! Networks are overly large 7
Existing Data Sources for -on-demand Movie magazines! Data about average user behaviour! Represents large user populations! Small number of observation points (weekly) Movie rental shops! Actual rental operations! Serves only a small user population! Initial peaks may be clipped Cinemas! Actual viewing operations! Serves only a small user population! Few number of titles! Short observation periods Comparison with the Zipf Distribution rental probability 1.9.8.7.6.5.4.3..1 probability curves for 5 movie titles 4/ 6/ 96 Well-known and accepted model Easily computable Compatible with the 9:1 rule-of-thumb z(i) 4/3/96 4 6 8 1 movie index Model for Large User Populations N C z( i) =, C = 1/ 1/ j i j= 1 Zipf Distribution Verified for VoD by A. Chervenak! N - overall number of movies! i - movie i in a list ordered by descreasing popularities! z(i) - hit probability Verification: Small and Large User Populations Many application contexts! all kinds of product popularity investigations! http://linkage.rockefeller.edu/wli/zipf/ collects applications of Zipf s law " natural languages, monkey-typing texts, web access statistics, Internet traffic, bibliometrics, informetrics, scientometrics, library science, finance, business, ecological systems,... media control index Verification: Movie Magazine Movie magazine! Characteristics of observations on large user populations! Smoothness! Predictability of trends! Sharp increase and slower decrease in popularities 8 6 4 Highlander 3 5 1 15 5 weeks top 1 ranking 4 6 8 1 Highlander 3 5 1 15 5 weeks Verification: Small and Large User Populations Similarities! Small populations follow the general trends! Computing averages makes the trends better visible! Time-scale of popularity changes is identical! No decrease to a zero average popularity Differences! Large differences in total numbers! Large day-to-day fluctuations in the small populations Typical assumptions! 9:1 rule! Zipf distribution models real hit probability 8
Problems of Zipf Does not work in distribution hierarchies! Access to independent caches beyond first-level are not described Not easily extended to model day-to-day changes! Is timeless! Describes a snapshot situation Optimistic for the popularity of most popular titles Chris Hillman, bionet.info-theory, 1995! Any power law distribution for the frequency with which various combinations of letters appear in a sequence is due simply to a very general statistical phenomenom, and certainly does not indicate some deep underlying process or language. Rather, it says you probably aren t looking at your problem the right way! Verification: Zipf Variations Permutation model for day-to-day relevance changes popularity index change relevance change 1 8 6 4 5 1 15 5 popularity index change popularity index change relevance change of a real movie 1 8 6 4 5 1 15 5 relevance change of a real movie 1 8 6 4 5 1 15 5 Approaches to Long-term Development Model variations for long-term studies! Static approach " No long-term changes " Movie are assumed to be distributed in off-peak hours! CD sales model " Smooth curve with a single peak " Models the increase and descrease in popularity! Shifted Zipf distribution " Zipf distribution models the daily distribution " Shift simulates daily shift of popularities! Permutated Zipf distribution " Zipf distribution models the daily distribution " Permutation simulates daily shift of popularities Modelling: Requirements Model should represent movie life cycles! To reflect the aging of titles! To observe movement of movies through a hierarchy of servers! To make observations with respect to a single movie! To support the idea of pre-distribution Model should work for large and small user populations! To allow variations in client numbers! To prevent from built-in smoothing effects Model can not be trace-driven! The number of movies is too small! The observation time is too short! The user population size is not variable! One title can not be re-used without similarity effects Verification: Zipf Variations New Model: Movie Life Cycle Rotation model for day-to-day relevance changes relevance change of a real movie popularity index change relevance change 1 8 6 4 5 1 15 5 popularity index change popularity index change 1 8 6 4 5 1 15 5 relevance change of a real movie 1 8 6 4 5 1 15 5 Characteristics! Quick popularity increase! Various top popularities! Various speeds in popularity decrease! Various residual popularity 9
New Model: User Population Size movie hits movie hits 1.5 1.5 7 6 5 4 3 1 5 draws per day 5 1 15 5 days 5 draws per day 5 1 15 5 days Smoothing effect of larger user populations Day-to-day relevance changes Probability distribution of all movies by new releases movie hits movie hits 9 8 7 6 5 4 3 1 5 draws per day 5 1 15 5 7 6 5 4 3 1 days 5 draws per day 5 1 15 5 days References Ann Chervenak: Tertiary Storage: An Evaluation of New Applications, PhD thesis, University of California, Berkeley, 1994 Carsten Griwodz, Michael Bär, Lars Wolf: Long-Movie Popularity Models in -on-demand Systems, ACM Multimedia, Seattle, WA, USA, Nov. 1997 Charles Krasic, Jonathan Walpole: Priority-Progress Streaming for Quality-Adaptive Multimedia, ACM Multimedia Doctoral Symposium, Ottawa, Canada, Oct. 1 Ralf Steinmetz, Klara Nahrstedt: Multimedia Fundamentals, Volume I: Media Coding and Content INF Processing 57 media servers and (nd distribution Edition), systems Prentice Hall, 3, Carsten Griwodz ISBN & Pål Halvorsen Problems with Data Sources Lack of additional real-world data! No verification data for medium-sized populations available Missing details! Genres " Popularity rise and decline depends on genres " Single users behaviour can be predicted! Single day probability variations " Children s choices at daytime, adults choices at night! Regional popularity differences " Ethnic groups " Regional information! Comebacks " Sequels inspire comebacks Detail overload! Simplifications are required for large simulations Conclusion Simple Zipf models are not suited for simulation of server hierarchies Trace-driven simulation can not be used Our model is sufficient for general investigation on caching! Long-term movie life cycles can be modelled nicely! Optimistic assumptions due to smoothness are removed! Variations in movie behaviour are supported! Day-to-day popularity changes are realistic It is not sufficient yet for advanced caching mechanisms! Single-day variations are missing! Genres are missing 1