Information Systems University of Koblenz Landau, Germany Semantic Multimedia Management - Multimedia Processes Steffen Staab http://isweb.uni-koblenz.de With acknowledgements to Lynda Hardman and Raphael Troncy (cf. their WWW-08 tutorial)
Overview of Canonical Processes staab@uni-koblenz.de 2
Example: CeWe Color PhotoBook Application for authoring digital photo books Automatic selection, sorting and ordering of photos Context analysis methods: timestamp, annotation, etc. Content analysis methods: color histograms, edge detection, etc. Customized layout and background Print by the European leader photo finisher company http://www.cewe-photobook.com staab@uni-koblenz.de 3
CeWe Color PhotoBook Processes My winter ski holidays with my friends staab@uni-koblenz.de 4
CeWe Color PhotoBook Processes staab@uni-koblenz.de 5
CeWe Color PhotoBook Processes staab@uni-koblenz.de 6
CeWe Color PhotoBook Processes staab@uni-koblenz.de 7
CeWe Color PhotoBook Processes staab@uni-koblenz.de 8
CeWe Color PhotoBook Processes staab@uni-koblenz.de 9
Canonical Processes 101 Canonical: reduced to the simplest and most significant form possible without loss of generality Formalization of each process in UML diagrams Process Process artifacts Process actors External world artifacts staab@uni-koblenz.de 10
Premeditate Process where initial ideas about media production are established Design a photo book of my last holidays for my family Create argument-based sequences of videos of interviews after September 11 <<external world artifact>> Ideas, decisions and artifacts * <<output>> Premeditate <<process artifact>> Premeditate Artifacts <<inv olv es>> 1..* <<process actor>> Premeditate Actor staab@uni-koblenz.de 11
Create Media Asset Process where media assets are captured, generated or transformed <<process actor>> Creation Actor 1..* <<process artifact>> Premeditate Artifacts (from Premeditate) <<inv olv es>> <<process artifact>> Message (from Construct Message) Create Media Asset <<output>> 1..* <<media asset>> Media Asset Capturing Generation Transforming <<inv olv es>> 1..* <<process actor>> Capturing Device <<inv olv es>> 1..* <<process actor>> Generation Program <<inv olv es>> 1..* <<process actor>> Editing Program staab@uni-koblenz.de 12
Annotate Process where annotation is created <<anchor>> Artifact Anchor * * subject subject * * <<process artifact>> <<output>> A ny Process Artifact * Annotate <<inv olv es>> 1..* <<process actor>> Annotation Actor 1..* * <<annotation>> Artifact Annotation <<process actor>> Human Annotator <<process actor>> Computing Annotator staab@uni-koblenz.de 13
Semantic Annotate The annotation uses some controlled vocabularies Subject matter annotations of your photos Rhetorical annotations in Vox Populi <<process artifact>> A ny Process A rtifact * Annotate <<output>> 1..* <<annotation>> Artifact Annotation <<schema>> Vocabulary Semantic Annotate <<output>> <<annotation>> Semantic Artifact Annotation link <<term>> Vocabulary Term described in term s of staab@uni-koblenz.de 14
Package Process where process artifacts are logically and physically packed <<process artifact>> Any Process Artifact * contains * <<external world artifact>> Input given by a user * Package <<output>> 1..* <<composite artifact>> Multimedia Package Physical Package Logical Package staab@uni-koblenz.de 15
Query Process where a user retrieves a set of process artifacts based on a given query <<external world artifact>> User Query Input <<process artifact>> Any Process Artifact Query <<inv olv es>> <<output>> <<composite artifact>> Set of Process Artifacts 1..* <<process actor>> Query User staab@uni-koblenz.de 16
Construct Message Process where an author specifies the message they wish to convey Our holiday was sporty, great weather and fun Create clash about whether war is a good thing <<external world artifact>> Ideas, decisions and artifacts Construct Message <<output>> <<process artifact>> Message <<inv olv es>> 1..* <<process actor>> Message Author staab@uni-koblenz.de 17
Organize Process where process artifacts are organized according to the message Organize a number of 2-page layouts in photobook Use semantic graph to select related video clips to form linear presentation of parts of argument structure <<process artifact>> Message (from Construct Message) 0..1 Organize <<output>> <<com posite artifact>> Document Structure <<process artifact>> Any Process Artifact contains staab@uni-koblenz.de 18
Publish Process where final content and user interface is created <<composite artifact>> Document Structure (from Organize) <<input >> <<media asset>> Media Asset for Publication Publish <<output>> <<composite artifact>> Published Document <<annotation>> Annotation for Publication <<inv olv es>> <<process actor>> Publisher staab@uni-koblenz.de 19
Distribute Process where final interaction between end-users and produced media occurs <<com posite artifact>> Published Document (from Publish) Distribute <<output>> <<external world artifact>> Real World Output <<inv olv es>> <<process actor>> Distribution Hardware and Software staab@uni-koblenz.de 20
Canonical Processes Possible Flow staab@uni-koblenz.de 21
Sum Up Community agreement, not yet another model Large proportion of the functionality provided by multimedia applications can be described in terms of this model Initial step towards the definition of open web-based data structures for describing and sharing semantically annotated media assets staab@uni-koblenz.de 22
Discussion Frequently asked questions Complex processes Interaction Complex artifacts and annotations can be annotated Towards a more rigorous formalization of model Relationship to foundational ontologies Semantics of Annotations staab@uni-koblenz.de 23
Literature Lynda Hardman: Canonical Processes of Media Production. In Proceedings of the ACM Workshop on Multimedia for Human Communication - From Capture to Convey (MHC 05), November 2005. Special Issue on Canonical Processes of Media Production http://www.ifi.uio.no/mmsj/upcomming.html http://www.cwi.nl/~media/projects/canonical/ Lynda Hardman, Zeljko Obrenovic, Frank Nack, Brigitte Kerhervé and Kurt Piersol: Canonical Processes of Semantically Annotated Media Production. In Multimedia Systems Journal, 2008 (to appear) Philipp Sandhaus, Sabine Thieme and Susanne Boll: Canonical Processes in Photo Book Production. In Multimedia Systems Journal, 2008 (to appear) Stefano Bocconi, Frank Nack and Lynda Hardman: Automatic generation of video documentaries. In Journal of Web Semantics, 2008 (to appear). staab@uni-koblenz.de 24