Position Statement for Multimodal Workshop

Size: px

Start display at page:

Download "Position Statement for Multimodal Workshop"

Maryann Houston
5 years ago
Views:

1 Psitin Statement fr Multimdal Wrkshp Stéphane H. Maes Directr f Architecture Mbile, Vice and Wireless and Advanced Technlgies Oracle Crpratin 1. Persnal Interest Stéphane has several years f experience and histry with multimdal interactins. At IBM< He pineered numerus activities, including speech and speaker recgnitin technlgy, vice brwsing (ML), embedded speech recgnitin, multi-channel and device-independent authring and servers (XFrms, XSLT, ), multimdal cmputing and cnversatinal cmputing and their use in mbile r car envirnments. In his current rle at Oracle Crpratin, Stéphane versees strategy, research and develpment f tls and middleware (declarative and imperative, JSP and JSF, prtlets) t supprt multi-channel applicatins, multimdal and multi-device applicatins as well as applicatins ptimized fr specific interactin channels including: J2ME, MS.NET Cmpact Framewrk, SALT, device-specific / Ecmascript MP, WML, XFrms/, ViceXML (brwser specific and brwser-independent), JSPs, JSFs and prtlets. During the years, Stéphane has created, chaired, actively participated, ften as editr, r mnitred related standard activities including: SVAP/SRAPI, W3C (XFrms, XSLT,, I, Vice, DI, WS), OASIS, WS-I IETF (SPEECHSC, DSR), ETSI (DSR), 3GPP (SES, multimdal), ITU (DSR), WAP / OMA (Multimdal and multi-device enabler). In particular, Stéphane is the champin fr the multimdal and multi-device enabler activity at OMA. In additin, he is editr f the OMA requirement and architecture dcuments fr multimdal and multidevice. In the rest f this psitin statement, we present ur view f the wrk dne at OMA. It reflects the psitin f Oracle Crpratin and it shuld nt be cnstrued as a liaisn frm OMA. OMA specificatins may ultimately differ and address ther bjectives r rely n different design pints than the cnsideratins presented belw. 2. Multi-mdal and Multi-device supprt by Mbile Infrastructure The OMA wrk fcuses n enabling supprt by the OMA Service Envirnment (OSE) architecture f multimdal interactins. It strives t supprt:

2 The different authring appraches currently prpsed in the industry including the W3C I wrk The different implementatin and deplyments mdels fr multimdal brwsers including sequential switches, thin and fat clients with lcal r distributed cnversatinal engines. 2.1 Multimdal and Multi-device Executin Mdel This sectin describes a basic flw fr multimdal and multi-device interactins, believed t supprt the different use cases and requirements identified in the OMA multimdal and multi-device Requirement Dcument. It supprts the fundamental executin mde f multimdal and multi-device applicatins. It is envisaged that OMA multimdal and multi-device services satisfy this executin mdel. It is f key interest that the industry as a whle endrses such a mdel. In this dcument, the term user agent is used lsely t designate the cmpnent that renders the presentatin data int physical effects that can be perceived and interacted with by the user. Fr a given mdality this may be a separate brwser r platfrm r ne r multiple cmpnents internal t a brwser r platfrm. The fundamental executin mdel f multimdal and multi-device applicatins is the fllwing: A user interactin in ne f the available mdalities (user agent) results int interactin events These events are passed t a synchrnizatin manager that handles a representatin f the interactin event and determines the impact f the interactin based n the state f the applicatin and synchrnizatin rules. This in turns results int updates f the state f the applicatin and update instructins sent t all the registered mdalities (user agents) available t the user. This is summarized in Figure 1, where each user agent may represent a different mdality (e.g. ViceXML brwser and -MP brwser r GUI and Vice Java applicatins) r different devices (e.g. smart phne and PDA r kisk). User Interactin 1 2 Optinal Immediate update Mdality r Dev a A (User )... Mdality r Dev a B (User ) 3 Exchange f interactin 6 Update f the different mdalities r device Rule Updates 5 4 Applicatin f a rules Figure 1 Fundamental executin mdel f multimdal r multi-device applicatins; independent f prgramming mdel r cnfiguratin.

3 The user actin may r may nt result int an immediate update f the affected mdality state prir t the synchrnizatin (step 2). Immediate updates prvide faster respnse in the mdality that the user currently uses, but may lead t prblems r cnfusing behavirs fr example with cmpsite inputs, cncurrent r almst cncurrent inputs in different mdalities and cnversatinal multi-mdal applicatin (i.e. where inputs are t be understd r disambiguated first). Other events than user input may als trigger interactin events t transmit thrugh step (3). When the service uses dialg management, it wuld typically be respnsible fr establishing and managing the multimdal synchrnizatin. The fllwing basic architecture implements this executin mdel (Figure 2). Vice User Figure 2 architecture t supprt multimdal and multi-device interactins illustrated fr vice and GUI interactin. Each f these mdule r prtins f these mdules may be partitined r cmbined n a single device r distributed acrss several devices r servers. 2.2 Assciated Mbile Deplyment Cnfiguratins Figure 3 t Figure 8 present examples f multimdal r multi-device cnfiguratins that implement the multimdal and multi-device executin mdel and architecture discussed in sectin 2.1. Except fr the multi-device cnfiguratin described in Figure 8, the figures illustrate multimdal interactins with vice and GUI. Nthing impses these mdality r t limit the synchrnizatin t tw mdalities r devices. Except fr the sequential cnfiguratin, they can supprt any synchrnizatin granularity authrized by applicatin, netwrk r user. The speech recgnitin framewrk (SRF) refers t a framewrk currently studied by 3GPP (3GPP TR Feasibility Study fr Enabled Services (Release 6). The speech recgnitin framewrk (SRF) enables t distribute the audi sub-system and the speech services by sending encded speech and meta-infrmatin between the client and the server ver a packed switched netwrk. The SRF may use cnventinal cdecs like AMR r Distributed Recgnitin (DSR) ptimized cdecs. The SRF can be deplyed ver a packet switched (PS) netwrk. Over a generic PS netwrk, SRF will require: Uplink and dwnlink transprt f audi (e.g. RTP) Sessin establishment, signalling and cntrl Cdec negtiatins Quality f service negtiatin and prvisining

4 Service administratin. The distributin f prcessing fr ther mdality may require extensin similar framewrks. Audi Sub-system Vice Brwser Netwrk ViceXML http Presentatin Figure 3 Example f sequential cnfiguratin (n vice and data supprt simultaneusly) fr vice and GUI interactin. This cnfiguratin des nt require SRF: it can be deplyed n 2G r 2.5G netwrks. Only ne mdality is available at a given mment. The user may switch at any time r when allwed r impsed by the applicatin. Cdec Recgnitin Framewrk Brwser Engines Pssible Event crdinatin,... Netwrk ViceXML Presentatin Figure 4 Example f Thin Cnfiguratin (vice and data supprt) with server-side speech engines lcal t speech brwser fr vice and GUI interactin.

5 Cdec Recgnitin Framewrk Brwser SRCP Engines Pssible Event crdinatin,... Netwrk ViceXML Presentatin Figure 5 Example f Thin Cnfiguratin (vice and data supprt) with server-side speech engines remte with respect t speech brwser fr vice and GUI interactin. Engines Cdec Brwser ViceXML Netwrk Presentatin Figure 6 Example f Fat client cnfiguratin with lcal speech engines fr speech and GUI interactin. This can be cmbined within a brwser implementatin.

6 Cdec Brwser Recgnitin Framewrk SRCP Engines ViceXML Netwrk Presentatin Figure 7 - Example f Fat client cnfiguratin with server-side speech engines fr speech and GUI interactin. The speech engines are remte cntrlled by SRCP. WAP Phne Wireless PDA Kisk WAP Brwser Brwser Brwser Figure 8 Example f Multi-device cnfiguratin. Cnfiguratins as illustrated in Figure 5 and Figure 7 require remte engine remte cntrl APIs r prtcls, as develped by IETF as part f SPEECHSC. Other mdalities may require mdality-specific extensins. In the multi-device cnfiguratin illustrated in Figure 8, the synchrnizatin manager may be lcated n the server r n a mbile device. In all cases, the cnfiguratin may change dynamically and it may be f interest fr the user t supprt seamless transitin between these cnfiguratins, fr example with mechanisms like discvery, negtiatin, replicatin, etc The different cnfiguratins and functins is a functin f the devices / channels, tasks and envirnment. Cnfiguratins may require registratin and negtiatin r dynamic prvisining f the device when accessing a multimdal r multi-device service. It als depends n having apprpriate mechanisms t query, r examine the device capabilities and cnfiguratin mdes.

7 2.3 Multimdal and Multi-device Authring Methds fr authring f multimdal and multi-device applicatins can be divided int several types as summarized in Figure 9. Type A Type B Type C Mdality A Mdality B Mdality A Mdality A Mdality B Mdality B Mdality B e.g. Page lad Mdality B Mdality B Mdel Etc e.g. c-brwser applicatins e.g. +Vice, SALT Figure 9 Different types f multimdal r multi-device authring methds e.g. XFrms with r withut DI authring Different authring appraches have been prpsed s far: Type A multimdal and multi-device authring where the applicatin is authred as stand alne presentatin fr each mdality with different data mdels and synchrnizatin r c-visit tags (e.g. Cbrwser authring). Type B multimdal and multi-device authring, where the applicatin is authred fr ne mdality with events and event handlers that specify what t d in the thers. Appraches like + Vice ( and SALT ( fllw this apprach. The presentatin assciated t ne mdality r device may r may nt share at authring a cmmn data mdel. Type C multimdal and multi-device authring, where the applicatins are authred at the level f the data mdel (e.g. XFrms in cntainer, JSP, ) and the presentatins fr each mdality r device are bund t the data mdel and manually authred r autmatically generated frm the data mdel (CSS). results frm the binding t the data mdel. The executin mdel and architecture prpsed supprts the three authring types, fr example by transfrming at runtime type A and type B int type C. Indeed, the mdule respnsible fr the applicatin f the synchrnizatin rules can interprets multimdal applicatins and executes the synchrnizatin. This renders the prpsed executin mdel independent f the authring language and cmpatible with the different appraches that have been prpsed s far, including +Vice, SALT and Xfrms-based synchrnizatin, with binding t pre-cmpiled presentatin r t a device-independent representatin. Nte als that Figure 2 des nt address the steps internal t the user agents. Fr example, a vice r handwriting user agent will interface with speech engines t prcess input and generate utputs. 2.4 Items t specify fr Mbile supprt mechanisms f user agents by expsing as part f the Brwser enhancement wrk item: Access t interactin events frm the GUI brwser (e.g. DOM UI events, XML events) Mechanisms t update the presentatin in the GUI brwser (e.g. DOM manipulatin, Xupdate)

8 Distributin f these exchanges (.e.g based n SOAP): Discvery, registratin and deregistratin f mdalities with synchrnizatin manager: Addresses Capabilities Security Privacy Fat client brwser enhancements part f the Brwser enhancement Authring f multimdal applicatins: Evaluatin f use f type A, B and C pssible with mbile prfile selectins Pssible nn-declarative specificatins Pssibly additinal ptins like supprting: TBD Sessin management and synchrnizatin t supprt suspend and resume and switches between thin and fat cnfiguratins etc

INVENTION DISCLOSURE

INVENTION DISCLOSURE 1. Inventin Title. Light Transprt and Data Serializatin fr TR-069 Prtcl 2. Inventin Summary. This inventin defines a light prtcl stack fr TR-069. Even thugh TR-069 is widely deplyed, its prtcl infrastructure