Experiences Developing a Wayland Based IVI HMI Ossama Othman 28 May 2013
Context Provide human-machine interface (HMI) better suited for Tizen IVI Driver safety React to vehicle state Options Leverage existing user interfaces: mobile, desktop Develop IVI-specific HMI 2
Problems Mobile and desktop user interfaces Suboptimal screen geometry: resolution, orientation Do not react to vehicle state Not safe for drivers Many features are not useful or suitable for IVI Underlying graphics subsystem may be too large or slow Develop IVI-specific HMI May require kernel and graphics hardware expertise May require additional resources for in-house development Cost Compositor User Space Input Framework Kernel KMS evdev DRM Graphics Driver Hardware Graphics Input 3
Solution Develop IVI HMI by leveraging Wayland and HTML5/JS Efficient Lightweight Rapid prototyping Simple architecture Open-source Avoids legacy issues with mobile and desktop 4
5 HMI Components
HMI Service Provides home screen Facilitates layout of application graphical surfaces Out of scope Application lifecycle management Sound handling and policy Input management 6
Compositor Handles interactions with graphics hardware Combines and renders multiple graphical surfaces into an image displayed on-screen Wayland Core compositor protocol used by IVI HMI Additional IVI HMI-specific Wayland-based protocols to be defined Weston Reference Wayland compositor HMI will leverage extensibility to fill feature gaps 7
Application Run-time Provides reusable set of functionality to applications HTML5 Webkit Web API implementations Toolkits Native C library Middleware Toolkits 8
Client Application Draws surfaces in a buffer Cairo OpenGL etc Shares buffer with compositor through Wayland Examples Media player Browser 9
10 Component Structure
11 HMI Server Side
Server Side Architectures HMI Service Stand-alone Integrated Server side architecture remains transparent to the client Just an implementation detail from the client point of view Client does not determine where surfaces are placed 12
Stand-alone HMI Service HMI service is another Wayland based client Separate process from compositor Communicates with compositor just as client application would Example: GENIVI style layer management 13
Stand-alone HMI Service Sequence Diagram sd Stand-alone HMI Serv ice Application libwayland Weston Weston HMI Plugin HMI Service wl_surface_attach() wl_surface_commit() surface_commit() configure_surface() layout_surface() map_surface() weston_surface_update_transform() 14
Integrated HMI Service HMI Service integrated with compositor Embedded directly in compositor plugin Performs same functions as stand-alone service 15
Integrated HMI Service Sequence Diagram sd Integrated HMI Serv ice Application libwayland Weston Weston HMI Plugin wl_surface_attach() wl_surface_commit() surface_commit() configure_surface() layout_surface() weston_surface_update_transform() 16
HMI Server Side Architecture Comparison Pros Cons Stand-alone Integrated Stand-alone Integrated Reusable / Flexible Simpler architecture Better fatal error recovery More efficient: less IPC Rapid prototyping Flexibility may not be needed Less efficient: more IPC Fatal error recovery may be more difficult Potentially easier to compromise compositor state 17
18 HTML5 HMI Problems
Context: HTML5 Based HMI HTML5 based HMI functionality similar to desktop window manager User interaction Layout of applications Both native and Web applications must be supported Application may not be full-screen Application may not be integrated with Web run-time (e.g. through NaCl) IVI-specific paradigms such as layers should be supported Layer is a collection of graphical surfaces Sample layers: home screen decorations, input layer, split screens GENIVI style layer management 19
Problem: Gap Between HTML5 and Native Sides HTML5 based HMI cannot readily layout surfaces created by native application HMI has no knowledge of surfaces created by native applications JavaScript events are not visible to native application Recall that native application may not be integrated with Web run-time Events generated by compositor are not visible to HTML5 based HMI 20
Problem: Supporting the Layer Concept No concept of layers in HTML5 May be emulated using multiple HTML5 Canvases Canvas based approach not always suitable Not all Web applications use HTML5 Canvases Not applicable for native applications 21
Solution Compositor Plugin exposes layer functionality required by HMI Web run-time plugin Provides JavaScript API that allows interaction with the compositor Forwards appropriate events to and from native and application and HTML5 based HMI Alternatives Use WebSockets to communicate between compositor plugin and HTML5 based HMI IVI-specific Node.js extensions 22
23 Other Problems
Desktop Shell-isms Direct use of wl_shell in application or toolkit Specific to desktop shell model / use case Not entirely applicable to IVI Compositor wl_shell implementation may not be available Proposal for better handling of different Wayland shell types currently under review upstream 24
25 Conclusion
Summary Wayland Simple, lightweight, efficient and extensible protocol is ideal for custom HMI developmen HTML5 is a compelling technology for implementing an HMI Many useful HTML5/JS libraries Very active Web API standardization and development Native application integration is seemingly difficult but is tractable, nonetheless 26
References Tizen IVI Images Daily: http://download.tizen.org/releases/daily/2.0alpha/ivi-wayland/ Snapshots: http://download.tizen.org/snapshots/2.0alpha/ivi-wayland/ Tizen 3.0 based IVI images are a work in progress Wayland and Weston http://wayland.freedesktop.org/ http://cgit.freedesktop.org/wayland GENIVI http://www.genivi.org/ http://projects.genivi.org/ivi-layer-management/ 27
Legal INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY RELATING TO SALE AND/OR USE OF INTEL PRODUCTS, INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT, OR OTHER INTELLECTUAL PROPERTY RIGHT. Intel may make changes to specifications, product descriptions, and plans at any time, without notice. All dates provided are subject to change without notice. Intel is a trademark of Intel Corporation in the U.S. and other countries. *Other names and brands may be claimed as the property of others. Copyright 2013, Intel Corporation. All rights are protected. 28