ID B33C: Developing Rich Graphical User Interfaces Optimized for Embedded Systems

Transcription

1 ID B33C: Developing Rich Graphical User Interfaces Optimized for Embedded Systems Jason Clarke Co-Founder Crank Software Inc. 13 October 2010 Version: 1.0 In the market place today the myth about embedded Ethernet is that it very hard to achieve. In this presentation and lab, I will show you that Open-Source TCP/IP stacks with freely available tools are now true enablers for less costly embedded Ethernet products. 1

2 Jason Clarke Co-Founder Crank Software V.P. Sales and Marketing Education Computer Engineer Technology Degree from St. Lawrence College Work Experience 13 years experience with embedded systems 10 years developing and selling real time operating systems at QNX Worked devloping BSPs, drivers, flash filesystems, and various upper level servers before moving to Field Engineer Co-Founded Crank Software Inc. 3 years ago and focused on providing customers with rich user interface for embedded systems. 2

3 Agenda Overview of Crank Software The changing market for graphical displays UI Designer, the new member of the embedded team How does it all come together on my end target Live Demonstration Import of graphical content UI development Prototype and Simulation Embedding on target Q&A 3 3

4 About Crank Software Product s Crank Storyboard Suite streamlines the entire UI lifecycle from UI design to simulation to being optimized for the resourceconstrained target device. Service Provides custom UI and UI framework s engineering services. Crank s products deliver competitive advantage, high ROI, and low TCO to Tier 1 manufacturers through to up-and-coming manufacturers. Crank s services deliver leading edge innovation on time and on budget. Crank Software has two main offering in it s portfolio. The first is Crank s Storyboard Suite which enables UI designer and embedded developers to quickly create rich user interfaces for embedded devices. The second part to Crank Software business is professional services department that has both a broad and deep knowledge of the embedded market. 4

5 Renesas & Crank Software 10/20/

6 Microcontroller and Microprocessor Line-up Superscalar, MMU, Multimedia Up to 1200 DMIPS, 45, 65 & 90nm process Video and audio processing on Linux Server, Industrial & Automotive High Performance CPU, Low Power Up to 500 DMIPS, 150 & 90nm process 600uA/MHz, 1.5 ua standby Medical, Automotive & Industrial High Performance CPU, FPU, DSC Up to 165 DMIPS, 90nm process 500uA/MHz, 2.5 ua standby Ethernet, CAN, USB, Motor Control, TFT Display Legacy Cores Next-generation migration to RX General Purpose Up to 10 DMIPS, 130nm process 350 ua/mhz, 1uA standby Capacitive touch Ultra Low Power Up to 25 DMIPS, 150nm process 190 ua/mhz, 0.3uA standby Application-specific integration Embedded Security Up to 25 DMIPS, 180, 90nm process 1mA/MHz, 100uA standby Crypto engine, Hardware security Renesas Electronics America Inc. All rights reserved. 6 Here are the MCU and MPU Product Lines, I am not going to cover any specific information on these families, but rather I want to show you where this session is focused <CLICK> 6

7 Microcontroller and Microprocessor Line-up Superscalar, MMU, Multimedia Up to 1200 DMIPS, 45, 65 & 90nm process Video and audio processing on Linux Server, Industrial & Automotive High Performance CPU, Low Power SuperH Up to 500 DMIPS, 150 & 90nm process 600uA/MHz, 1.5 ua standby Medical, Automotive & Industrial High Performance CPU, FPU, DSC Up to 165 DMIPS, 90nm process 500uA/MHz, 2.5 ua standby Ethernet, CAN, USB, Motor Control, TFT Display Legacy Cores Next-generation migration to RX General Purpose Up to 10 DMIPS, 130nm process 350 ua/mhz, 1uA standby Capacitive touch Ultra Low Power Up to 25 DMIPS, 150nm process 190 ua/mhz, 0.3uA standby Application-specific integration Embedded Security Up to 25 DMIPS, 180, 90nm process 1mA/MHz, 100uA standby Crypto engine, Hardware security Renesas Electronics America Inc. All rights reserved. 7 These are the products where this presentation applies <Click> 7

8 The Growing Demand for LCD Cleaner 10/20/

9 Touchscreen Display Sales Small to Medium Sized Touchscreen Revenue Forecast Driven by high volume cellular telephone market Mass market effect LCD and Touchscreen prices drop It is no secret that there is a growing demand for touch screens and LCD displays for almost every product that we interact with or that terminates data. The smart phone and cell phone markets are driving this demand but there is a trickle down effect that is being seen across almost all markets in the embedded device space. Also driving this need for a display on all device is the ever decreasing price of these displays, processors and other supporting hardware. 9

10 Increasing Role of Display Technology New Products Graphical display now a requirement; Consumer Demand Touch interface shifting from hard to soft buttons Additional product flexibility, more refresh opportunities Displays on the devices that have historically had no need for a touch screen are now seeing it as a must have in future product rather than nice to have. Consumer demand for products that resemble their iphone with soft buttons rather than hard buttons is current trend with no end in sight. The advantage for the OEM with this trend is the while providing a richer user experience to their end customer they can at the same create a more flexible product that can provide updates and feature enhancements completely in software providing greater reuse for their hardware design. 10

11

12 The iphone Effect The iphone has raised the bar for UI expectations out of reach of most consumer device manufactures. Unfair Playing Field Bill of materials (BOM) costs are close to purchase price Backend revenues subsidise higher BOM costs Hardware accelerated platform with tight system integration Apple dedicated significant resources to this product The iphone is the product that open everyone s eyes to the fact that product can be completely done using just a touchscreen for the interface and provide a great user experience. However the downside to this is that is is now the expectation of the market. The iphone is the bar that all new devices with displays are held to. The issue with this is that the it isn t fail playing field since most embedded devices in the consumer space have no backend revenue stream so the purchase price needs to include profit. The iphone on the other had can be sold for next to no profit since it s backend revenue stream is so great. This makes it very hard to compete with a product who s BOM is an order of magnitude bigger and still be expected to deliver the same user experience. But however unfair it might be it is still the expectation you are held to by the market. 12

13 Technology Requirements Support rich visual content Blending, animations and transparencies Mostly 2D, but demand for 3D effects growing Integrate and align with embedded systems Constrained CPU and memory resources Efficiently leverage hardware capabilities Communicating with the system Scalable and Portable technology Single source solution across product lines Operating System, Main Processor, Graphics hardware 13

14 Typical Implementation Choices Widget Libraries Provide abstraction for common elements Difficult to create a unique look and feel Requires the user to be a developer Requires compilation, platform specific Graphic Rendering Libraries Best performance Optimized for specific hardware Hard to develop for, no tool solution Graphical Engine Data driven runtime model May be cross platform, or portable Often provides a tool solution No compilation required When we look at the graphic software solutions there three main categories they fall into. The Widget Library is the standard embedded solution that that has been used for years and most people are familiar with. It was designed for industrial, medical and military device who where the first to adopt using a GUI. Functionality was the main concern of the UI for these early applications and just having a UI was key differentiator. The Graphical Rendering Libraries are becoming more popular lately with high performance libraries that allow you leverage accelerated hardware for making an optimal performing UI. These are great for performance but provide no visual development environment and need to be completely written in code that was designed with the concerns of the hardware before design and architecture. The Graphical Engine is a data driven model that is allows a user to define their UI in a system independent method that is later interpreted by target specific runtime. This method provides the best option for a balance between design, portability and performance 14

15 Who s the new guy? 10/20/

16 Complimentary Skill Sets Artistic Talent Technical Expertise The new UI development team is no longer just embedded engineers. It is made up of many new stake holders that need to be able to contribute and maintain control of the end user interface. Each of these stake holder have a different level of technical expertise that makes it hard to work together 16

17 The Need for A UI/Graphic Designer Greater consumer awareness of display Engineering user interfaces no longer adequate A great UI can have more value than product function The UI is now a major product differentiator UI design occurs early, iterates frequently No longer a static component of product development User interface churns at the same level as the product development User Interface design has become about User Experience Still pictures of screens and use cases don t express behavior User experience first class consideration during development

18 UI Designer in the Embedded Lifecycle 10/20/

19 Integration of the UI Designer Handoff Handoff Process User interface design elements created Initial content comes from a prototype, beta or mock-up Prototype elements provided to SW developers Graphic Designer hands-off images, wireframes and storyboard SW developers incorporate them into product Transform graphic assets into implementation compatible assets Implement associated behaviour, often unspecified or assumed Product approaches release Churn and cycle through labour intensive adjustments

20 Integration of the UI Designer Handoff Challenges More resources used than required Designer employed part-time, not fully engaged Software developer serves as technology interpreter Designer doesn t control of final product User experience is often not what was intended Many cycles wasted iterating back and forth Team burnout and weariness of change cycles Substandard end product results 20

21 Integration of the UI Designer Square Peg Square Peg Process User Interface design elements created Same artifacts generated as in hand-off Parallel development enabled Graphic designer builds UI using tools Desktop technologies Photoshop and Flash Software developer incorporates technology Porting effort to embedded platform Integrate ported technology and content Tune embedded display to match desktop

22 Integration of the UI Designer Square Peg Challenges Downstream work effort Future ports to new systems Wasted in house expertise Interface challenges How to communicate with target system Abstraction layer to isolate technology Incompatible technology fit Designed for desktop; infinite resources Needs to be customized for embedded 22

23 Integration into the Embedded System 10/20/

24 Hidden Complexity The complexity of the UI lies beneath the surface Rich User Interface Embedded System Integration Data/System IO Testing Performance Portability Developer Resources System Efficiency Debugging Design a beautiful and intuitive UI is very important in today's systems, but how do we get this great UI to work on our system? A rich UI is just the tip of the iceberg, there are many complexities that that lie just below the surface and will need to addressed. Choices made early in the system architecture can come back to bite you by schedule delays and can sometimes cause complete reworks. 24

25 Embedded System Integration Rich Displays and Animations v.s. Resource Constrained System Two concepts at odds with each other Rich Displays Memory consuming graphics/fonts Animations CPU cycles for compositing Few developers understand full picture Minimize resource consumption, maintain quality Concurrent development support: System & UI Everything about a rich user interface is completely at odds with a desired embedded system design. Large rich display require more memory for images and fonts, while animations eat away at cpu cycles. 25

26 Leverage Hardware Capabilities Multiple levels of hardware acceleration Optimized blit Filled rectangle Alpha blending and chroma keying Polygon routines Matrix transformations Multi-layer and drawing surface support Language support: OpenVG OpenGL (ES) 26

27 Leverage Hardware Capabilities (cont) Hardware offload challenges Many solutions abstract HW details Too much abstraction leaves untapped potential More than just dedicated GPU support Language support, processor optimizations etc Plan a scalability roadmap Leverage all flavours of hardware support throughout an entire product line Technology constantly evolving and improving 27

28 System Communication Performance is only one consideration How to integrate with the overall system? How does data enter/exit the interface? How much effect on your design? Is migration between platforms seamless? Flash as a case study Sandboxed technology, difficult to extend Designed for desktop with network stack 28

29 Other Graphical Content Providers How do multiple providers integrate Web Browser Video Map Content Along with understanding how your UI is going to leverage the hardware and communicate through the entire systems comes another issue of understanding who you will integrate with other graphical content providers. Many systems and not only media players require support for video and web browsers and you need to make sure these items can integrate with you UI and also take advantage of hardware offload when available. 29

30 Portability... Cost Savings tomorrow Consumer products are fickle Often extends to many embedded products Any reasonable measure to drop HW BOM cost UI should be as portable as code, but not code UI costs often negated in porting estimates OS support may not include graphics support Scalable solutions often leveraged Consideration for evolution of the technology Trend towards integration direct to core It s always hard to do but making sure your software solution is portable can provide time and cost savings down the road. With the rapid speed at which hardware changes customers constantly find themselves moving to new platforms. 30

31 Testing and Verification Testing rarely a design decision concern Testing is a challenge, UI testing even more so Testing less likely if no framework support Can UI be exercised and validated? Can it be automated or is it a manual process? How to monitor quality and progress? Desktop tools generally not applicable Most do not consider remote targets 31

32 Utopia: A Lofty Goal Development Environment Support efficient prototype-to-product development Targeted at running inside embedded systems Enable parallel and collaborative development Runtime Environment Scalable, modern and efficient feature set Portable technology, easy to migrate systems Facilitates design testing, debugging performance 32

33 Prototype To Product Development Create a familiar development environment Desktop systems, Mac and Windows OS Allow graphic designers to work with their tools Photoshop, Flash for content development Avoid duplication and redundant activities Deploy what you build Focus individuals on their core competencies

34 Demonstration of Crank s Storyboard Suite targeting MS /20/

35 Questions? 35 35

36 Thank You 10/20/

37 Appendix 10/20/ The appendix slides are here just for discussion and also anyone looking at the presentation without attending the session will be missing out on the live demo. 37

38 Crank Work Flow Crank Focus 38

39 UI designer in the embedded design cycle Integrating with the industry standard graphical tools allows the user to move quickly from design to prototype to final product. 39

40 Storyboard the entire UI Storyboard the entire UI with easy to use drag and drop interface. 40

41 Easily define UI behaviour Add trigger events and resulting actions quickly and easily to bring the UI to life. 41

42 Verify design with built in simulator One click to simulate UI with on development host and easily verify design decisions. 42

43 Content rich to assist in design 43

44 Architected for scalability and portability OS Input Method Data Change Image Storyboard Engine Gesture s Timers IO / Event Manager Script Animation Action Manager Text Extern al Render Extension Screen Manager Custom Custom Custo m Use Storyboard Engine as a library or as a separate task Plugin architecture allows for scalability Limit OS dependency for portability OS Render Method (SDL, AGTDK, WindML, Direct FB, Win32,...) Render Manager 44

45 Event Driven Touch Gesture Voice System Message Hard Button OS Input Method Gesture s Timers IO / Event Manager Custom PRESS Data Change Script Animatio n Action Manager Custo m Data Change Script System Message Custom Action Events trigger Actions Events can be generated from multiple sources Multiple source can generate the same events Multiple actions can be triggered from a single event Events can be handled on all levels (controls, layers, screen, app) Controls have support for focus 45

46 System integration External applications or tasks use Storyboard IO library to communicate to and from the Storyboard Engine. This promotes a clean and modular architecture by keeping the system logic outside of the UI. Storyboard Engine Storyboard IO Library External Task IO Interface Storyboard IO Library External Task The IO library allows the external task Change image, text and attributes on controls Inject events into system Receive notifications Support for multiple channels The IO can use any existing API as transport mqueue Pipes Message passing TCP/IP Shared memory Proprietary API 46

47 External Renders External Task Storyboard Engine External Render Buffer External Render Buffer Storyboard creates a shared memory buffer for the external application to draw to External renders could be video player, web browser, flash player, 2 nd Instance of Storyboard,... 47

48 Renesas Electronics America Inc. 48