Fault tolerance in consumer products. Ben Pronk

Size: px

Start display at page:

Download "Fault tolerance in consumer products. Ben Pronk"

Christopher Nash
5 years ago
Views:

1 Fault tolerance in consumer products Ben Pronk

2 Content Consumer electronics, some background Reliability and software in consumer products Current solutions Future outllok 2

3 Consumer electronics, some background KID S BEDROOM MASTER BEDROOM camera Internet radio broadband INTERNET TV-to-TV link wireless LAN home theater audio portable display RF remote control STUDY KITCHEN Media Server (DVD+RW/HDD LIVING ROOM 3

Consumer electronics, some background In consumer devices we find increasing Digital connectivity and interaction Ethernet, Wireless, USB, NFC, Bluetooth, HDMI,

4 Consumer electronics, some background In consumer devices we find increasing Digital connectivity and interaction Ethernet, Wireless, USB, NFC, Bluetooth, HDMI, 1394 Cross-functionality TV as monitor, PC as TV-receiver, MP3 player in TV Steep rise in number of supported formats Mpeg2, Mpeg4, H264, AVS, DivX, JPEG, MP3,.. 4

5 Consumer electronics, some background Code size (ROM) 100 MB 10 MB 1 MB 100 KB 10 KB 1 KB x software increase every 6-7 years TV Moore s law for memory (10x every 5 years) No SW 1 KB 64 KB 2 MB 30 MB 5 Source: Philips Consumer Electronics

6 Consumer electronics, some background No or only limited remote service capabilities No large log files or trace buffers No extensive service reports No remote monitoring If it fails in the eye of the customer it will be returned to the shop The business aspects Consumer electronics is a low margin business Just a few percent of returns will eat away the margins Will undermine your position in the market (internet logs) Will cost you shelf space in retail chains Even an update in the manufactoring/logistic chain is very costly. So functionality, connectivity and software content go up While maintinaing the same low call rate 6

7 Reliability and software in CE products Historically a CE product is some electronics and a cage Reliability was determined by PCB/electronics reliability This is reasonably well understood and well predictable Nowadays, with a million lines of code and full interconnectivity Software and interoperability issues become a major issue And already determine a significant part of the call rate 7

8 Reliability and software in CE products TV, historically normal electronics wear Bad signal quality Limited content dependence only example teletext TV-now Many many content dependencies, digital TV, all sorts of codecs Many unexpected connections Software crashes And yes also still normal electronics 8

9 Reliability and software in CE products Note that also the user interaction becomes increasingly complex Much more complex installation Connection/network with all kinds of devices (wired/wireless) Increasingly complex PC-type of functions Also this has lead to a large growth in call rate CE device manuals and UI s are not known to be intuitive People don t read the manual anyway React unexpectedly on errrors (and draw different conclusions than you) 9

10 Reliability and software in CE products On the internals CE devices are still very much resource constrained systems So limited ram/flash normally no disk Make use of embedded (sometimes home build) OS s Flat memory space, no protection against overwrites With limited capabilities on protection and recovery Do have to recover always autonomously, There is no repair outside the power switch Yet: Have to implement a steep rising set of requirements Integrate a lot of third party components Become increasingly more multi-cpu (3-4 CPU s is no exception) Have to work around failures in the IC s/hw 10

11 Current solutions How do we cope, quite simple measures No assumptions on HW state, reprogram full settings repeatedly Simply reset model No partial resets due to OS memory model and simple process structure Full reset of a processor in case of error/crash Liveness checks between threads/processors Consistency checks (in background tasks) Data structures Stack overflow Code consistency 11

12 Current solutions How do we cope, maintaining basic functionality Most of the time a TV plays video/audio This function has to be maintained If ths is the case users won t even note resets Approach: Maintain A/V streaming path when host CPU resets Keep hardware state unmodified after a warm reset As long as A/V path is only HW (or a separate CPU) this works Does not work for Streaming data coming in from a network in the host CPU Any further data that needs host intervention (UI, teletext) 12

13 Current solutions MCU-DDR Controller Control / Security MIPS-4KeC TM2270 Control / Security Support functions MPEG-demux Support functions CAB VFE TMDS CLOCKS BOOT RESET JTAG-DMA UART (2x) System Control VCP-RX HDMI-RX MIPS Device Control and Status Network PCI/XIO I2C (3x) CA / TS IO Pipelined Memory Access Network MPEG-decode MBVP (2x) CPIPE (2x) VCP-PSUD VCP-PSUV SPDIF in SPDIF out Audio in Audio out TriMedia Device Control and Status Network DENC LVDS-TX V-DAC AFE DemDec DSP HDMI-RX2DTL A-DAC Audio DSP Bridge 13 Philips Semiconductors 2006, All rights reserved 13

14 Current solutions Hot boot Watchdog timing in MIPS and Trimedia, Warm reset, which means restart of the MIPS under special conditions Shared memory and Trimedia continues to operate as is UI and other host controlled aspects disappear At restart the HW is not reprogrammed The MIPS re-synchronizes with trimedia A trimedia application cleans up, buffering, connections, Finally MIPS reselects a use case setting everything effectively in a mode All by all a complex mechanism, that itself needs a lot of testing. 14

15 Current solutions And of course testing, testing, testing Long test cycles, many sets in parallel, duration tests One set makes about hours in its lifetime Many test-hours are needed to have some statistical proof What does this say in a software world Field tests to cope with differences in conditions Trucks driving through difficult areas in the world Recordings of all relevant streams and exceptions Compatibility tests with e.g. different media, streams, other equipment etc A million types of crappy disks Different types of encoded data, non-standard encodings Many many peripheral devices to interface with 15

16 Future The situation will get worse in the future Inherent reliability of silicon devcies decreases e.g. More statistical errors in deep submicron devices (45, 32 ) E.g. NAND flash error rate keep rising The growth in formats and functionality will not stop As will the growth in software sizes. So we have to explore approaches over and above the mechanisms 16

17 Future More isolation of components More multi-cpu use, with more de-coupled resources Use of standard operating systems with memory protection Linux, WindowsCE Use of process structure and memory protection to safeguard partial restart of the system, 17

18 Future Various investigations ongoing, NXP research as well as Trader Areas of investigation: Hardware monitoring of real time data Audio, Video Memory bus access Dynamic adaptation of behaviour (rescheduling) Error detection HW-monitoring Software state consistency checks, potentially hardware assisted Architecture/model based reliability check 18

19 Future Areas of investigation: Program spectra, computer aided de-bugging Indication of suspicious code through likelyhood ordering Stresstest upport for memory, bus, CPU etc load Usability investiagtion What do users perceive as an error And what as an adequate recovery mechanism 19

20 20

Fujitsu System Applications Support. Fujitsu Microelectronics America, Inc. 02/02

Fujitsu System Applications Support 1 Overview System Applications Support SOC Application Development Lab Multimedia VoIP Wireless Bluetooth Processors, DSP and Peripherals ARM Reference Platform 2 SOC