Fundamentals of Touch Technologies

Transcription

1 Fundamentals of Touch Technologies Geoff Walker Senior Touch Technologist Intel Corporation BS-EE & BS-English, Polytechnic Institute of New York University Walker Mobile, LLC; NextWindow; Elo TouchSystems; Handspring; Fujitsu Personal Systems; GRiD Systems; Hewlett-Packard Active SID member: Chapter Chair, Symposium-paper subcommittee member, Guest Editor for Touch & Interactivity Information Display Published >65 articles on touchscreens, mobile displays, & mobile computers; presented at >30 conferences in China/Japan/Taiwan/US Globally-recognized expert on touch technologies File Download: v1.1 1

2 Agenda 1 Course structure Topic Allocation Introduction 6% Main Content 88% Capacitive 1 32% Resistive 2 9% Acoustic 3 8% Optical 4 20% Embedded 5 13% Other 6 2% Active Stylus 7 6% Software 8 10% Conclusions 6% TOTAL 100% 100% 6 fundamental touch technologies with ~20 types 2

3 Agenda 2 Admin Introduction to Touch Technologies Capacitive (1) 1A - Projected Capacitive (P-Cap) ITO Replacement Materials 1B - Surface Capacitive Resistive (2) 2A - Analog Resistive 2B - Analog Multi-Touch Resistive (AMR) 2C - Digital Multi-Touch Resistive Acoustic (3) 3A - Surface Acoustic Wave (SAW) 3B - Acoustic Pulse Recognition (APR by Elo Touch Solutions) 3C - Dispersive Signal Technology (DST by 3M Touch Systems) 3

4 Agenda 3 Optical (4) 4A - Traditional Infrared 4B - Waveguide Infrared (DVT by RPO) 4C - Multi-Touch Infrared 4D - Camera-Based Optical 4E - Planar Scatter Detection (PSD by FlatFrog) 4F - Vision-Based Embedded (5) LCD Architecture Refresher & Terminology Review Hybrid In-Cell Mutual Capacitive for IPS & Non-IPS LCDs On-Cell P-Cap In-Cell Light-Sensing In-Cell Pressed Capacitive In-Cell Voltage-Sensing In-Cell Self-Capacitive Embedded Touch Issues 4

5 Agenda 4 Other Touch Technologies (6) Force-Sensing Active-Stylus Technologies (7) Electromagnetic Resonance (EMR) Stylus P-Cap Stylus Other Active Stylus Technologies Software (8) Multi-Touch Operating-System Application-Development Support Middleware 5

6 Agenda 5 Conclusions Touch Technology vs. Application Usability, Performance, and Integration Characteristics Touch Technology Primary Advantages and Flaws Predictions for the Future Suggested Reading on Touch Recommended Conferences and Trade Shows on Touch 6

7 Introduction Source: Gizmodo (Michelangelo's "The Creation Of Adam, in the Sistine Chapel, 1511) File Download: 7

8 Two Basic Categories of Touch Opaque (non-transparent) touch Dominated by the controller chip suppliers Atmel, Cypress, Synaptics, etc. One technology (projected [self] capacitive) Sensor is typically developed by the device OEM Notebook touchpads are the highest-revenue application Synaptics, Alps and ELAN have the majority of the market Sensors are all two-layer projected capacitive There is no further discussion of opaque touch in this course Transparent touch on top of a display Dominated by the touch module manufacturers (150+ worldwide) 6 fundamental technologies with ~20 types 8

9 Overall Touchscreen Market Units (Billions) Revenue ($Billions) CAGR = 15.4% $35.0 $30.0 $25.0 $20.0 $15.0 $ CAGR = 16.3% $11.1 $16.0 $21.4 $25.8 $28.6 $30.3 $31.3 $ $ $ Source: DisplaySearch Touch-Panel Market Analysis 2012 Annual Report (July 2012) Touch in 2007 was 308M units & $1.3B 9

10 Touchscreen Market by Technology (Units) % of Units Shipped 100% 2% 2% 3% 4% 4% 4% 4% 4% 4% 4% 2% 1% 1% 5% 6% 7% 7% 8% 90% 9% 10% 80% 70% 53% 33% 26% 20% 17% 14% 12% 11% 60% 50% 40% 30% 85% 74% 58% 64% 69% 72% 74% 74% 74% Others In-Cell On-Cell Resistive P-Cap 20% 10% 0% 42% 24% 13% 2008A 2009A 2010A 2011E 2012E 2013E 2014E 2015E 2016E 2017E Source: Guoxin Securities, TPK, and DisplaySearch (February, 2012) 10

11 Touch Technologies by Size & Application Mobile (2 17 ) Stationary Commercial (10 30 ) Stationary Consumer (10 30 ) Large-Format ( >30 ) # Touch Technology 1A Projected Capacitive M M L L 1B Surface Capacitive M 2A Analog Resistive M M L 2B Analog Multi-Touch Resistive (AMR) E E 2C Digital Multi-Touch Resistive E 3A Surface Acoustic Wave (SAW) M L 3B Acoustic Pulse Recognition (APR) E L 3C Dispersive Signal Technology (DST) L 4A Traditional Infrared (IR) M M 4B Waveguide Infrared 4C Multi-Touch Infrared E E E E 4D Camera-Based Optical M M 4E Planar Scatter Detection (PSD) E 4F Vision-Based (In-Cell Optical) E 5 Embedded (In-Cell/On-Cell Capacitive) M E 6 Force Sensing E M = Mainstream L = Low-volume E = Emerging 11

12 Touch Technologies by Materials & Process Transparent Conductor (ITO) Touch Technology No Transparent Conductor Continuous Surface capacitive Analog resistive Low Resolution Analog multi-touch resistive (AMR) = Mainstream = Emerging or Low-Volume Patterned High Resolution Projected capacitive Embedded (In-cell & on-cell) Edge Conductors Acoustic Pulse Recognition (APR) Dispersive Signal Technology (DST) Surface acoustic wave (SAW) Traditional infrared (IR) Multi-touch infrared Camera-based Planar scatter detection (PSD) Vision-based Force sensing No Edge Conductors 12

13 A Simple Touch Isn t Simple 1 Touch classification from the University of Toronto Example: iphone/ipad Source: Daniel Wigdor 13

14 A Simple Touch Isn t Simple 2 It s far more complex than just how many touches? The Breadth vs. Depth Problem? Design software once for common capabilities (wide breadth, limited functionality) Re-design software for each platform s capabilities (narrow breadth, deep functionality) 14

15 Touch Is An Indirect Measurement This is one reason why there are so many technologies Touch Technology Projected capacitive, Embedded (capacitive) Surface capacitive Resistive (all forms) & Embedded (voltage-sensing) Surface acoustic wave Acoustic Pulse Recognition & Dispersive Signal Technology Infrared & Camera-based (all forms), Planar scatter detection Vision-based Embedded (light-sensing) Force sensing What s Being Measured Change in capacitance Current Voltage The ideal method of sensing touch has yet to be invented! Ultrasonic wave amplitude Bending waves Absence or reduction of light Change in image Presence of light Force 15

16 Capacitive Touch Technologies Projected Capacitive (P-Cap) ITO-Replacement Materials Surface Capacitive 16

17 Projected Capacitive Source: Apple File Download: 17

18 Projected Capacitive 1 iphone, ipad and other products using projected capacitive (p-cap) have set the standard for touch in SEVERAL BILLION consumers minds Multiple simultaneous touches Extremely light touch Flush surface (zero-bezel) Excellent optical performance Reliable and durable Fully integrated into the user experience effortless & fun Source: TabletPC2.com 18

19 Projected Capacitive 2 Self-capacitance principle Source: 3M 19

20 Projected Capacitive 3 Self-capacitance electrode variations Source: 3M Multiple separate pads in a single layer Each pad is scanned individually Rows and columns of electrodes in two layers Row & column electrodes are scanned in sequence 20

21 Projected Capacitive 4 The problem with self-capacitance Self Capacitance Mutual Capacitance No Ghost Points! Source: Touch International Touches that are diagonally separated produce two maximums on each axis (real points & ghost points) Ghost points: False touches positionally related to real touches 21

22 Projected Capacitive 5 Mutual-capacitance principle Projected electric field Touch Source: 3M No touch 22

23 Projected Capacitive 6 Mutual capacitance example (Apple iphone) Source: Apple Source: The Author Output is an array of capacitance values for each X-Y intersection 23

24 Projected Capacitive 7 Raw data including noise Filtered data Gradient data 10 fingers, 2 palms and 3 others Touch region coordinates and gradient data Touch regions Source: Apple Patent Application #2006/

25 Projected Capacitive 8 Interlocking diamond electrode configuration 4.5 mm typical BLUE (Sense) RED (Drive) Source: 3M Source: Atmel 25

26 Projected Capacitive 9 Self-Capacitance Older technology, but still used Limited to 1 or 2 touches with ghosting Lower immunity to LCD noise Lower touch accuracy Sensor is usually a diamond pattern Harder to maximize SNR Simpler, lower cost controller Usually a single-layer sensor Mutual Capacitance Newer technology Two or more unambiguous touches Higher immunity to LCD noise Higher touch accuracy Allows more flexibility in pattern design Easier to maximize SNR More complex, higher-cost controller Usually a two-layer sensor (or one-layer with bridges ) 26

27 Projected Capacitive 10 Summary of all p-cap constructions Embedded sensor On-cell = Both electrodes on top of color filter glass (or OLED glass) Example = Samsung S1/2/3, Toshiba Excite 7.7 Hybrid In-Cell = Drive electrodes on TFT array, sense on top of CF glass Example = HTC EVO Design, Sony Xperia S In-Cell = Both electrodes on TFT array Example = iphone-5 & ipod Touch-5 Glass-only sensor (CG)G-DITO or (CG)G2 = one glass with ITO on each side (Apple patent) Example = iphone & ipad 1-4 (CG)G-SITO or (CG)G1 = one glass with ITO on one side with bridges Example = Kindle Fire & HD, HTC Sensation, many others (CG)GG = two sheets of glass with ITO on one side of each Rare today 27

28 Projected Capacitive 11 Example of (CG)G-DITO 1.6 mm typical Source: The Author 28

29 Projected Capacitive 12 P-cap constructions (continued) Film-only sensor (CG)FF = two single-sided ITO films Example = Samsung Galaxy Tab 7/8.9/10, HTC One X (CG)F-DITO or (CG)F2 or (CG)Fxy = one double-sided ITO film Example = Apple ipad mini (CG)F1 = one film with ITO on one side with bridges Example =? Glass and film sensor OGS or (CG)2 = cover-glass with ITO on one side with bridges Example = Google Nexus 4 & 7, many others (CG)1F = cover-glass with ITO on one side and one single-sided ITO film Example = Microsoft Surface RT 29

30 Projected Capacitive 13 Example of OGS (G2) Source: The Author 30

31 Projected Capacitive 14 Comparing p-cap constructions Transmissivity Border width due to routing Cost of ITO film or deposition Lamination yields Thickness & weight Existing equipment and/or method experience Terminology note Lamination = Adhering film to glass, or film to film Bonding = Adhering touchscreen to display Direct bonding = No air-gap Air-bonding = Air-gap (gasket around periphery) 31

32 Projected Capacitive 15 PET film versus glass PET Glass Temperature Tolerance Melts at 250 C Melts at 570 C Aging Effects Yellowing, curling, No known effect surface deformation Transparency 85% =>90% Resolution Capability 50 µm 1 µm Stackup Thinner Thicker Weight Light Heavier Moisture Resistance Good Excellent Lamination Yield Excellent Good Mechanical Strengthening None Tempering Cost $ $$ 32

33 Projected Capacitive 16 One-Glass Solution (OGS) Also called touch on lens, sensor on cover, direct patterned window and many other names Advantages Eliminates a fourth sheet of glass (G-DITO), making the end-product thinner and lighter Competitive weapon against embedded touch from LCD suppliers Disadvantages Requires close cooperation with cover-glass makers, or increased vertical integration (preferable) Yields are lower (more complex operations) Bendable cover glass can affect touch performance Harder to shield touchscreen from LCD noise Variation: G1F Sense electrodes on cover glass; drive electrodes on PET film 33

34 One-Glass Solution (OGS) Versus Embedded Touch 1 One-Glass Solution (OGS) versus Embedded Touch isn t really about technology, it s about The Touch-Panel Industry vs. The LCD Industry 34

35 OGS Versus Embedded 2 Thickness Almost the same Weight Same Off-screen buttons Embedded can t do this; OGS can Performance Approximately the same Controller Almost the same Cost Approximately the same Current yields are roughly equal The end-user can t tell the difference (except for the off-screen buttons) 35

36 OGS versus Embedded 3 What the device OEM buys OGS Touch-panel/cover-glass from TP supplier Standard LCD from LCD supplier Nothing from CG supplier Embedded This is a relatively small business already being attacked from multiple directions Different LCD with embedded touch from LCD supplier Cover glass from CG supplier This is a robust NOTHING from TP supplier $16B business (2012) that s not going to give up without a fight 36

37 OGS versus Embedded 4 What the device OEM would like to buy Laminated LCD/touch-panel/cover-glass module Problem LCD suppliers don t want to get into the cover-glass business each device s cover-glass is unique Solution Touch-panel suppliers are very willing to get into the cover-glass business (vertical integration) TPK spent $250M capex on cover-glass equipment in 2011 Touch-panel suppliers can get LCDs on consignment or buy/sell and build the complete module 37

38 OGS versus Embedded 5 The influence of the controller suppliers It s fundamentally the same controller for OGS & embedded Simple case: Touch controller & display driver communicate (linked) What s in today s smartphones with embedded touch Optimized case: Touch controller is integrated with display driver It s a specialized, resolution-specific ASIC The controller supplier can swing the pendulum Synaptics wants to work very closely with LCD suppliers to create optimum embedded designs Atmel is considering the embedded opportunity, but it s far from a defined strategy for them Cypress? The 20+ other controller suppliers? Do the controller suppliers want to trade 30 module customers for 5 LCD customers? 38

39 OGS versus Embedded 6 The problem of LCD product-management An LCD with embedded touch is a different product In many cases the LCD supplier may have two versions of a given LCD one with touch and one without The touch version will cost more, so it can t be the only version 39

40 Miscellaneous P-Cap Construction Issues Sheet method vs. Piece method Sheet: Harden first, then cut (requires 2 nd strengthening) Example: Wintek (cumulative yield = 75% typical) Piece: Cut first, then harden Example: TPK (cumulative yield = 60% typical) Bonding to LCD OCA (film) vs. OCR (liquid) Trend is towards OCR Cover-glass material Industry would like to switch from glass to plastic (e.g., PMMA) Apple s flooded-x patent for self-shielding Currently in lawsuit; the author believes the patent will be invalidated due to prior art 40

41 Projected Capacitive 17 Simplified p-cap controller block diagram C mutual Analog Front- End (AFE) Analogto-Digital Converter (ADC) Digital Signal Processor (DSP) Sensor Driver Touch Sensor Touch Controller Source: Display- Search 41

42 Projected Capacitive 18 P-cap controller suppliers Atmel Cypress Synaptics vs. AD Semi AMT Azoteq Cirque EETI EMC FocalTech Ilitek Maxim Melfas M-Star Pixcir Raydium Renasas Samsung Sentelic SiS Silicon Labs Sitronix ST Micro TI Weltrend 42

43 Projected Capacitive 19 Controllers Key variable is the number of electrodes (matrix size) Larger screens generally require multiple (ganged) controller chips High signal-to-noise ratio (SNR) is a key characteristic enables passive stylus Controller innovation areas Performance, power, cost, stylus & palmrejection, hover, gloves, water-resistance, pressure, gestures, proximity, haptics When will controller commoditization start to be significant? Source: Synaptics LG-Prada mobile phone with Synaptics p-cap touchscreen; launched 3 months before the iphone! 43

44 Projected Capacitive 20 Options (ITO-based ) Top-surface treatment (AR, AG, AF, AC, AB ) Degree of indexing matching on ITO (invisibility) Stackup variations, as already described Number of electrodes per inch (dpi/resolution) Size range 2 to ITO up to 32 (3M s 46 is a special case) Wires up to

45 Projected Capacitive 21 Advantages Unlimited multi-touch Extremely light touch (zero force) Enables zero-bezel industrial design High optical quality (with index-matched ITO) Very durable (protected sensor) Unaffected by debris or contamination Works with curved substrates (on PET) Disadvantages Finger or tethered pen only This is rapidly changing! High cost Mostly in the sensor; ITO replacements will help Challenging to integrate due to noise sensitivity & tuning 45

46 Integrating a P-Cap Touchscreen Into a Mobile Device After the mechanical & industrial design are done, it s really all about just one thing: Tuning Every new product must have the p-cap touch-screen controller tuned to account for all the variables in the configuration Basic configuration (e.g., OGS vs. embedded) Sensing pattern Glass thickness Adhesive thickness LCD noise LCD frame mechanics Air-gap or bonded etc. All controller manufacturers either supply tools (e.g., Synaptics Design Studio 4 ) or they do it themselves for their OEM customers 46

47 Projected Capacitive 22 Applications Consumer devices Mobile phones Tablets, Ultrabooks, AiOs Almost any consumer device Vertical-market devices Signature-capture terminals Through-glass interactive retail signage Source: Lenovo Source: HP Source: CNET Source: Verifone 47

48 Projected Capacitive 23 In the USA, this is probably the fastest-growing commercial p-cap application Goodbye, Resistive! Source: Verifone 48

49 Projected Capacitive 24 Adoption of P-Cap In Commercial Markets (Forecast) Healthcare Rapid, within FDA-cycle constraints Buying for the future with a very long product life Zero-bezel, multi-touch, light touch are all important Gaming Rapid, within gaming regulation constraints Casinos want to attract the Millennium Generation Multi-touch is very important; zero-bezel is less so Point of Information Moderate Software-driven; zoom gesture could be the key Industrial Slow Multi-touch may be important; zero-bezel & light touch are less so Point of Sales Very slow Zero-bezel is the only driver; flat-edge resistive is good enough 49

50 Projected Capacitive 25 Most commercial-application users are also consumers, so expectations driven by the de facto standard will help push p-cap into commercial applications BUT, it won t be super-fast Sensor cost is still a big problem, with low yield in sizes larger than tablets Integration still requires a little magic Application requirements are always tougher in commercial applications Product lifetimes are much longer Few touchscreen suppliers are focused on commercial, since consumer is so big Source: 3M 50

51 Projected Capacitive 26 Suppliers Modules TPK (biggest), Wintek, Nissha, Panjit, Digitech, CMI, Young Fast, Touch International, 3M, Ocular, and >20 more Sensors (only) Laibao, Cando (now part of TPK), CSG, HannsTouch, Nissha, and >10 more Controllers (only) Atmel, Cypress, Synaptics, Maxim, Avago, Pixcir, Sitronix, EETI, SIS, Melfas, MasTouch (now park of TPK), Focal Tech, TI, and >10 more Supplier countries Taiwan, USA, China, Japan, Korea, UK, Israel, South Africa 51

52 Projected Capacitive 27 Market trends P-cap has become a de facto standard (it s won the war!) Growth is starting to slow down Capacity expansion continues but at a slower pace Top three controller suppliers account for ~70% of revenue Top five module suppliers account for ~50% of revenue Intel announced that all Haswell (2013) Ultrabooks must have touch Market research firms are expecting more notebook penetration Commercial applications are beginning to transition to p-cap A few small-order suppliers are appearing, but it s still hard to buy The technology name has changed to just capacitive 52

53 Large-Format P-Cap 1 One more sensor variation: 10-micron wires between two sheets of PET or glass Commonly used for large-format touchscreens Two main suppliers: Visual Planet & Zytronic, both in the UK 9 floor-to-ceiling Visual Planet touchscreens in the University of Oregon Alumni Center Source: The University of Oregon 53

54 Large-Format P-Cap 2 3M has managed to get ITO electrodes to work in a 46-inch display (larger than any other with ITO) They won t disclose their secret sauce Source: Photo by Author 54

55 Large-Format P-Cap 3 Jeff Han from Perceptive Pixel (acquired by Microsoft in mid-2012) showed an 82 at CES 2012 (with active stylus) and a 72 at Digital Signage Expo (DSE) 2012 Metal electrodes (not ITO) although Jeff wouldn t talk about the electrode material or who is manufacturing the touchscreens Source: Photos by Author 55

56 Large-Format P-Cap 4 Both the 72 & 82 look much better than the traditional Zytronic zig-zag 10-micron wire pattern 72 at DSE 2012 Zytronic wires 72 at DSE 2012 Source: Photos by Author 56

57 Large-Format P-Cap 5 Large-format multi-touch applications Source: Zytronic 57

58 Large-Format P-Cap 6 Zytronic s new multi-touch large-format p-cap Previous Zytronic products were self-capacitive (2-touch max) Binstead s frequency-variation patent was the basis of sensing New product is mutual-capacitive with very dense electrode pattern Traditional measurement of capacitance reduction caused by finger ~1.5 mm electrode spacing in 6 mm x 6 mm cell Density reduces visibility because the human visual system sees a more uniform contrast 10-micron insulated copper wires allow crossover ( single layer ) 100 s Ω/m at 10 µm Can be applied to glass or film (including curved surfaces) Initial controller handles all sizes up to 72 ; possible Minimum 10 touches with palm rejection 58

59 Large-Format P-Cap 7 Applications for curved large-format touchscreens Source: Zytronic 59

60 ITO Replacement Materials Source: Asylum Research (800 nm scan of ITO) File Download: 60

61 ITO Replacements 1 Why replace ITO? Costly to pattern & needs high temperature processing Highly reflective (IR = 2.6) & tinted yellow; brittle & inflexible Relies on potentially politically unstable Asian zinc mines* Replacement material objectives Solution processing (no vacuum, no converted LCD fab) Higher transmissivity & lower resistivity (better than ITO) Lower material & process cost than ITO Six replacement candidates Metal mesh Silver nanowires Carbon nanotubes Conductive polymers Graphene ITO inks * 63% of estimated 2007 production of indium 61

62 ITO Replacements 2 Metal mesh has started shipping in touchscreens, and it s looking better than silver nanowires MNTech in Korea was the first to ship metal-mesh at the end of 2012 Atmel (partnered with CIT in the UK) was the second to ship metalmesh (XSense ) for a smartphone and a 7 tablet in 2H-2012 FujiFilm started production of their silver-halide-based metal-mesh product in 2Q-2013 Unipixel should start production of UniBoss in 2H-2013 Metal mesh roll-to-roll printable in a single pass at room temperature They re one of the very few suppliers using printing for patterning: almost everyone else uses photolithography ( roll-to-roll with pauses ) Many other companies are entering this market; some in China are already shipping quietly 62

63 ITO Replacements 3 Metal mesh has significant advantages Patterning via printing allows both operating and capex cost to be very low Electrodes and border connections are printed simultaneously, which allows borders as narrow as 3 mm (typically 9 mm with ITO) Sheet resistivity is much lower than ITO (under 10 ohms/square) Reduces p-cap charge time, which allows larger touchscreens Increases SNR and linearity Mesh pattern creates electrical redundancy, which improves yields Transparency is better than ITO Highly flexible bend radius typically 4 mm Optical problems have been solved Invisible black-colored 4-micron mesh, with no moiré pattern (with up to dpi LCDs) 63

64 ITO Replacements 4 However, there is one significant problem in the current metal-mesh supply-chain Film-based module-makers ( back-end suppliers) such as O-Film and Young Fast are all making their own metal mesh They don t want to use other suppliers mesh; if they re forced to do so by an OEM, they increase the price to make up the missing margin This is making it increasingly difficult for mesh-only suppliers (Atmel, Fujifilm, Unipixel, etc.) to market their product It s also masking most of the cost-reduction potential of metal mesh 64

65 ITO Replacements 5 Silver nanowires Cambrios Synthesis of inorganic material (e.g., silver) from soluble precursors, followed by assembly of the resulting materials into nanostructures Cambrios has been coating rolls of PET with their material ( ClearOhm ) in a roll-to-roll production facility since early 2007 Cambrios was actually the first to ship an ITO-replacement in a consumer electronics product Others Source: Nikkei Business Publications PolyIC, Sinovia, Sigma Technologies, Carestream, Ferro, Suzhou NanoGrid, Saint-Gobain, Cima NanoTech, Blue Nano, and others 65

66 ITO Replacements 6 Carbon nanotubes (CNTs) Leaders have changed over the last few years Was Eikos and Unidym Now C3Nano, Canatu, Toray, SWeNT and others Traditional CNTs have separation and purification problems Performance isn t good enough yet to beat silver nanowires Canatu s Carbon NanoBuds (nanotubes + buckyballs) seem to be the best current bet, with high-volume production in 2014 Better optical performance than silver nanowires Very low reflectivity and lower haze Dry patterning, allows printing electrodes and connections simultaneously (like metal mesh) More flexible (bend radius 0.5 mm!) 66

67 ITO Replacements 7 Conductive polymers (e.g., PEDOT/PSS)* Kent Displays is currently using them; Fujitsu stopped Fujitsu claimed 5X to 10X longer resistive touch-screen lifetime but when the price of indium dropped, they couldn t justify continuing Fujitsu claimed to be doing roll-to-roll film manufacturing BUT, conventional wisdom is that PEDOT has inferior transparency and degrades under UV Development leaders Agfa, Heraeus, Kodak Issues Low performance * poly(ethylene dioxythiophene) / poly(styrene sulfonate) 67

68 ITO Replacements 8 Graphene Like unrolled carbon nanotubes, a one-atom thick sheet Promising strength, transparency, and conductivity, but development is still in its infancy Development leaders Vorbeck Materials Issues Difficult to produce defect-free layers of graphene Ripples or multiple layers reduce electrical and optical properties Difficult to prevent interaction with the substrate Results in reduced conductivity Currently manufactured with an expensive vacuum process No scalable, low-cost method of making single layers of graphene (yet) 68

69 ITO Replacements 9 ITO inks On-again, off-again market interest NanoMarkets forecast is effectively zero through 2017 ITO ink is nanoparticles (e.g., 10 nm) of ITO dispersed in a solvent with additives Leaders are Sumitomo and Ulvac Can be inkjet-printed at atmospheric pressure, but requires high-temperature (450 C) thermal annealing to achieve lowest sheet resistivity That means realistically it s a glass-only material (not roll-to-roll) Nobody s currently using it in touchscreen-sensor production Performance to date hasn t been good enough Metal mesh and/or silver nanowires seem much more promising 69

70 ITO Replacements 10 Realities (summary) It s about the ITO in touchscreens, not in LCDs ITO used in LCDs is 1-2% of cost (~$4 for a 40 display) LCD makers are extremely reluctant to make changes in fabs It s not really about flexible displays, at least not right now It s not really about the indium supply or cost It s about the processes that it requires, not about the ITO itself The dominance of patterned-ito touchscreens (p-cap) over uniform-ito touchscreens (resistive) has changed the picture A 13 p-cap tablet touchscreen is $25 sensor, $5 controller; that $25 sensor could come down to under $10 70

71 ITO Replacements 11 Predictions Most current capital-intensive, glass (fab)-based, p-cap module suppliers will resist printed ITO replacements because they have to maintain a targeted return on their invested capital ITO replacements represent a competitive threat to them Film-based module suppliers, formerly second-class citizens, now have a powerful weapon that allows them to compete head-to-head with glass-based module suppliers Printed sensor-film (only) producers may be facing a serious supply-chain problem Five years from now, as much as 50% of p-cap sensors will be made using an ITO-replacement material 10 years from now, p-cap fabs will be like many passive-lcd fabs today (fully depreciated and unused) 71

72 ITO-Replacement Startup: ClearJet ClearJet (Israel) Inkjet-printing silver nano-particle drops < 10 µm thick Ink dries from center outward, leaving coffee rings ~100 µm 95% transparency, 4 ohms/square resistivity 72

73 Surface Capacitive Source: 3M File Download: 73

74 Surface Capacitive KHz Tail Scratch-resistant top coat Hard coat with AG Electrode pattern Conductive coating (ATO, ITO or TO) Glass Source: 3M Optional bottom shield (not shown) Source: 3M 74

75 Surface Capacitive 2 Variations Rugged substrate Size range 6.4 to 32 Controllers 3M, Microchip (Hampshire), egalax, and Digitech Advantages Source: 3M Excellent drag performance with extremely smooth surface Much more durable than analog resistive Resistant to contamination Highly sensitive (very light touch) Source: Interactive Systems 75

76 Surface Capacitive 3 Disadvantages No multi-touch Finger-only (or tethered pen) Calibration drift & susceptible to EMI Moderate optical quality (85% - 90%) Applications Regulated (casino) gaming Point-of-Sale (POS) terminals Point-of-Information (POI) kiosks Medical equipment Suppliers 3M is the only significant supplier left Status Source: 3M It will be an irrelevant, obsolete technology in 5-7 years 76

77 Wacom s Improved RRFC Surface Capacitive Technology 1 How it works A linear voltage AND a ramp-shaped electrostatic field is created on the surface by applying AC on 2 corners & DC on the other two corners Controller switches signals around all 4 corners, creating 4 ramp fields vs. single flat field in standard capacitive, and measures current in each case Resulting touch-event signal is independent of all capacitance effects except those due to finger-touch Controller does additional digital signal processing to compensate for factors that affect accuracy and drift Source: Wacom (Trademark = CapPLUS) RRFC = Reversing Ramped Field Capacitive 77

78 Wacom s Improved RRFC Surface Capacitive Technology 2 Advantages Solves all the problems of traditional surface capacitive Works in mobile & stationary devices (10 to 32 now; 46 capable) Unaffected by grounding changes, EMI, variations in skin dryness & finger size, temperature, humidity, metal bezels, etc. Works outdoors in rain and snow Works through latex or polypropylene gloves Allows 4X thicker hardcoat for improved durability Uses same ASIC as Wacom s EMR pen digitizer, so dual-mode input is lower cost & more efficient (e.g., in Tablet PC) Disadvantages (2 very big ones!) No multi-touch Sole-source supplier 78

79 Resistive Touch Technologies Analog Resistive Analog Multi-Touch Resistive (AMR) Digital Multi-Touch Resistive 79

80 Analog Resistive Source: Engadget File Download: 80

81 Analog Resistive 1 (ITO) (PET) Source: Bergquist Source: Elo Touch Solutions 81

82 Analog Resistive 2 (4-Wire Construction) X-Axis Voltage gradient applied across glass Voltage gradient applied across coversheet Equivalent circuit Voltage measured on coversheet Bus bars Y-Axis Voltage measured on glass 82

83 Analog Resistive 3 (5-Wire Construction) X-Axis Voltage gradient applied across glass in X-axis Voltage gradient applied across glass in Y-axis Equivalent circuit Contact point on coversheet is a voltage probe Linearization pattern Y-Axis Contact point on coversheet is a voltage probe 83

84 Analog Resistive 4 Types 4-wire (low cost, short life) is common in mobile devices 5-wire (higher cost, long life) is common in stationary devices 6-wire & 7-wire = obsolete 5-wire; 8-wire = replacement only Constructions Film (PET) + glass (previous illustration) is the most common Film + film (used in some cellphones) can be made flexible Glass + glass is the most durable; automotive is the primary use Film + film + glass, others Options Surface treatments (AR, AG, AF, AC, AB), rugged substrate, dual-force touch, high-transmissivity, surface armoring, many others 84 (50-uM glass) Source: Schott

85 Analog Resistive 5 Size range 1 to ~24 (>20 is rare) Controllers Many sources Single chip, embedded in chipset/cpu, or universal controller board Advantages Works with finger, stylus or any non-sharp object Lowest-cost touch technology Widely available (it s a commodity) Easily sealable to IP65 or NEMA-4 Resistant to screen contaminants Low power consumption Source: Liyitec Source: Microchip 85

86 Analog Resistive 6 Disadvantages Not durable (PET top surface is easily damaged) Poor optical quality (10%-20% light loss) No multi-touch Applications Mobile devices (shrinking) Point of sale (POS) terminals Automotive Industrial Wherever cost is #1 Source: Sinocan Source: Renu 86

87 Analog Resistive 7 Suppliers Young Fast, Nissha, Nanjing Wally, Truly, EELY, Mutto, J-Touch 60+ suppliers Market trends Analog resistive is shrinking in units and revenue P-cap dominates in most consumer applications Analog resistive is still significant in commercial applications Especially POS and industrial-control terminals 87

88 Analog Multi-Touch Resistive Source: Touch International File Download: 88

89 Analog Multi-Touch Resistive 1 Multiple names AMR (Analog Multi- Touch/Matrix Resistive) MARS (Multi-Touch Analog Resistive Sensor) Hybrid analog-digital Primary limitation Can t touch with two fingers on the same square Typical AMR design for consumer product Source: The Author 89

90 Analog Multi-Touch Resistive 2 Actual Product 21.5 analog multi-touch resistive by eturbotouch 28 x 17 lines = 17 mm x 16 mm squares (90 pins) 23 = 35 x 22 lines 15 mm x 13 mm (114 pins) 90

91 Analog Multi-Touch Resistive 3 Gateway ZX6910 AiO with 23 AMR touchscreen from eturbotouch Example of a failed consumer product with 15x13 mm AMR Drawing parallel lines with two closely held fingers Source: Photos by Author 91

92 Analog Multi-Touch Resistive 4 Controllers AD Semi & others; some home-grown (e.g., Touch International) Suppliers eturbotouch, Touch International, Mildex, Mutto, EETI Advantages Multi-touch (but without two touches on the same square) Simple & familiar resistive technology Lower cost than p-cap Disadvantages Poor durability (PET top surface) Poor optical performance Non-zero touch force Applications Industrial & other commercial Source: Apex 92

93 Digital Multi-Touch Resistive File Download: 93

94 Digital Multi-Touch Resistive 1 Stantum s product (ivsm) Interpolated Voltage-Sensing Matrix Stantum s strategy is to license controller IP to IC manufacturers, not to sell touchscreens Aimed at tablets Fine pitch results in a much higher number of connections than AMR (400+ on a 10 tablet screen) I/O s per controller Source: Author 94

95 Digital Multi-Touch Resistive 2 Controllers ST Micro is currently the only one Number of touch points is controller-dependent (2-10) Advantages Multi-touch Simple & familiar resistive technology Lower cost than p-cap Disadvantages Poor durability (PET top surface) Poor optical performance Non-zero touch force Applications Commercial mobile applications such as education 95

96 Digital Multi-Touch Resistive 3 Stantum s successes (against a BIG P-cap headwind) Co-developed a pen & finger solution with Nissha for 5.7 to 12-inch tablets Licensed IP to a US-based semiconductor vendor developing a controller optimized for 5.7 to 12 tablets Design win with a tier-1 OEM for a pen & finger A4 e-reader targeted at education and note-taking Two 7 tablets for military applications (one by Harris) 10.4 professional lighting-control application (Europe) Signed a licensing agreement with a tier-1 OEM for a mobile enterprise tablet 96

97 Digital Multi-Touch Resistive 4 One of Stantum s shipping (military) OEM products A new 7-inch Android tablet that's so hard-as-nails it would make a Galaxy Tab go home and call its mother (Engadget) Source: Harris 97

98 Digital Multi-Touch Resistive 5 The funny thing is, Stantum s original products were the first commercial products to use multi-touch! In 2005, when the company was selling music controllers under the name Jazz Mutant Source: Jazz Mutant 98

99 Acoustic Touch Technologies Surface Acoustic Wave (SAW) Acoustic Pulse Recognition (APR by Elo) Dispersive Signal Technology (DST by 3M) 99

100 Surface Acoustic Wave Source: Kodak File Download: 100

101 Surface Acoustic Wave 1 Glass substrate (45 ) Source: A-Touch Rayleigh wave Source: Onetouch 101

102 Surface Acoustic Wave 2 Source: Elo Touch Solutions 102

103 Surface Acoustic Wave 3 How two touches are supported by SAW X & Y reflectors Source: US Patent Application 2010/ Diagonal reflectors for third axis data 103

104 Surface Acoustic Wave 4 Both Elo Touch Solutions and General Touch (China) are emphasizing zero-bezel and two-touch SAW This makes sense because SAW and Win7/8 will be important in commercial applications for at least the next five years Both companies put the piezos and reflectors on the back of the glass to achieve zero-bezel For two-touch zero-bezel, Elo uses a single set of multiplexed reflectors on the back of the glass (see US ) instead of the two sets of reflectors used on top of the glass for two-touch normal bezel 104

105 Surface Acoustic Wave 5 Elo Touch Solutions zero-bezel SAW Source: Photos by Author 105

106 Surface Acoustic Wave 6 Size range 6 to 52 (but some integrators won t use it above 32 ) Advantages Clear substrate (high optical performance) Finger, gloved hand & soft-stylus activation Very durable; can be vandal-proofed with tempered or CS glass Disadvantages Very sensitive to any surface contamination, including water Relatively high activation force (50-80g typical) Requires soft (sound-absorbing) touch object Can be challenging to seal Applications Kiosks Gaming Source: Euro Kiosks Network 106

107 Surface Acoustic Wave 7 Suppliers Elo Touch Solutions and General Touch have >90% share <10 suppliers Market trends Two-touch and zero-bezel SAW should help reduce loss of share to p-cap in commercial applications SAW will continue to grow moderately through 2017 Chinese suppliers other than General Touch have significant difficulty competing due to distribution and brand limitations 107

108 Acoustic Pulse Recognition (APR) Source: Elo TouchSystems File Download: 108

109 Acoustic Pulse Recognition 1 Bending waves Plain glass sensor with 4 piezos on the edges Table look-up of bending wave samples ( acoustic touch signatures ) Source: Elo Touch Solutions 109

110 Acoustic Pulse Recognition 2 Variations Stationary APR from 10 to 52 with controller board Mobile APR from 2.8 to 10 with controller ASIC Advantages Works with finger, stylus or any other touch object Very durable & transparent touch sensor Very simple sensor (plain glass + 4 piezoelectric transducers) Resistant to surface contamination; works with scratches Totally flush top surface ( Zero-Bezel ) Disadvantages No touch & hold ; no multi-touch Requires enough touch-object velocity (a tap) to generate waves Control of mounting method in bezel is critical 110

111 Acoustic Pulse Recognition 3 Outlook: Not good! It s not available as a component (touchscreen) because it requires unique calibration and specialized integration Unsuitable for applications that use the Windows UI because of the lack of touch-and-hold Unsuitable for public-access applications because of the need to tap (everyone today expects p-cap s light touch) Unsuitable for consumer electronics applications because of the lack of multi-touch Elo Touch Solutions (sole-source!) withdrew APR from digital signage applications because of poor performance (they re using Lumio s camera-based optical instead) What s left? POS terminals! 111

112 Acoustic Pulse Recognition 4 APR and Sensitive Object Elo Touch Solutions (then part of Tyco Electronics) purchased Sensitive Object ( S.O. ) in January, 2010 for $62M Sensitive Object s technology ( ReverSys ) is so similar to APR that the two companies cross-licensed in July, 2007 In the 3 years since Elo purchased S.O., there have been zero new products that can be attributed to the acquisition Source: Sensitive Object 112

113 Dispersive Signal Technology (DST) Source: 3M File Download: 113

114 Dispersive Signal Technology 1 Shielded silver trace to piezo Piezo Plain glass sensor with 4 piezos in the corners Real-time analysis of bending waves in the glass ( time of flight calculation) Source: 3M 114

115 Dispersive Signal Technology 2 Visualization of the effect of bending waves on a rigid substrate Source: 3M Waveform that would be sampled by APR Waveform resulting from processing by DST algorithms 115

116 Dispersive Signal Technology 3 Size range 32 to 55 (available only on displays sold by 3M-trained integrators) Advantages Works with finger, stylus or any other touch object Very durable & transparent touch sensor Very simple sensor (plain glass + 4 piezoelectric transducers) Operates with static objects or scratches on the touch surface Fast response; highly repeatable touch accuracy; light touch Disadvantages No touch & hold ; no multi-touch Control of mounting method in bezel is critical Applications Interactive digital signage; point-of-information (POI) Status: 3M has discontinued all new development 116

117 Acoustic Touch Startup: Sentons Sentons The next generation in high-performance multi-touch interfaces A fabless analog semiconductor startup Rumored to be doing something new in acoustic/bending-wave, with first products in 2014 Five+ people from Telegent, the analog mobile-tv chip startup that flamed out in July 2011 Source: Sentons Website 117

118 Optical Touch Technologies Traditional Infrared (IR) Waveguide Infrared (DVT by RPO) Multi-Touch Infrared Camera-Based Optical Planar Scatter Detection (PSD) Vision-Based 118

119 Traditional Infrared File Download: 119

120 Traditional Infrared 1 Cross-beam light paths increases resolution and fault-tolerance in infrared touchscreens (Elo) Source: Elo Touch Solutions 120

121 Traditional Infrared 2 Variations Bare PCB vs. enclosed frame; frame width & profile height; no glass substrate; enhanced sunlight immunity; force-sensing Size range 8 to 150 Controllers Mostly proprietary, except IRTouch (China) Advantages Scalable to very large sizes Multi-touch capable (only 2 touches, and with some ghost points) Can be activated with any IR-opaque object High durability, optical performance and sealability Doesn t require a substrate 121

122 Traditional Infrared 3 Multi-touch in traditional infrared Limited to 2 not-so-good touches Ghost points are the problem, and there s no good solution Source: Drawing by Author 122

123 Traditional Infrared 4 Disadvantages Profile height (IR transceivers project above touch surface) Bezel must be designed to include IR-transparent window Sunlight immunity can be a problem in extreme environments Surface obstruction or hover can cause a false touch Low resolution High cost Applications Large displays (digital signage) POS (limited) Kiosks Suppliers IRTouch Systems, Minato, Nexio, OneTouch, SMK, Neonode 10+ suppliers 123

124 Traditional Infrared 5 Mobile Infrared: Neonode mobile phone implemented with traditional IR touch (2009) Same battery life as iphone Low bezel profile height (~1.7mm) Finger-only No multi-touch Neonode couldn t complete in the cellphone market and went bankrupt in 2009 Source: Neonode & Pen Computing Sony e-book readers (2010) Source: PC World 124

125 Traditional Infrared 6 Neonode in 2013 has become the largest supplier of touchscreens for ereaders! Amazon Kindle and B&N Nook both use Neonode Neonode has strong IP on methods of minimizing border width and profile height They re also in transition from traditional infrared architecture to multi-touch infrared architecture Neonode has announced design wins in e-readers, smartphones, tablets, toys, printers, gaming consoles, in-flight infotainment systems, and automotive consoles How much of it is real beyond e-readers is unclear Neonode doesn t supply any actual hardware, just licenses and engineering implementation consulting services 125

126 Waveguide Infrared Source: RPO File Download: 126

127 Waveguide Infrared 1 Objective Reduce IR touchscreen cost by replacing multiple IR-emitters with a single LED and using optical waveguides to distribute the light and to channel it to a line-scan CMOS pixel array Traditional Infrared Source: RPO 127

128 Waveguide Infrared 2 RPO s actual construction (3.5 screen) Source: Photo by RPO; Annotation by Author 128

129 Waveguide Infrared 3 RPO timeline Announced IR optical-waveguide infrared touch at SID Showed improved performance at SID Showed larger sizes at SID Appeared in a 13.3 LG Display notebook at SID 2010 Went into voluntary administration (liquidation) in April 2011 Sold all assets to an NPE (patent troll) in February 2012 (along with Poa Sana s assets it s a long story!) Why did it fail? There wasn t any particular application for which it was best Waveguide technology limited touchscreen size to under ~14 RPO bet on one big partner in 2010 who cancelled their project abruptly, leaving the company with insufficient $$$ to keep going 129

130 Multi-Touch Infrared Source: Citron File Download: 130

131 Multi-Touch Infrared 1 A little bit of history on the 2 nd -oldest touch technology IR touch first appeared in 1972 (PLATO IV instructional terminal) IR touch was used in HP s first microcomputer, the HP150, in 1983 After 30+ years of stability, it s changed from single-touch to (briefly) 2-touch, and now multi-touch! Source: University of Illinois Source: VintageComputing.com 131

132 Multi-Touch Infrared 2 PQ Labs method = IR emitter = IR receiver 6 to 32 touches 32 to 103 Source: Author 132

133 Multi-Touch Infrared 3 PulseIR (Image Display Systems) method = IR emitter = IR receiver 2 to 40 touches 5 to 103 Source: Author 133

134 Multi-Touch Infrared 4 TimeLink method = IR emitter = IR receiver 10 to 48 touches 15 to 100 Source: Author 134

135 Multi-Touch Infrared 5 Variations Number of touch points: 2 to 48 (determined by the controller) Architecture: 3 different ways of organizing the IR emitters and receivers (so far) PQ Labs is licensing; others may be also Controller Proprietary; generally requires a large amount of processing Advantages High number of multi-touch points Object-size recognition Controller maintains position & size data for all touching objects Similar advantages to those of traditional infrared Works with a finger, stylus or any other IR-opaque touch object Scalable to very large sizes (at some cost) High durability and sealability 135

136 Multi-Touch Infrared 6 Disadvantages Relatively low resolution (can get stair-stepping in lines) Increased processing load as size and number of touches goes up Different minimum-object-size spec for stationary & moving objects Large objects close to emitters can decrease performance As with any traditional IR system, pre-touch (or pen-up ) is mostly a big problem that gets worse as the screen size increases Most can t meet Win8 Logo due to pre-touch and accuracy Applications Multi-player games on large horizontal displays Multi-user interactive digital signage 3D design and interaction; data visualization for business NOT interactive whiteboard displays due to pre-touch/pen-up 136

137 Multi-Touch Infrared 7 Latest new multi-touch infrared product: Projected Infrared Touch (PIT) from General Touch Proprietary design using traditional opto layout (like PQ Labs) Meets Win8 Logo Bezel is a light-guide/prism (2.5 mm high, 4 mm wide) that allows IR emitters & receivers to be located under the cover-glass, outside the LCD frame (reduced parallax due to no PCB on top) Source: General Touch 137

138 Multi-Touch Infrared 8 Additional PIT features 15 to 42 size range standard; over 42 is custom First sizes to launch in 2Q-2013 are 21.5 & 23 (for AiO) 2-touch for lowest cost; 5-touch for Win8; 10-touch for high-end Only the controller changes Entire surface is touch-active, including the 20 mm (MS) border Active icons can be silk-screened in the border s black matrix Pre-touch meets the Win8 spec of 0.5 mm Exceptionally low for any infrared touchscreen Touch surface can be any material that meets surface flatness spec Can be sealed to IP65 138

139 Camera- Based Optical This picture was drawn on a 46" LCD equipped with a NextWindow optical touch-screen by a visitor to the AETI Exhibition in London on January 24, File Download: 139

140 Camera-Based Optical 1 Win7 = 2 touches; 2 cameras did it (inadequately) Win8 = 5 touches; 6 cameras are required Source: NextWindow 140

141 Camera-Based Optical 2 Two touches with two cameras (Win7 market focus) had two main limitations Ghost touches Source: Author Occlusions The quality of the touch experience depended on the sophistication of the algorithms that handled ghost touches and occlusions 141

142 Camera-Based Optical 3 Another alternative: A mirror creates virtual cameras Source: Lumio Results SMART invented it in 2003 but shelved it Lumio tried it in 2010 but found that four real cameras were better Lower cost No mirror alignment issues Less sensitivity to environment Fewer pixels required for same resolution Less CPU processing 142

143 Camera-Based Optical 4 Another alternative: Baanto ShadowSense Baanto s sensors aren t cameras; they re PIN diodes (photodetectors) 940 nm LEDs provide back illumination instead of retroreflectors Source: Baanto ( FOV = Field of View) 143

144 Camera-Based Optical 5 Baanto s competitive comparison 4 sources of sensor-data work much better than two! Suppliers NextWindow (SMART) Lumio IRTouch Xiroku/eIT Baanto LG Displays Qisda Several more in China Source: Baanto 144

145 Camera-Based Optical 6 Advantages Stylus independence Scalability to large sizes (15 to 120 ) Multi-touch (2-5 touches) Object-size recognition Low cost Disadvantages Profile height (~3 mm on a 19 screen) The unintended touch problem Screen rigidity requirement Applications Consumer touch monitors & AiOs HP TouchSmart all-in-one computer Source: HP Interactive digital signage, point-of-information, & education 145

146 Camera-Based Optical 7 Optical is able to meet the Win8 touch specifications NextWindow s newest desktop-component product (15 to 30 ) uses six CMOS cameras (4 in the corners + 2 on the top edge) Source: Jennifer Colegrove (DisplaySearch) at FineTech

147 Camera-Based Optical 8 The range of form-factors and configurations in which optical touch is used is expanding Cameras & lasers at top of video wall Source: Photos by Author 147

148 Camera-Based Optical 9 Outlook Touch on the consumer desktop (in Win7 AiOs) failed to take off due to lack of any applications Touch penetration hit 30% in 2010 but dropped to ~10% in 2012 Win8 may drive more penetration, but there is still the gorilla arm usage-model question Adaptive AiOs will help address that issue Camera-based optical touch is ideal for large-format, but The interactive digital-signage market hasn t emerged yet Interactive information on large screens is still a niche market The education market (whiteboards) has been slow to adopt optical because of entrenched resistive and electromagnetic technologies 148 Dell ST2220T (Win7) Touch Monitor

149 Planar Scatter Detection (PSD) Source: FlatFrog File Download: 149

150 Planar Scatter Detection 1 IR Emitter Total Internal Reflection (TIR) IR Receiver Substrate (any transparent material, flat or curved) Scattered Light (FTIR) Source: FlatFrog 150

151 Planar Scatter Detection 2 Characteristics it shares with p-cap Flush surface ( zero bezel ) Very light touch Multi-touch (40 touches) Windows 8 Logo Characteristics that are better than p-cap Plain glass or plastic substrate (0.3 mm) no ITO Works with glove, stylus or other objects (400 dpi) Pressure-sensitive (10 bits) Insensitive to EMI/RFI High scan-rate for larger screens (up to 1 KHz) Lower cost Source: FlatFrog 151

152 Planar Scatter Detection 3 Size range 32 (with display) at launch in May, 2012 Practical size range 15 to 84 Disadvantages Initial product is a 32 display for $5,500 MSRP (+$190 housing) Designed for indoor use (no sunlight) without dust or smoke Limited to 30 C ambient due to display Sensitive to contamination on surface Scaling to larger sizes is similar to traditional infrared ~200 IR emitter-receiver pairs required for 32 display; 96 pairs for 22 FlatFrog is a small company with limited resources 152

153 Planar Scatter Detection 4 Applications Realistic today: Gaming, digital signage, POI, medical, hospitality, command & control Future: consumer electronics, education Full disclosure: Source: FlatFrog Intel has invested in and is doing a joint development project with FlatFrog to extend and improve the technology beyond what they have already done Supply-chain (availability) will also be improved 153

154 Optical-Touch Startup: RAPT RAPT Opto-electro-mechanical (rumored to be similar to FlatFrog) The most robust multi-touch system on the planet and quite different from current solutions in the market Source: RAPT Website 154

155 Vision- Based Source: Perceptive Pixel File Download: 155

156 Vision-Based 1 Principle (simplest version) Total Internal Reflection (TIR) LED Acrylic Pane Multiple touch points; Image taken without a diffuser (Source: Perceptive Pixel) Baffle Diffuser Scattered Light Projector Frustrated Total Internal Reflection (FTIR) Video Camera Source: Perceptive Pixel 156

157 Vision-Based 2 Microsoft Surface (v1, 2007) Surface computing is about integrating the physical and virtual worlds through the use of vision-based touch Source: Information Display Projector resolution 1024x Touch resolution 1280x960 Source: Popular Mechanics 5 1 Screen with diffuser 2 IR LED light source 3 Four IR cameras 4 DLP projector 5 Vista desktop 157

158 Vision-Based 3 Samsung SUR40 with Microsoft Surface (v2.0, 2012) 4 thick Document on surface Source: TechCrunch.com Source: Microsoft 158

159 Vision-Based 4 Samsung SUR40 40 full-hd (1920x1080) Samsung LCD (55 ppi) 4 thickness includes 2.9 GHz PC with embedded 64-bit Win-7 Corning Gorilla Glass bonded to LCD Display still has some bezel height (not a flush surface) In-cell touch: 8 display pixels per asige IR light sensor (8 ppi) By far the most sophisticated in-cell light-sensing so far IR light source is added to the backlight asige sensor is 15X more sensitive than asi, but that means the touch-screen is 15X more sensitive to ambient IR 50+ simultaneous touch points Surface image-processing software is Microsoft s primary value-add $8,400 targeted at enterprise Microsoft has a 3-4 year exclusive on the SUR40, which means that Samsung doesn t see much value for themselves 159

160 Vision-Based 5 Advantages Ideal data source for analysis by image-processing software Object recognition by reading tokens on objects Potentially unlimited number of touch points Disadvantages Projection All the usual disadvantages of projection LCD in-cell light-sensing High sensitivity to ambient IR (in SUR40 implementation) Applications Interactive video walls ; digital signage; high-end retail University research (low-cost, easy to build) 160 Source: NORTD

161 Embedded Touch Technologies LCD Architecture Refresher Terminology Review Hybrid In-Cell Mutual Capacitive On-Cell P-Cap In-Cell Light-Sensing In-Cell Pressed Capacitive In-Cell Self-Capacitive In-Cell Voltage-Sensing Embedded-Touch Issues 161

162 LCD Architecture Refresher Top polarizer Color filter glass Color filter Transparent electrodes (Non-IPS VCOM) Alignment layer Liquid crystal Alignment layer LCD cell TFT array & transparent electrodes (IPS VCOM) TFT glass Bottom polarizer Brightness enhancement film Light guide Backlight Source: The Author 162

163 IPS vs. Other LCD Architectures Source: Presentation Technology Reviews 163

164 Embedded Touch Key defining characteristic Touch capability is provided by a display manufacturer instead of a touch-module manufacturer Touch-module manufacturers can t do in-cell or on-cell Marketing Terminology Alert! Some display manufacturers call all their embedded touch in-cell, even though they may be supplying hybrid on on-cell Some display manufacturers use a brand name to encompass all their embedded touch products For example, Touch On Display from Innolux Some display manufacturers direct-bond or air-bond an external touchscreen to their display and call it out-cell 164

165 Three Different Physical Integration Methods Used In Embedded Touch Term Integration Method Fab Method In-Cell Touch sensor is physically inside the LCD cell Touch sensor can be: Light-sensing elements Micro-switches (voltage-sensing) Capacitive electrodes (charge-sensing) Addition to TFT process On-Cell Hybrid (On-Cell/ In-Cell) Touch sensor is an array of ITO electrodes on the top surface of the color filter glass Same as standard p-cap Touch sensor is an array of ITO electrodes with one electrode on top of the color filter glass and the other electrode inside the cell Segmented VCOM electrode on the TFT glass (IPS LCD) Segmented VCOM electrode on the underside of the color filter glass (non-ips) Addition to color filter process Addition to both TFT and color filter process 165

166 Three Different Touch-Technologies Used In Embedded Touch Capacitive-sensing (four types) Standard p-cap (addition of two on-cell mutual-capacitive electrodes) Hybrid in-cell mutual-capacitive (addition of one in-cell and one on-cell electrode; locations depend on architecture) Pressed capacitive (addition of two in-cell mutual-capacitive electrodes per sensing element)(obsolete) Self-capacitive (addition of one in-cell self-capacitive electrode per sensing element)(obsolete) Light-sensing or optical (not fully successful) Addition of an in-cell photo-sensing element into some or all pixels Voltage-sensing or switch-sensing (obsolete) Addition of in-cell micro-switches for X & Y into some or all pixels 166

167 Hybrid In-Cell Mutual Capacitive for IPS LCDs 1 Source: The Author Principle Electrodes arranged to provide true mutual capacitance sensing in an IPS LCD while providing a high signal-to-noise ratio (> 50 db) Existing ITO static-shield on top of color filter glass (under the polarizer) is segmented into sense electrodes VCOM electrodes on TFT array are re-grouped into drive electrodes Requires cooperation between touch controller & LCD driver for timing First developed by JDI (Sony) & Synaptics 167

168 Hybrid In-Cell Mutual Capacitive for IPS LCDs 2 Source: BOE 168

169 Hybrid In-Cell Mutual Capacitive for TN/VA LCDs Principle Source: The Author Electrodes arranged to provide true mutual capacitance sensing in a non-ips LCD while providing a high signal-to-noise ratio New on-cell ITO sense electrodes are deposited on top of the color filter glass (under the polarizer) VCOM electrodes on the inner side of the color filter array are re-grouped into drive electrodes Requires cooperation between touch controller & LCD driver for timing 169

170 First Phones Shipped with Hybrid In-Cell Mutual-Capacitive (2012) Sony Xperia P and HTC EVO Design 4G (not the iphone 5) Similar LCDs 4-inch 960x540 LTPS (275 ppi) with different pixel arrays Same touch solution Synaptics ClearPad 3250 (four touches) <100 µm thinner than one-glass solution! Source: Sony Source: HTC 170

171 Apple iphone 5 In-Cell Structure Both sense and drive electrodes are in the TFT array, created by switching existing traces so they become multi-functional Required adding one layer (12-mask LTPS 13 masks) They re the only one using this structure; will Apple s patent block others? Apple s yield problems are well-known Apple has said they may change to Innolux Touch On Display (Innolux s brand name for ALL of their embedded touch structures) in iphone 6 Probably will be on-cell; maybe hybrid in-cell Source: CNET 171

172 On-Cell P-Cap Principle Source: The Author ITO P-cap electrode array is deposited on top of the color filter glass (under the top polarizer) Exactly the same function as discrete (standalone) p-cap Shown above is one ITO layer with bridges; it could also be two layers with a dielectric instead 172

173 First AM-OLED with On-Cell P-Cap Touch Samsung S8500 Wave mobile phone with Super OLED on-cell p-cap touch (2/10) 3.3-inch 800x480 (283 ppi) AM-OLED Super OLED is Samsung s (weak) branding for on-cell touch Sunlight readable AR coating & no touchscreen overlay Window = direct-bond cover-glass Source: Samsung booth graphic at Mobile World Congress 2010 Source: Samsung 173

174 Two Other Possible In-Cell P-Cap Configurations Patents, but maybe no products yet Apple, Samsung, Sharp Samsung, Apple Source: BOE 174

175 In-Cell Light-Sensing (Limited Commercial Success) Source: DisplaySearch Principle Source: Samsung Photo-sensor added in each pixel (rare) or group of pixels (4 to 16) IR sensor (asi or asige) added to TFT array IR emitters added to backlight Does not depend on ambient light (as in original design from 2003) Works with finger or light-pen; can work as a scanner Adding a cover-glass to protect the surface of the LCD reduces touch sensitivity because the finger is further away from the sensors 175

176 First Product with In-Cell Light- Sensing Touch (Unsuccessful) Sharp s PC-NJ70A netbook (5/09) Optical in-cell touch in 4 CG-silicon 854x480 touchpad LCD (245 dpi) 1 sensor per 9 pixels LED backlight Stylus & 2-finger multi-touch Scanning (object recognition) Japan-only; $815 Problems Required adding IR emitters into backlight S L O W (25% of typical touchpad speed) Short battery life Source: Sharp 176

177 Latest Product with In-Cell Light- Sensing Touch (Limited Success) Samsung SUR-40 asige sensor is 15X more sensitive than asi, but that means the touch-screen is 15X more sensitive to ambient IR Maximum Surface-2 lighting for acceptable performance Lighting Type Max Lux Compact Fluorescent 600 Cool White LED 560 Vapor Lamps 530 Sunlight (filtered 400 through window) Example Output Metal Halide 370 Warm White LED 300 Sunlight (direct) 160 Halogen 60 Incandescent 50 Environmental Lighting Optimizer Output 177

178 Unique Product with In-Cell Light-Sensing Touch (Successful) Integrated Digital Technologies light-pen monitor 21.5 in-cell light-sensing monitor with IR light-pen Supports two-touch with two pens Backplane by Taiwan CPE Source: IDTI Source: Photo by author 178

179 In-Cell Pressed Capacitive (Limited Commercial Success) Also called Charge Sensing Principle Source: LG Display Pressing the LCD changes the dielectric constant of the liquid crystal, which changes the capacitance between the conductive column spacer (CS) and the flat electrode in the TFT array. Electrode pairs can be in one pixel or in a group of pixels. Works with any touch object within damage limits of top polarizer Human body capacitance and dimensional change between electrodes are NOT relevant factors Requires deflecting the LCD surface (cannot add a cover glass) 179

180 First Product with In-Cell Pressed-Capacitive Touch 1 Samsung ST10 camera with 3 480x320 transflective TFT with in-cell pressed-capacitive touch (4/09) First use of any in-cell touch in a commercial product Works with finger or stylus, but with visible pooling Surface hardness = low Touch-screen includes electrostatic haptic feedback Camera includes MP3, PMP & text-viewer functions One sensor per 8 pixels (60x40 sensing matrix) Source: Samsung 180

181 First Product with In-Cell Pressed-Capacitive Touch 2 Excerpt from Samsung ST-700 digital camera manual 181

182 In-Cell Self-Capacitive Touch (Unsuccessful except maybe LGD?) Source: Drawing = Samsung & Author; Information = Toshiba Mobile Display Principle A single electrode per sensing element in the TFT array is connected to a reference capacitor. When a finger touches the LCD, the voltage at the electrode changes due to the capacitive coupling of the user s body-capacitance to ground. Works only with finger; no pressure is required Adding a cover glass reduces touch sensitivity; reduction in SNR can make touch non-functional in noisy environments 182

183 In-Cell Voltage-Sensing, also Called Digital Switching (Unsuccessful) Principle Source: Samsung Pressing LCD surface closes X & Y micro-switches in each pixel or group of pixels Requires deflecting the LCD surface (cannot add a cover glass) Works with any touch object within damage limits of top polarizer 183

184 Embedded-Touch Issues Will fully in-cell mutual capacitive (both electrodes in the TFT array) ever happen? It s already happened in the iphone-5, but nobody else has done it Can the size limit of today s hybrid mutual-capacitive be expanded beyond 12? Probably. The problem is sensing a larger number of electrodes faster. Metal mesh can help. Is light-sensing in-cell touch ever going to be fully successful? Probably not. It s been 10 years and the problems aren t solved yet Which is ultimately going to win, embedded or discrete touch? Embedded for high-volume, discrete for everything else 184

185 Embedded Touch-Technology Simple Summary Hybrid in-cell is clearly today s winner iphone-5 implemented full in-cell, but won t repeat it On-cell may not be a long-term winner Light-sensing in-cell still isn t fully successful, and it may never be All older forms of embedded touch (pressed capacitive, self-capacitive, and voltage-sensing) should now be considered DEAD 185

186 Other Touch Technologies Force-Sensing 186

187 Force Sensing Source: Vissumo File Download: 187

188 Force Sensing 1 Principle Suspend the touch-screen from force-sensors (strain gauges or piezos) such that movement is constrained to only the z-axis Variations IBM TouchSelect : Strain gauges (early 1990s, unsuccessful) Vissumo: Beam-mounted sensors (ran out of money in 2009) F-Origin: Spring-arm mounted sensors (recovered well after shrinking to just one person) FloatingTouch: Flexible adhesive pad sensors (start-up) NextInput: Piezo-capacitor sensor array (start-up) Force sensor (4) Slot (4) Frame Touch area (Vissumo s design) 188

189 Force Sensing 2 4 strain gauges supporting one touch panel Irregularly shaped, raised, textured, wooden touch surface Motor attached to and penetrating touch panel with printed speed control keys and push-pull control lever Vissumo s Amazing Demo Box Glass-covered LCD integrated into touch panel with soft keys printed on back of glass Raised, marble touch surface with toggle switches penetrating touch panel Multi-page book with touchable & movable metal pages Snap-dome keys attached to touch panel; removable padded and textured keys; speaker attached with holes through the touch panel. Source: Photo by author 189

190 Force Sensing 3 F-Origin s spring-arm suspension F-Origin s system block diagram Source: F-Origin 190

191 Force Sensing 4 Advantages Touch-object independence (touch with anything) Touch-surface independence (any rigid material) Can use bezel or zero-bezel No other touch technology can handle 3D substrates with embedded moving objects Adjustable sensitivity (press lightly to highlight, harder to select) Minimizes false touches Continuous calibration filters out environmental conditions Disadvantages Limited multi-touch (2-touch = 8 sensors) Mechanical nature reduces reliability Most sensors add volume Applications Source: Vissumo Mostly commercial, although NextInput is aiming at consumer 191

192 Active Stylus Technologies Electromagnetic Resonance (EMR) P-Cap Stylus Other Active Stylus Technologies 192

193 Stylus 1 Tablet PCs, PDAs, and early smartphones (e.g., Trio) have always had styli (1989 to 2007), so why are we so finger-focused now? Steve Jobs and the iphone in 2007 Who needs a stylus? Microsoft s failure to make the stylus-based Tablet PC a success with consumers caused them to de-emphasize the stylus and focus on finger-touch in Windows 7 193

194 Stylus 2 Is the stylus coming back into the consumer space? YES! All the major p-cap controller suppliers support active & passive PC OEMs want to differentiate their products from Apple s Legacy Windows software on a Win8 tablet needs a stylus Android (in Ice Cream Sandwich) supports stylus messages Samsung has shipped >15M Galaxy Notes in two sizes Consumption isn t enough; a stylus is great for creation Source: Atmel 194

195 Stylus Use-Cases In Windows Taking notes, typically with MS OneNote Notes are automatically converted into text in background; being able to search your ink notes is very powerful Annotating documents Typically Office or PDF Quick sketches Typical whiteboard-type sketches Artistic drawings It s unbelievable what a real artist can do Precision pointing device, e.g. with Windows 8 Desktop When you re trying to select tiny UI elements 195

196 Electromagnetic Resonance (EMR) Stylus Source: Wacom File Download: 196

197 EMR Stylus 1 Cordless pen without battery L Source: Wacom CTip CMain CSide Side switch Pressure-sensitive capacitor (CTip) Coil (L) Transmitted RF Pen equivalent circuit Received RF LCD Sensor grid Sensor grid schematic (10µ copper) Many wires Controller chipset 5-8 wires Serial/USB interface to host Source: Wacom 197

198 EMR Stylus 2 Variations Sensor substrate (rigid FR4 vs. flexible mm PET) Pen diameter (3.5 mm PDA pen to 14 mm executive pen) Size range 2 to 14 Controllers Proprietary Advantages Very high resolution (1,000 dpi) 2 14 Controller for 10.4 Source: Wacom Pen hover (mouseover = move cursor without clicking) Sensor is behind LCD = high durability & no optical degradation Batteryless, pressure-sensitive pen 198 Single controller can run both pen digitizer & p-cap finger touch

199 EMR Stylus 3 Disadvantages Electronic pen = disables product if lost; relatively expensive Difficult integration requires lots of shielding in mobile computer Sensor can t be integrated with some LCDs Single-source for mobile CE devices (Wacom) = relatively high cost Applications Phablets and tablets E-book readers Opaque desktop graphics tablets Integrated tablet (pen) monitors Suppliers Wacom, Hanvon, Waltop, UC-Logic/Sunrex, KYE Wacom Bamboo Tablet 199

200 EMR Stylus 4 Samsung Galaxy Note sketching demo at CES 2012 The Galaxy Notes use both a p-cap touchscreen AND a Wacom EMR stylus (2 sensors!) Source: Photos by Author 200

201 P-Cap Stylus Source: N-Trig File Download: 201

202 P-Cap Stylus (N-Trig) KHz Source: N-Trig 202

203 P-Cap Stylus 2 Variations One-way digital RF transmission from stylus to p-cap sensor, with both sense & drive electrodes acting as antennas N-Trig has by far the most-developed user experience Two-way transmission between stylus and p-cap sensor Stylus receives p-cap sensor drive-signal, amplifies it, adds digitally encoded stylus information, and transmits it back to sensor Stylus generates intense e-field at tip E-field adds capacitance to p-cap sensor operating as usual (finger subtracts capacitance) 203

204 P-Cap Stylus 3 Advantages Uses existing (single) p-cap sensor Pen hover (mouseover = move cursor without clicking) Stylus tip can be very small (< 1 mm) High resolution and accuracy Disadvantages Stylus requires power source (battery or super-capacitor) Stylus technology is unique to each p-cap controller supplier 204

205 P-Cap Stylus 4 Synaptics 7300 active stylus demo at Computex 2012 Note the 1 mm tip! And it hovers in Windows! Synaptic s passive pen-solution offers pen functionality suitable for Android OS (i.e., no hover, and no ink as a data type ) Source: Synaptics 205

206 Other Active-Stylus Technologies Combination ultrasonic & infrared Used in many clip-on and clipboard-style digital note-taking accessories; also available for ipad Embedded CMOS-camera stylus by Anoto Widely licensed for digital-pen note-taking accessories and form-filling applications Used by LG Displays in large-format touch Used in Panasonic 4K 20 professional tablet shown at CES 2013 Infrared LED light-pen Used by idti in their light-sensing in-cell touch Visible laser-pointer Used by isiqiri in large-format touch Also works with idti light-sensing in-cell touch 206

207 Software Multi-Touch Operating-System Application-Development Support Device Interfaces Middleware 207

208 Multi-Touch Sources: Engadget, Do Device and Good Times & Happy Days File Download: 208

209 Multi-Touch Multi-touch is defined as the ability to recognize two or more simultaneous touch points Multi-touch was invented in 1982 at the University of Toronto (not by Apple in 2007!) Pinching gestures were first defined in 1983 (not by Apple in 2007!) Windows 7 (2009) & Windows 8 (2012) both support multi-touch throughout the OS and are architected to support an unlimited number (~100) of simultaneous touch points 209

210 Multi-Touch Architecture Application Capable of decoding multiple streams of moving points and taking actions in response Operating System Capable of delivering multiple streams of moving points (and acting on a defined subset of them) Touchscreen Controller & Driver Capable of delivering sets of simultaneous points to the OS Touchscreen Sensor Capable of sensing multiple simultaneous points 210

211 Multi-Touch Technologies Touch Technology Multi-Touch Capable? (#) Win-7 Logo Capable? Win-8 Logo Capable? Commercial MT Product Example P-Cap Yes (unlimited*) Yes Yes Apple ipad Multi-Touch Infrared Yes (20-40) Yes Yes General Touch PIT Planar Scatter Detection Yes (20-40) Yes Yes FlatFrog MT-3200 Camera-Based Optical Yes (5) Yes Yes NextWindow DT-6S Embedded (Hybrid & On-Cell) Yes (unlimited*) Yes Yes Sony Xperia P Embedded (Light-Sensing In-Cell) Yes (unlimited*) Yes Maybe Samsung SUR40 Vision-Based (Projection-Based) Yes (unlimited*) Yes Maybe Microsoft Surface v1 Digital Multi-Touch Resistive (Stantum) Yes (10) Yes Maybe None Surface Acoustic Wave (Elo & GTT) Yes (2) Yes No Lenovo AiO PC Traditional Infrared Yes (2) Yes No Nexio 42 Monitor Waveguide Infrared (RPO) Yes (2) Yes No None Analog Multi-Touch Resistive (AMR) Yes (10) No No Gateway AiO PC Acoustic Pulse Recognition (Elo) Maybe (2) No No -- Dispersive Signal Technology (3M) No No No -- Analog Resistive No No No -- Surface Capacitive No No No -- Force-Sensing Maybe (2) No No -- * Controller-dependent, not sensor-dependent 211

212 Multi-Touch Gestures On Non-Multi-Touch Screens Gesture-enhanced single-touch technologies Capability of sensing two-finger gestures on single-touch analog-resistive and surface-capacitive touchscreens Restrictions depend on implementation Some require that fingers be moving (2 static touches = 1 touch) It can never pass any Windows Touch Logo Why it exists: Marketing! Gestures are HOT, so device manufacturers want them Today, multi-touch is mostly used to enable two-finger gestures For mobile devices, p-cap is 2X-3X the cost of analog resistive, so enabling two-finger gestures on analog resistive is attractive The result Poor user experience, due to the difficulty of keeping two fingers pressed hard enough against the screen 212

213 Why Multi-Touch Has Become So Important 1 Apple Apple established multi-touch as a must-have for coolness. The result is that people of all ages expect every display they see to be touchable with multiple fingers Gaming Gaming is a natural for multi-touch. Try playing air hockey without multi-touch Unintended touches One of the major values of multi-touch is to allow the system to ignore unintended touches ( palm rejection, grip suppression, etc.). As desktop screens become more horizontal (recline) this will become even more important. 213

214 Why Multi-Touch Has Become So Important 2 Multi-user collaboration When two people want to collaborate on a large screen (e.g., a student and teacher on an interactive whiteboard LCD), multi-touch is essential Identifying which touch belongs to which user is still unsolved It IS currently possible to uniquely identify multiple simultaneous styli 214

215 How Many Touches Are Enough?...1 The industry has multiple answers Microsoft settled for 5 touches in Win8 (they wanted 10) The p-cap touchscreen suppliers under 30 either say 10 or as many as possible (e.g., 3M s p-cap supports 60+ touches) The large-format touchscreen suppliers say that 40 is enough In practice it depends on the hardware and controller firmware implementation Ideally the touchscreen should ignore all other touches beyond however many the product is guaranteeing This is usually called palm rejection and its implementation is absolutely critical to the user experience 215

216 How Many Touches Are Enough?...2 The answer actually depends on the application For a small mobile device, 2-5 (one hand) are enough For a single-user app on any device (even an 82 screen), it s hard to see why more than 10 (two hands) are needed For a multi-user app, it depends For a 55-inch gaming table, 40 (8 hands) is not unreasonable The key touchscreen specification is probably response time (latency) For a 65-inch interactive whiteboard LCD, 20 (4 hands) is probably enough, although an argument can be made for 40 BUT, the key touchscreen specifications are entirely different: minimum stylus tip size, pre-touch, jitter, ink-lag, etc., can all be critical From a video of a very cool multi-player game on the FlatFrog website 216

217 #1 Reference On Multi-Touch Multi-Touch Systems that I Have Known and Loved If you can only manipulate one point you are restricted to the gestural vocabulary of a fruit fly. We were given multiple limbs for a reason. It is nice to be able to take advantage of them. Bill Buxton, 2008 Principal Researcher, Microsoft Research 217

218 Operating Systems Source: Microsoft, Google, Apple File Download: 218

219 For Windows, the Logo Is the Starting Point A set of touch performance standards designed to ensure a high-quality user experience 5 touch-point minimum Touchscreen jitter Extra input behavior High-resolution timestamp Input separation Noise suppression Physical input position Reporting rate Response latency Cold boot latency Touch resolution User experience Pre-touch Pen tests 219

220 Windows 8 Touch The Win8 Touch Logo specification is based on p-cap Win7 spec was based on optical, which had little relevance Win8 spec creates a common touch capability for mobile phones, tablets, notebooks, and desktops This may be very significant for multi-platform applications! Basic spec requirements Minimum of 5 simultaneous touches; must ignore an additional 5 Tablets must be zero-bezel; otherwise 20 mm border minimum Respond to first touch in < 25 ms Subsequent touches must be < 15 ms at 100 Hz for all touches Better than 0.5 mm accuracy with < 2 mm offset from actual location No jitter when stationary; < 1 mm when moving 10 mm Pre-touch < 0.5 mm Finger separation >= 12 mm horizontal/vertical, 15 mm diagonal But on-screen keyboards and normal human behavior violates this! 220

221 Windows Touch Application Development There are multiple development environments commonly used in Windows 7 & 8, each of which handles touch differently Native C++ (Win32/COM) Managed environment (.NET Framework) Silverlight & WPF (Windows Presentation Foundation) Adobe Flash Modern (Win-8) using C# and XAML or HTML5 and JavaScript Modern apps today only represent one aspect of business computing: reporting/dashboards, with moderate-to-light data updating From my perspective As a hardware person, I find the level of detail required to do anything significant in touch software to be excruciating 221

222 Android Touch Application Development Android has an extensive and growing API for touch & stylus I hear complaints about the degree of bugginess From what I can tell, the level of tediousness is a little better than Windows The Android API supports up to 256 touches, but the actual number depends on the hardware & firmware implementation in the device 2 to 5 isn t unusual Fragmentation of Android (different versions from each OEM) appears to make developing a robust run-on-anything Android touch application very difficult The language decision is easy it s Java or nothing 222

223 ios Touch Application Development ios seems to have the most constrained touch application development environment But it s not any easier than Android -- in the chapter on touch in Programming ios 5 (an O Reilly book), the words messy and tricky seem to occur a lot The language decision is easy it s Objective-C or nothing 223

224 Interfaces 1 In Windows, touch devices must be HID-compliant (USB Human Interface Device) This makes interfacing simple, at least most of the time Win 8 treats mouse, finger and stylus as co-existing equals Outside of Windows, one of the most common interface protocols is TUIO Source: TUIO.com 224

225 Interfaces 2 Linux includes multiple touch interfaces X Window System (single & multi-touch), TSlib, evtouch, evdev, TUIO, and others A Source Pointer Device Driver is available from Touch-Base that unifies all these interfaces Some touch devices have their own protocol e.g., Zytronic large-format p-cap A Universal Pointer Device Driver is available from Touch-Base that is available for many operating systems and supports nearly all known touch controllers ever manufactured 225

226 Middleware Source: Newsthinking File Download: 226

227 Middleware 1 From the last five slides, it should be pretty obvious why a simplifying/unifying layer (touch middleware ) would be very valuable The bad news is that there is no really general-purpose touch middleware on the market yet The best (in the author s opinion) is Snowflake Good starting point for commercial applications Includes 30+ multi-touch apps (entertainment, presentation, creativity, media-browsing, etc.) Includes an SDK Runs on Win 8/7/Vista/XP, Mac OS X Lion & Snow Leopard, and Linux Ubuntu Snowflake simplifies handling touch & gesture events, audio, video, images, PDFs, 3D, on-screen keyboards, multiple languages, QuickTime integration, web browsing, etc. 227

228 Middleware 2 Snowflake home screen Source: NuiTeq 228

229 Middleware 3 Other alternative middleware Omnitapps Less complete, Windows only, no SDK, more for product marketing Intuilab Commercial multi-touch application platform with Kinect, RFID, etc. GestureWorks (Ideum) Robust Flash multi-touch development environment 22 Miles Sales productivity application for ios, Android, Windows & Mac Sotouch Application platform for wayfinding and presentations Fingertapps (Unlimited Realities) Multi-touch demo software 229

230 Conclusions Touch Technology versus Application Usability, Performance, & Integration Characteristics Touch Technology Primary Advantages and Flaws Prediction of the Future Suggested Reading on Touch Recommended Conferences & Trade Shows on Touch 230

231 Touch Technology vs. Application Application Example Analog Resistive Multi-Touch Resistive Surface Capacitive Projected Capacitive Touch Technologies Kiosk Point of Info (POI) Museum information O X O X O O X O O O X X X X X Kiosk Commerce Digital photo printing O X O O O X X X O O X X X X X Kiosk Ruggedized Gas pump X X O O O O X X X X O X X X X Point of Sale (POS) Restaurant; lottery O X O O O O X X O X O X X X X Office Automation Office monitor O X O X O X X X X X X X X X X Industrial Control Machine control O O O X O O X X X X O X X X X Medical Equipment Medical devices O X X O O X X X O X X X X X X Healthcare Patient info monitor O X X X O X X X O X X X X X X Military Fixed & Mobile Submarine console O X O X X O X X X X X X X X X Training & Conference Boardroom display O X X X O O X O X O X X X X X Legal Gaming Casino machine X X O X X X X X X X X X X X X Amusement Gaming Bar-top game X X O X O X X X O X X X X X X In-Vehicle GPS navigation O X X O X X O X X X X X X X X ATM Machine ATM machine X X O O O O X X X X X X X X X Mobile Device Smartphone O O X O X X O X O X O O O O O Appliance Refrigerator door O X X O X X X X O X X X X X X Architectural Elevator control X O X X X X X X X X O X X X X Consumer AiO & Monitor HP TouchSmart O X X X O X X O X X X X X X X Music Controller Jazz Mutant O O X O X X X X X X X X X X X Digital Signage Thru-window store X X X O O O X O O O X X X X X SAW Traditional IR Waveguide IR Optical APR DST Force Sensing LCD In-Cell (Light) LCD In-Cell (Voltage) LCD In-Cell (Charge) LCD On-Cell (Charge) 231

232 13 Usability Characteristics Touch Technologies Desirable Characteristic Analog Resistive Multi-Touch Resistive Surface Capacitive Projected Capacitive SAW Traditional IR Waveguide IR Optical APR DST Force Sensing LCD In-Cell (Light) LCD In-Cell (Voltage) LCD In-Cell (Charge) LCD On-Cell (Charge) Usability Touch with any object H H L L M H H H H H H M M M L No unintended touch H H H H H L L L H H H H H H H Multi-touch L H L H M M M M L L L H H H H Touch & hold H H H H H H H H L L H H H H H High durability L L M H H H H H H H H M L L H High sensitivity (light touch) M M H H M H H H M H L H H H H Fast response & drag M M H H M M H H M H L L H M M Stable calibration M H L H H H H H H H H H H H H Very smooth surface L L H M M M M M M M M M L L M No liquid crystal pooling H H H H H H H H H H H H L L H Resistant to contaminants H H M H L M L M H H H L L L H Works in rain, snow & ice H H L H L L L L L L H L L L H Works with scratches L L M H H H H H M H H L L L H 232

233 13 Performance Characteristics Desirable Characteristic Analog Resistive Multi-Touch Resistive Surface Capacitive Projected Capacitive Touch Technologies SAW Performance High optical performance L L M M H H H H H H H H H H M High resolution H M H H M L H H M M L M H L H High linearity H H M M M M H M M M H H H H M High accuracy & repeatability H M M H H M H M M M H H H H H Low power consumption H H L M L L M M H L H H L M M Insensitive to vibration H H H H H H H H H M L H H H H Insensitive to EMI & RFI H H L L H H H H H H H L L L M Insensitive to ambient light H H H H H M H M H H H L H H H Insensitive to UV light L L H H H H H H H H H H M M H Touch-object size recognition L M L H L L H H L L L M H M H Measures Z-axis L L L M M L L L L L H L L L M Handwriting recognition H M L M L L M H L L L M H L M Works with bi-stable reflective H H L H L L M L H L L M L L H Traditional IR Waveguide IR Optical APR DST Force Sensing LCD In-Cell (Light) LCD In-Cell (Voltage) LCD In-Cell (Charge) LCD On-Cell (Charge) 233

234 13 Integration Characteristics Touch Technologies Desirable Characteristic Analog Resistive Multi-Touch Resistive Surface Capacitive Projected Capacitive SAW Traditional IR Waveguide IR Optical APR DST Force Sensing LCD In-Cell (Light) LCD In-Cell (Voltage) LCD In-Cell (Charge) LCD On-Cell (Charge) Integration Substrate independence M M L H L H H H L L H L L L L Scalable M L M H M M L H H H H L L L L Easy integration H M L L M M M H L L M H H H H Flush surface (low profile) M M M H M L M L H H M H M M H Narrow border width H M M H L L M L H H M H H H H Thin and light H H L H L L M L L L L H H H H Easy to seal H H H H L M M L H H M M L L M Can be vandal-proofed L L M H H M M L H H H L L L L Works on curved surface M M L H L L L L L L H H L L H Can be laminated to LCD H H H H M M H H L L L H H H H HID (Plug & Play) interface L L L L L L L H L H L L L L L Simple controller H M L L L L M M M L H L H M M Controller chip available H H L H H L H L H L H L L L L 234

235 There Is No Perfect Touch Technology! Touch Technology Major Advantage Major Flaw Projected Capacitive Multi-touch High cost Surface Capacitive Touch sensitivity High drift Analog Resistive Low cost Low durability Analog Multi-Touch Resistive Multi-touch Resolution Digital Multi-Touch Resistive High resolution Low durability Surface Acoustic Wave Durability Soft touch object Acoustic Pulse Recognition Any touch-object No touch & hold Dispersive Signal Technology Any touch-object No touch & hold Traditional Infrared Reliability High cost Waveguide Infrared Low cost Contamination Multi-Touch Infrared Multi-touch Performance Camera-Based Optical Scalability Profile height Planar Scatter Detection Flush surface High cost Vision-Based Multi-touch Rear projection Embedded (Hybrid & On-Cell) Integration Volume required Embedded (Light-Sensing) Integration Ambient IR Force-Sensing 3D substrate Multi-touch 235

236 A Prediction of Which Technologies Will Win in the Next Five Years 1 Application Winning Technology Runner-Up Technology Automotive Projected Capacitive Analog Resistive Casino Gaming Projected Capacitive Surface Capacitive Consumer AiOs and Monitors Projected Capacitive Camera-Based Optical Consumer Games Analog Resistive Projected Capacitive Consumer Tablets Projected Capacitive Embedded & Notebooks Interactive Digital Signage Camera-Based Optical Traditional & Multi- Touch Infrared e-readers Traditional Infrared EMR Stylus Industrial Terminals Analog Resistive Projected Capacitive Kiosks Surface Acoustic Wave Projected Capacitive Mobile Phones Embedded Analog Resistive POS Terminals Analog Resistive Projected Capacitive Source: Author (3/13) 236

237 A Prediction of Which Technologies Will Win in the Next Five Years 2 # Touch Technology 5-Year Prediction 1A Projected Capacitive Dominant 1B Surface Capacitive Significant Reduction 2A Analog Resistive Major Reduction 2B Analog Multi-Touch Resistive (AMR) Disappear 2C Digital Multi-Touch Resistive Small Niche 3A Surface Acoustic Wave (SAW) Moderate Growth 3B Acoustic Pulse Recognition (APR) Small Niche 3C Dispersive Signal Technology (DST) Disappear 4A Traditional Infrared Reduced Large-Format; Increased Small-Medium 4B Waveguide Infrared (DWT) Already Gone 4C Multi-Touch Infrared Moderate Growth 4D Camera-Based Optical Increased Large-Format; Decreased Desktop 4E Planar Scatter Detection (PSD) Viable Niche 4F Vision-Based Viable Niche 5 Embedded Significant Growth 6 Force-Sensing Disappear Source: Author (3/13) 237

238 Suggested Reading on Touch 1 September 2012 March 2011 March 2010 Even the oldest issue still contains useful information (e.g., on surface capacitive) December 2007 December

239 Suggested Reading on Touch 2 At $49/year for 10 issues, this is an excellent value as a source of touch news and touch-conference reports 239

240 Suggested Reading on Touch 3 Use Google Alerts to track your favorite touch keywords 240

241 Suggested Conferences and Shows on Touch & Interactivity 1 SID s Display Week (Vancouver, Canada in 2013) Exhibits, Symposium Touch Papers on Thursday, Sunday Short Course, Monday Technology Seminars, Tuesday Exhibitors Forum, Wednesday Touch-Gesture-Motion Conference SID s International Display Workshop (ITW - Japan) Computex (Taipei - consumer products) InfoComm (Large-format products) DisplaySearch Emerging Display Technologies (USA) FPD International (Japan) China Touchscreen (Shenzhen, China; not Shanghai) ACM s Interactive Tabletops & Surfaces (ITS, USA) ACM s SIGGRAPH (USA) 241

242 Suggested Conferences and Shows on Touch & Interactivity 2 Shows with commercial touch applications National Retail Federation (NRF) Healthcare Information Management Systems Society (HIMSS) Global Gaming Expo (G2E-USA & G2E-Asia) Digital Signage Expo (DSE) Customer Engagement Technology World (CETW) (Formerly KioskCom ) Integrated Systems Europe (ISE) 242

243 Thank You! File Download: Intel Corporation mobile 2200 Mission College Blvd office Santa Clara, CA fax

244 Legal Disclaimer All products, computer systems, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice. Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, visit Intel Performance Benchmark Limitations Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Results have been simulated and are provided for informational purposes only. Results were derived using simulations run on an architecture simulator or model. Any difference in system hardware or software design or configuration may affect actual performance Intel does not control or audit the design or implementation of third party benchmarks or Web sites referenced in this document. Intel encourages all of its customers to visit the referenced Web sites or others where similar performance benchmarks are reported and confirm whether the referenced benchmarks are accurate and reflect performance of systems available for purchase. Intel processor numbers are not a measure of performance. Processor numbers differentiate features within each processor family, not across different processor families. See details. Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request. Hyper-Threading Technology requires a computer system with a processor supporting HT Technology and an HT Technology-enabled chipset, BIOS and operating system. Performance will vary depending on the specific hardware and software you use. For more information including details on which processors support HT Technology, see Intel Virtualization Technology requires a computer system with a processor, chipset, BIOS, virtual machine monitor (VMM) and applications enabled for virtualization technology. Functionality, performance or other virtualization technology benefits will vary depending on hardware and software configurations. Virtualization technology-enabled BIOS and VMM applications are currently in development. Intel Turbo Boost Technology requires a PC with a processor with Intel Turbo Boost Technology capability. Intel Turbo Boost Technology performance varies depending on hardware, software and overall system configuration. Check with your PC manufacturer on whether your system delivers Intel Turbo Boost Technology. For more information, see 64-bit computing on Intel architecture requires a computer system with a processor, chipset, BIOS, operating system, device drivers and applications enabled for Intel 64 architecture. Performance will vary depending on your hardware and software configurations. Consult with your system vendor for more information. Lead-free: 45nm product is manufactured on a lead-free process. Lead is below 1000 PPM per EU RoHS directive (2002/95/EC, Annex A). Some EU RoHS exemptions for lead may apply to other components used in the product package. Halogen-free: Applies only to halogenated flame retardants and PVC in components. Halogens are below 900 PPM bromine and 900 PPM chlorine. Intel, Intel Xeon, Intel Core microarchitecture, and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries Standard Performance Evaluation Corporation (SPEC) logo is reprinted with permission Roadmap not reflective of exact launch granularity and timing please refer to ILU guidance Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.

245