DISPLAY INSPECTION SYSTEM T. Babinec 1, P. Cip 1 1

Transcription

1 Ročník 2011 Číslo II DISPLAY INSPECTION SYSTEM T. Babnec 1, P. Cp 1 1 Department of Control and Instrumentaton, Faculty of Electrcal Engneerng and Communcaton, VUT, Brno, Kolejn 2906/4, Brno E-mal : xbabn01@stud.feec.vutbr.cz, xcppa00@stud.feec.vutbr.cz Abstract: Graphcal user nterfaces are often the best soluton for creatng an easy to use human machne nterface. Ther development has an teratve character and requres perodcal testng of the functonalty and consstency of the graphcal output. Ths paper deals wth the desgn of a sem-automated system for functonalty and qualty nspecton of graphcal user nterfaces n varous devces and screen types. The presented approach s based on mage processng algorthms, whch mnmzes the need for human nteracton durng the test procedures. INTRODUCTION One of the essental characterstcs of a modern human machne nterface (HMI) s the ablty to communcate wth user n some graphcal form. Ths s due to the mportance of vsual percepton among other human senses. It s a well know fact, that mages have the ablty to express large amount of nformaton n a very effcent and easy to understand way. As a consequence varous sorts of electronc devces (e.g. PDAs, cell phones, GPS, etc.) utlze one or more graphcal screens whch can be based on a broad varety of dfferent technologes. Several the most wdespread technologes are the black&whte fxed segment and dot matrx dsplays, whch are typcal for smple or low-end devces. Another mportant group ncludes full-colour LCD (lqud crystal dsplays), TFT (thn-flm transstor) and OLED (organc lght emttng dode) dsplays. An example of an unconventonal but rather nterestng screen technology s the E-INK (electronc nk), whch sgnfcance and popularty among users has been growng rapdly n the recent years. The development and mass producton of nowadays complex electronc devces requres extensve functonalty and qualty nspecton. A very mportant component of the nspecton s the HMI evaluaton, whch can be dvded to hardware (HW) and software (SW) orented approaches. An example of nspecton system orented on the detecton of faulty HW features s descrbed n [1]. Completely dfferent category of nspecton mechansm s requred by end-user devce manufacturers and graphcal user nterface (GUI) developers. Ther never endng competton for better lookng, more user frendly and ntellgent software envronments leads to extremely complex screen content. The content s usually compled from sophstcated menu structures and other graphcal objects, whch change dynamcally accordng to actual stuaton and user nput. Generally such GUI development requres repeated fne adjustments to the SW structure, whch may cause unexpected behavoral and random errors n the whole user nterface. Therefore a rather extensve check of GUI consstence and functonalty s requred after every sgnfcant change n the SW. The test procedures are monotonous and tme consumng. As long as they are performed by human operators, the test results can be affected by operator s fatgue or other nfluences that are dffcult to evaluate. In order to mnmze or completely replace human nteracton durng the tests, an automated nspecton system has to be desgned. In [2] a method for automatc GUI nspecton based on a source code analyss has been presented. Ths paper descrbes the desgn of a semautomatc nspecton system based on computer vson algorthms. The system enables analyss of already dsplayed graphcal nformaton on varous types of screen devces. SYSTEM DESIGN In order to satsfy the development goals, whch are presented n the next subsecton, a complete nspecton system ncludng HW and SW components had to be desgned. The development was focused at mtatng humanlke behavour durng the testng process. Therefore the system facltates vsual evaluaton together wth the possblty of on-lne human nput smulaton as a reacton to currently dsplayed nformaton. Requrements Maxmum separaton between the nspecton system and tested devces, whch mnmzes mutual nfluence and error propagaton. A smple way for rapd test procedure scrptng, provded by well organzed programmng nterface. An uncomplcated functonalty expanson. Mnmum need for human nteracton.

2 Hardware set-up As t s depcted n fgure 1, the HW set-up of the nspecton system can be assembled from ordnary components accordng to the specfc needs of the tested devce. Basc HW element s an Imagng Source ndustral camera connected to a PC. In order to be able to process colour nformaton wth a pxel resoluton suffcent for majorty of varous dsplay HMIs a 41BU02 camera model equpped wth 1280x960 RGB CCD sensor wth Bayer encodng capable of 15 frames per second was used. An adjustable stand ensures proper camera postonng, whch has to be adapted to avalable optcs and dmensons of the tested screen. For stablzng the lghtng condtons or passve screen (e.g. E-INK) nspecton an addtonal lght source may be requred. Image processng and potental devce control over some nput smulator module s accomplshed by the nspecton SW nstalled on the PC workstaton. scrptng wth full-featured programmng languages lke C# and therefore satsfes one of the development goals. Fg. 2: Inspecton software archtecture OPERATION & RESULTS In order to ensure better results and determnstc behavoural of the desgned nspecton system, a system calbraton s requred before ntatng the actual test procedure. Implemented calbraton procedure, nspecton functonalty, acheved results and proposed scheme for GUI content defnton are dscussed n detal n the followng subsectons. Fg. 1: Inspecton system hardware set-up Software envronment The mplemented nspecton SW has a modular structure (see fgure 2). For ntal system calbraton and experment set-up a Screen Inspector SetUp module was created. Its purpose and functonalty s descrbed more closely n the followng secton of ths paper. The communcaton wth camera HW, propertes management and mage grabbng s mplemented n the module UsbCam. To enable onlne and automatc control of the nspected devce a custom bult functonalty represented by Input Smulator module may be lnked to the system. The mage processng core of the software envronment s hdden nsde the three-level module. The mplemented computer vson functons are based on the programmng resources avalable from OpenCV lbrary [3]. Hgh performance computer vson algorthms are wrtten n C/C++ language and defne the LowLevel part of the mage processng core. Ths functonalty s wrapped nsde the HghLevel object herarchy, whch was created usng CLR C++ and s ntended for.net managed envronment. At the same tme HghLevel objects and methods represent a sort of a programmng nterface, whch allows fast and uncomplcated test procedure Fg. 3: Screen template (top) and ts mask (bottom) Dsplay descrpton The expected GUI content s descrbed usng a XML document, whch s an essental part of every test procedure. The XML structure s desgned to acheve the most flexble but at the same tme smple way for

3 defnton of herarchcal composton of graphcal objects. The objects tested lke screens, buttons, etc. are dvded nto classes wth specfc behavoral, whch s descrbed n the test scrpt. Each basc screen element has unque features descrbed as a collecton of deal btmap templates (e.g. fgure 3 top) combned wth detecton masks (e.g. fgure 3 bottom) and rectangle coordnates defnng areas ntended for optcal character recognton (OCR). The detecton mask s an 8bt grayscale mage wth the same resoluton as the correspondng template. The meanng of regon dfferentaton wll be explaned further n the text. Combnaton of these fundamental features and nformaton about screen element s membershp n some defned parental object (panel, menu, screen ) creates a coherent devce GUI descrpton. Ths approach also enables smple functonalty expanson. Snce dsplay descrpton s a rather specfc and devce dependant matter, the nspecton SW lbrary provdes only elementary methods, whch are supposed to be used n order to derve more complex functonalty for detecton and nspecton of specfc screen objects. Thanks to ths qualty, the end-user s able to ndependently boost the provded nspecton lbrary wth no or mnmal support from the nspecton system developers. System set-up and calbraton Indvdual steps of the system set-up and calbraton are llustrated n fgures 3 and 4. Overall t s a semautomated procedure. The requred operatons are as follows: For successful calbraton at least 4 ponts correspondences between deal screen content template and mage of the screen captured on the camera have to be found. Consequently the equaton (1), whch models the mappng between 2D source pont [X S, Y S, 1] (mmedate screen pont) and 2D destnaton pont [X D, Y D, 1] (pont n the camera frame magng the screen) n homogenous coordnates, can be solved for 8 unknown parameters h 11...h 32 of the projecton matrx. Parameter m descrbes the scale ambguty of the transformaton. Further nformaton on coordnates mappng and the use of homogenous coordnates can be found for example n [4] and [5]. X D h11 h12 h13 X S m YD h21 h22 h23 YS (1) 1 h h32 The calbraton ponts can be defned manually, but the SW envronment also enables completely automated approach. The user has only to choose the template mage for currently dsplayed GUI screen. The calbraton screen should contan hgh amount of spatally unque and unformly dstrbuted nterest ponts. A very sutable example s the 2D pattern of bnary nose presented n fgure 4, but n most cases a well structured menu screen (as the one from fgure 5) should also suffce. Hardware set-up» Adjustment of mechancal propertes: Inspected devce postonng; camera/dsplay measurng dstance modfcaton.» Adjustment of optcal propertes and lghtenng condtons: Mechancal daphragm set-up and objectve focusng, surroundng lghtenng adjustment. Software set-up» Image propertes: Shades of gray/color magng; frames per second rate; gan, whte balance and gamma correcton adjustment.» Camera calbraton: Automatc screen locaton and projectve transformaton dentfcaton. Fgure 4 depcts wndows nterface for Screen Inspector SetUp module. In order to smplfy the subsequent dsplay nspecton the calbraton module ncludes a screen plane to camera frame homography transformaton dentfcaton [4]. Result of ths operaton s shown n fgure 5 rght. Fg. 4: System setup and calbraton GUI, wth an example of sutable calbraton pattern The locatons of the calbraton ponts n both mages and ther mutual correspondences are found usng speed-up robust features (SURF). However, even the SURF method can generate many false correspondences and thus a robust ntal estmaton of the transformaton parameters utlsng RANSAC algorthm has been used. Afterwards a smplex method (orgnally presented n [7]) for functon mnmzaton s appled n order to fne-tune the resultng homography matrx by mnmzng the sum of absolute dfferences between the screen template mage and the transformed screen mage. The calbraton step brngs many advantages. Not only that t ensures normalzaton of the captured

4 screen mages, but t can also sgnfcantly reduce nspected area of the camera frames and therefore decrease computatonal tme. Snce the calbraton s requred only at the begnnng of the test procedure, ts hgher computatonal complexty s not a setback. Fg. 5: Projecton matrx estmaton and screen mage rectfcaton (left: detected screen area, rght: rectfed mage) Dsplay nspecton The developed three-level nspecton lbrary (see fgure 2) mplements 3 basc mage processng algorthms. These are: defnton of specal meanngs of the correspondng screen pont. For example settng the most sgnfcant bt to 1 s nterpreted as a pont wth no effect to the mage comparson computaton. Ths s mportant f there are contnuously changng areas present n the screen. Ther actual appearance cannot be predcted and therefore should not be evaluated as a statc mage. Other bts from the mask mage may have dfferent meanngs accordng to the demands of the tested devces. Ther combnaton creates a greyscale mage representaton of the mask. The second basc mage processng functonalty of the nspecton system s the ablty to localze the poston of separate graphcal objects. The algorthm s based on template matchng. Ths method agan requres an deal template possbly complemented by mask mage. Localzaton result of the calculator con s shown n the fgure 6. mage smlarty check, object localzaton, OCR functonalty. More complex tasks can be solved usng ther combnaton. Ths should ensure future extensblty and easy adaptaton of the nspecton system to dfferent GUI and screen types. The mage smlarty check s prmarly ntended for the comparson between the deal screen template and the currently dsplayed content captured by camera. The actual comparson algorthm s based on the so called cosne crteron (for more nformaton see [6]) represented by the equaton (2). C A ab a b 2 a a b b 2 (2) Compared mages are regularly dvded nto small areas (e.g. 4x4 pxels). Pxel values from these areas are transformed to the one-dmensonal vectors a (deal template area) and b (dsplayed screen content area). The resultng crteron value C A s naturally normalzed to the range 1; 1. Because the mages have only nonnegatve pxel values, the negatve results do not occur. Geometrcal meanng of the equaton (2) s that C A represents the cosne of an angle between the vectors a and b. Ths mples that the crteron s nsenstve to lnear contrast changes. An mportant role durng the mage comparson has the mask mage presented n bottom part of fgure 3. Each pxel from ths mask serves as a flag regster for Fg. 6: Graphcal object (calculator con) localsaton The ablty to read dsplayed text and numercal data s the thrd mage processng functonalty ncluded n our nspecton system. A serous shortcomng of today avalable OCR algorthms s ther hgh senstvty to font styles. Therefore, the obtaned results should be subjected to some knd of postprocessng algorthm f possble. A smple example of such post processng s a cross-examnaton wth the expected text content. Fg. 7: Utlzaton of optcal character recognton

5 The OCR algorthm currently utlzed n the system s ntended for recognton of general texts. It works well wth wde varety of dfferent fonts and has the ablty to use language dctonares for correctons. Example of ts applcaton shows fgure 7. Despte ts flawless operaton n the above mentoned example, t completely fals wth numbers and characters on fxed segment dsplays, whch are stll very common and even appear as a knd of font n many sophstcated devces. Ths problem has caused the need for addng another OCR method specalzed on fxed segment fonts, whch s currently under development. CONSLUSION AND FUTURE WORK The ntroduced vsual nspecton system s ntended for effortless sem-automated evaluaton of graphcal user nterfaces. It s based on mage processng algorthms, whch n combnaton wth user nput smulator creates a powerful nspecton tool capable of human-lke approach to extensve functonalty checks of electronc devces. Due to the desgned hardware and software structure of the nspecton system, the potental for future expansons and adjustments to new types of tested devces s substantal. Utlzaton of computer vson allows complete separaton of the nspecton system from the tested devce and makes t a sutable tool for examnaton of varous screen technologes. The nspecton procedure programmng nterface s mplemented for.net envronment (usng CLR C++ and C# languages) and can be dstrbuted as a set of dynamc lnk lbrares. C# and other forms of compatble hgh-level.net languages ensure ntutve and rapd development of the nspecton programs. The future development of the core functonalty of the system wll be amed at expanson of modularty and specfcaton of standard nterfaces for vsual data nput and output. Other ways for mprovement le n the development of OCR algorthms and extenson of basc mage processng functonalty. [2] J. C. Slva, J. Cressac, J. Sarava, GUI Inspecton from source code analyss, OpenCert 2010, ISSN: , Avalable from WWW: < [3] G. Bradsk, A. Kaehler Learnng OpenCV, Sebastopol, O Relly Meda, Inc. 2008, ISBN: [4] P. Penna, and R. Patterson, Projectve geometry and ts applcatons to computer graphcs, USA, Prentce-Hall, ISBN [5] J. Žára, B. Beneš, J. Sochor, P. Felkel, Moderní počítačová grafka. Praha: Computer Press, ISBN [6] J. Jan, Medcal Image Processng Reconstructon and Restoraton: Concepts and Methods. Boca Raton: Francs & Taylor 2006, ISBN [7] J. A. Nelder, R. Mead, A Smplex Method for Functon Mnmzaton, Computer Journal, vol. 7, pp ACKNOWLEDGEMENT Ths work has been supported by the Czech Mnstry of Educaton under the project 1M0567 Centre for Appled Cybernetcs. REFERENCES [1] S. E. Black, R. L. K.N. Goodman, Wood, Mult- Channel Deep-Memory Dgtzng Archtecture for Automated Inspecton of Large Composte Surfaces, Autotestcon, 2006 IEEE, vol., no., pp , Sept. 2006