Multispectral color imaging for dermatology: application in inflammatory and immunologic diseases

Transcription

1 Multspectral color magng for dermatology: applcaton n nflammatory and mmunologc dseases Masahro Yamaguch )2), Masanor Mtsu ), Yur Murakam )2), Hroyuk Fukuda 2), Nagaak Ohyama )2), and Yasuo Kubota 3) ) Imagng Scence and Engneerng Laboratory, Tokyo Insttute of Technology 4259-R2-56 Nagatsuta, Mdor-ku, Yokohama , Japan E-mal: guch@sl.ttech.ac.jp 2)Akasaka Natural Vson Research Center, NICT, Japan 3) Department of Dermatology, School of Medcne, Kagawa Unversty Abstract Ths paper presents an expermental nvestgaton of the applcaton of multspectral magng n dermatology. The focus areas of ths work are as follows: a) the mprovng the color reproducton accuracy of skn lesons, b) explorng the spectral feature of skn dsease usng the multspectral color enhancement technque, and c) multspectral mage analyss amng at supportng quanttatve dagnoss. The experment focused on nflammatory and mmunologc dseases; the color of skn lesons assocated wth these dseases s beleved to be dffcult to reproduce by conventonal magng devces. In vew of ths fact, we demonstrate the effectveness of usng spectral nformaton for the color reproducton and quanttatve analyss of skn dsorders.. Introducton The color of skn lesons s of great mportance n dermatology. The leson sze s determned by a slght color dfference and the accurate observaton of skn color can help n determnng whether the leson s bengn or malgnant. In dermatology, hghly accurate color reproducton s requred for photographc mage recordng. Dgtal magng technology s appled to dermatology for constructng an mage and teledermatology. Proper advce by expert dermatologsts can be obtaned va teledermatology [,2] ; ths s partcularly useful when the dagnoss of skn dsease s relatvely dffcult for general physcans. The utlzaton of such dgtal magng systems can mprove the qualty of medcal servces. However, the color of the reproduced mage s not accurate n conventonal color magng systems, and t has been determned that the condton of skn lesons cannot be well rendered by conventonal dgtal mages [3]. Hgh-fdelty color reproducton technology s requred n varous felds such as prntng, electronc commerce, and the dgtal archvng of artworks or cultural hertages. For the accurate reproducton of the orgnal color through dgtal magng systems, the multspectral magng technology s promsng [4 7]. By usng the spectral nformaton, the reproduced color accuracy s consderably mproved as compared to conventonal RGB-based systems; moreover, the quanttatve spectral nformaton can be employed for the analyss or the nformaton retreval from the mage database. Multspectral magng technology has been appled to dermatology manly for the dagnoss of melanomas [8]. It has also been used to estmate the oxygen saturaton of blood, based on the spectral characterstcs of oxy- and deoxy-hemoglobn [9]. We are also conductng experments amed at the applcaton of multspectral magng to hgh-fdelty color reproducton and dagnostc support for skn dseases. In ths paper, we address the nflammatory and mmunologc dseases because they render subtle color varaton, whch s dffcult to be reproduced by conventonal RGB magng. We demonstrate the expermental results of the color reproducton accuracy of skn dseases as well as the vsualzaton and quantfcaton of the lesons by usng a 6-band multspectral camera. 2. Multspectral magng system The multspectral magng system used n ths experment comprses a 6-band hgh-resoluton multspectral camera (MSC) [0] and a multspectral mage database based on the system developed by Akasaka Natural Vson Research Center (NVRC), Natonal Insttute of Informaton and Communcaton Technology (NICT). The confguraton of the 6-band MSC s shown n fg.; a flter wheel wth 6 nterference flters s attached, and an mage wth 2-bt gray level and a resoluton of pxels for each band can be captured. Fgs. 2 and 3 show the appearances of the MSC and the expermental setup at the consultng room n the Department of Dermatology, Kagawa Unversty Hosptal. Two tungsten lamps are used for the llumnaton, and the spectral energy dstrbuton of the llumnaton lght s measured by usng a standard whte plate. Although t s possble to attach ether a telephoto lens or a wdeangle lens to the MSC, we use the former to fll the frame wth a 50 mm 50 mm area due to the lmtaton n the room sze and the magnfcaton expected by dermatologsts. Snce t takes about 0 s to capture a 6-band mage wth the rotatng flter wheel, the patent must not move durng the exposure. To avod moton blur, a small lamp and buzzer are used to capture the patent's attenton durng the perod when the patent has to keep stll. In addton, a frame that s slghtly larger than the area to be photographed s placed, and the patent s allowed to place the regon of the leson on the frame so that the mage can be captured wthout adjustng

2 Flter wheel Motor drver Flter Controller Camera Controller motor PC I/F PC CCD Camera USB Image DB Changeable optcal system unt Wde-angle or telephoto lens Fgure System confguraton of multspectral magng system developed for dermatology applcatons. Montor for GUI Image Dsplay Fgure 2 6band Multspectral camera Fgure 3 Expermental setup at the consultaton room n the Department of Dermatology, Kagawa Unversty hosptal. the framng or focusng. Although t s mpossble to photograph lesons n some cases due to the dffculty of postonng, we have lmted the target regons n ths experment to those that can be captured by ths system, such as hand, arm, back, and face. Captured mages are stored n the mage database n the Natural Vson data format (NV format) [], developed by the NVRC. The NV format contans profle nformaton of the mage capturng setup, such as the spectral senstvty of each band and the spectral radance of llumnaton, n addton to multspectral mage data. Usng the profle data, the stored mage can be utlzed for the natural color reproducton and for mage analyss usng quanttatve color nformaton. In the database system, the multspectral mage s assocated wth the patent and hs or her clncal nformaton. Hence, a clncal case can be retreved from the database by usng a combnaton of a text-based search usng the patent s clncal nformaton and an mage-based search wth spectral features of the skn dsease. Fg.4 shows a screen capture of the mage database, n whch the regon of nterest can be selected usng the segmentaton algorthm based on spectral reflectance. In addton, related clncal cases are retreved by the smlarty of spectral features, whch are Fgure 4 Screen capture of mage database system. In ths system, the user dermatologst specfes the regon of nterest (blue mark n the mage), and magebased retreval can be done by the spectral smlarty of the specfed regons. ndependent of the magng devce and the llumnaton envronment. 3. Color reproducton accuracy of skn dseases Although there have been many reports on the color reproducton of human skn, most of them deal wth the skn of healthy people. From the analyss usng the spectral reflectance of normal skn, t has been shown that far color reproducblty can be acheved from even a three-band mage f the pror nformaton (.e., bass functons or covarance matrx) on skn reflectance has been provded n the color estmaton. On the other hand, the spectral characterstcs of abnormal skn have not yet been well nvestgated n detal. Even though a multspectral mage wth enough number of bands mparts hgh color reproducblty, the number of bands requred to attan a compact and nexpensve magng devce s expected to be reduced. However, the requrements of the number of bands and the accuracy of color reproducblty have not been clarfed to date. In ths experment, we nvestgated the color reproducton accuracy by usng a 6-band multspectral mage, whch s consdered to acheve a suffcently hgh accuracy of color reproducton. As the frst step of the experment, spectrum-based

3 3 Color dfference vs Number of bass functons Normal Dermatomyosts Palmoplantar pustuloss 7 6 Average + 2σ 9 5 Average E*ab E * ab Number of bass functons Fgure 5 CIE color dfference between the colors reproduced by the orgnal 6-band multspectral mage and K (K =... 6) bass functons derved from normal skn for normal, dermatomyosts and pulmoplanter pustuloss cases. color reproducton was appled to the 6-band mage of skn dseases and dsplayed on a flat panel LCD after color calbraton by spectral characterzaton. Through vsual evaluaton by dermatologsts, t was confrmed that excellent color reproducblty s realzed n the clncal mages even better than mages captured on reversal flms. In the next step of the experment, a computer smulaton to estmate the color reproducton accuracy when usng a smaller number of bands was carred out; the color reproduced by the 6-band multspectral mage was consdered as the gold standard. In the computer smulaton, the pxel value of the 6-band multspectral mage that was dvded by the spectral radance of the standard whte plate was regarded as the 6-dmensonal spectral reflectance. We used the 6-dmensonal bass functons derved by the prncpal component analyss (PCA) of the spectral reflectance of normal skns measured prevously [2]. The spectral reflectance of normal and abnormal skn were obtaned from the mages and expanded by the bass functons. The color reproducton accuracy was subsequently evaluated as the color dfference between the trstmulus values reproduced from orgnal 6-band mage and the coeffcents for the lmted number of bass functons. Fg.5 shows the relatonshp between the number of bass functons and the average color dfference (CIE E * ab under CIE D65 llumnant) for the cases of normal skn, palmoplantar pustuloss, and dermatomyosts. The number of bass functons requred for a certan color estmaton accuracy approxmately corresponds to the requred number of bands for the magng devce, under the assumpton of usng the bass functons for color estmaton. Therefore, fg.5 roughly ndcates the lower lmt of the color dfference when usng the camera wth K bands. Fg.6 llustrates the color dfference when usng three bass functons, whch corresponds to the case of a three-band camera, for varous types of skn dseases. Inflammatory dseases (psorass 0 Normal palmoplantar pustuloss drug erupton psorass vulgars Fgure 6 CIE color dfference between the colors reproduced by the orgnal 6-band multspectral mage and three bass functons derved from normal skn. Bold bars represent the average and thn bars represent average + (standard devaton) 2 color dfferences of several areas selected from the mages. vulgars, palmoplantar pustuloss, erythroderma, and drug erupton) and mmunologc dseases (scleroderma and dermatomyosts) exhbt dfferent trats, namely, the colormetrc error s larger n mmunologc dseases than n nflammatory cases. The color dfferences n mmunologc dsease cases are larger than the dscrmnaton threshold of human vson when three bass functons are used. The reason for ths phenomenon s nterpreted as follows. The spectral reflectance s affected by the morphologcal structure nsde the skn,.e., the dsorder located n a deep regon, such as hypoderms or muscles n the case of scleroderma and dermatomyosts, whle dermatts manly affects the superfcal layers around the epderms and the basal layer. Due to the wavelength dependency of lght scatterng n the skn, long wavelength lght penetrates deeper regons than short wavelength lght. In mmunologc dseases, specfc textural features such as collagen dsorders n relatvely deep regons cause dfferent lght transport characterstcs especally for long wavelength, and the spectral transmttance s not well descrbed by the bass functons of normal skn. In these cases, more than fve bands are requred to acheve the colormetrc accuracy equvalent to the case n whch normal skn s photographed by a three-band camera. 4. Color enhancement of multspectral mage Snce the color of skn s determned by the dstrbuton of hemoglobn and dyes such as melann as well as the wavelength dependency of lght scatterng nsde the skn, there may be a strong relaton between the condton of skn dsease and the spectral feature of reflected lght. However, the spectral feature assocated wth the dagnoss has not yet been determned. In ths research, we conducted an experment to explore the spectral feature of skn lesons usng 6-band multspectral mage presented above. One of the dffcultes n observng the features n the spectral reflectance s the ndvdual varaton n the normal skn color. To erythroderma scleroderma dermatomyosts nevus splus

4 Reflectance Center wavelength (Band number) Oxy-hemoglobn Melann Deoxy-hemoglobn () (2) (3) (4) (5) (6) (7) (8) (9) (0) () (2) (3) (4) (5) (6) (nm) Fgure 7 The wavelength and enhanced features. The horzontal axs represents the center wavelength of each band of MSC [() (6)]. The plots are the spectral reflectance of melann and hemoglobn meda calculated from the absorpton coeffcents of the dyes. Snce the short wavelength lght does not penetrate deeper due to the lght scatterng property nsde the skn, the nfluence of absorpton at superfcal layers are domnant n short wavelength, whle the nfluence of deep layers appear n long wavelength components. Accordng to these consderatons, the features vsualzed by enhancng certan wavelength regons are also depcted. Enhanced color and hstologcal features Yellow - rch Volet - poor melann Red - superfcal capllares Blue - absorpton n deep layers (e.g., ven) Factors related to the enhancement Absorpton n superfcal layers Absorpton n deep layers Absorpton by melann Absorpton by hemoglobn vsualze the spectral feature along wth suppresson of normal color varaton, we appled a color enhancement technque for the multspectral mage [3], n whch a specfc wavelength component s enhanced; however, the color of the normal regon s kept unchanged. The method s descrbed below. Let us consder that a PCA s appled to N-band multspectral pxel values g(r), where r s a poston vector n the mage. Then, g(r) can be expressed by a lnear combnaton of N bass vectors (u, =...N) as follows: N gr () = α() ru, () = where α (r) s a coeffcent for every pxel and -th bass vector. If the pxels used n the PCA are obtaned from the normal part, then the bass vectors wth larger egenvalues represent the spectral characterstcs of the normal part. Let g p (r) denote multspectral pxel values reconstructed from only the top m(<n) sgnfcant bass vectors: m g ( r) = α ( r) u. (2) p = Then, g p (r) s subtracted from the multspectral pxel value so that the spectral dfference from the normal part s extracted as g( r) = g( r) g p ( r) N, (3) = α ( r) u = m+ where g(r) s termed as the dfferental multspectral mage n ths paper. To vsualze the mage feature correspondng to respectve wavelength bands, a specfed wavelength component s enhanced by multplyng an approprate weghtng factor to the correspondng element of g(r). The resultant mage,.e. a dfferental multspectral mage where a specfc wavelength component s enhanced, denoted as g e (r), s added to g p (r) to obtan the enhanced multspectral mage g e (r): g ( r) = g ( r) g ( r). (4) e e + p The color mage s generated under the assumpton that g e (r) s an mage vrtually obtaned from an MSC, and a color reproducton algorthm for multspectral mage s appled [7]. Snce ths method enhances hgher order components, t s sutable for fndng subtle changes from the spectral mage that has spatal color varatons. On consderng the applcaton of ths method to the multspectral mage of skn, the spectral feature of abnormal part s enhanced as the dfference from the normal skn, dependng on the specfed wavelength. In vew of the fact that long wavelength red lght penetrates deeper than short wavelength blue lght and consderng the spectral absorbance characterstcs of melann and hemoglobn, the enhanced mage wll exhbt the spectral features shown n fg.7. It s expected that the quantty of melann n the superfcal layer wll be observed n the component at approxmately 470 nm, the densty of blood capllares s reflected at approxmately 550 nm, and blood vessels such as vens n relatvely deep regons are enhanced n the 620-nm component. In fg.8, the results of spectral color enhancement appled to the clncal mage of psorass vulgars are shown. In ths case, m n eq. (2) s set to be 3. The regon of vascular dlataton and prolferaton are vvdly vsualzed by the 550-nm enhancement. Fg.9 shows the result of dermatomyosts, whch s one of the systemc autommune dsorders. In contrast to the case of psorass where the clarty of the enhanced regon s better at 550 nm than 580 nm, n the case of dermatomyosts, the 580-nm mage exhbts the lesons more clearly than 550-nm mage. Ths can be explaned by smlar characterstcs mentoned n secton 3, namely, the spectral feature appears longer wavelength snce deep

5 PCA Patent Multspectral camera Multspectral Image Normal component (a) (b) (c) Fgure 8 Color reproducton (a) result; (b) 550-nm and (c) 580-nm enhanced mages of psorass vulgars. Spectral color enhancement Spectral feature classfcaton Quantfcaton of dsease Spectral data processng Dfferental multspectral mage Specfy wavelength Fgure 0 Proposed algorthm for multspectral skn color analyss. (a) (b) (c) Fgure 9 Color reproducton (a) result; (b) 550-nm, (c) 580-nm enhanced mages of dermatomyosts. Before treatment Before 50% - 80% < 20% 20% - 50% > 80% regons such as muscles and hypoderms are affected n ths type of dsorder. Through ths experment, the spectral color enhancement technque has been shown to be useful for nvestgatng the spectral feature of skn lesons. The possblty of applyng multspectral magng technology for the quantfcaton or classfcaton of skn lesons such as psorass usng the 550-nm wavelength component and suppressng the normal varaton of spectral reflectance has been demonstrated. It has been also demonstrated that the color dfference between nflammatory dseases such as psorass and mmunologc dseases such as dermatomyosts can be evaluated usng multspectral mages. The color enhancement technque can be also appled to support the dagnoss va the mage-based telemedcne system. There are dffcultes n the communcaton between remote stes through the network because of lmtatons n the observaton flexblty and the absence of tactle nformaton. Thus, t would be valuable to send a spectral mage and vsualze supplementary mages wth a specfc wavelength enhanced, along wth the natural color mage. It s expected that wth the avalablty of supplementary nformaton on the features of the skn lesons, a more precse recognton of the condton of skn dsease would be possble. 5. Applcaton n dagnostc support n dermatology In ths secton, an example of computerzed dagnostc support usng the multspectral mage s presented. The dfferental multspectral mage g(r) ntroduced n secton 4 expresses the features of the skn leson when the varaton of normal color s removed. Let us consder the applcaton of the dfferental multspectral mage to computerzed dagnostc support n dermatology as shown n fg.0. By choosng one or several wavelength components that contan characterstc nformaton on a specfc dsorder, t wll be possble to extract the abnormal 3 weeks after After 0% 20% 40% 60% 80% 00% Fgure Example of applcaton: vsualzng blood quantty. The mages on the left are the photographs of the same patent wth psorass captured at dfferent tmes (before treatment and 3 weeks after). The mages n the mddle show the blood quantty dstrbuton, where the densty of blood-rch regon s expressed by pseudo color encodng. The rato of densty n the correspondng mage areas s shown n bar graphs on the rght. regon, or to derve quanttatve nformaton for dfferental dagnoss. In fg.0, the normal component s derved from PCA of orgnal multspectral mage. However, a set of bass functons yelded from PCA can be used for the mages of other patent or other dseases n prncple. Snce the result of dfferental multspectral mage depends on the selecton of normal part from the mage, t s needed to address a method to defne the bass functons that gves robust and accurate result ndependent on the patent and dsease n future. We have mplemented an algorthm to extract the regons where the blood quantty s ncreased. In the algorthm, pxels larger than certan threshold values (blood-rch pxels) are obtaned from the 550-nm component of the dfferental multspectral mage. Subsequently, a small wndow s set around every pxel, and all the pxels are classfed accordng to the densty of the blood-rch pxels wthn the wndow. Fg. demonstrates the applcaton of ths method to the mages of psorass vulgars captured at dfferent tmes: before the treatment and three weeks after. In fg., the densty of blood-rch pxels s expressed as 0 00%, and the pseudo color mages clearly exhbt the condtons of

6 psorass. If the maged regon can be adjusted, quanttatve evaluaton n the form of bar charts n the rght sde of fg. wll be possble. In ths experment, snce automatc adjustment was dffcult, correspondng regons between two mages were manually selected (as marked n the mages on the left sde) and the results n these small regons are plotted n the bar charts. For practcal applcaton, a larger regon, e.g., a whole arm should be evaluated. Although there are a substantal number of ssues to be addressed, t has been shown that the analyss of multspectral mage provdes effectve nformaton for supportng dagnoss n dermatology, such as the gradng of dsorders or the quanttatve evaluaton of treatments. 6. Concluson In ths paper, we present the applcaton of multspectral magng to color reproducton, color enhancement, and mage analyss for dermatology, focusng on nflammatory and mmunologc dsorders that exhbt subtle color appearance. Through the experment on color estmaton, we show that the spectral reflectance has dfferent characterstcs n normal and abnormal skn dependng on the types of dsorders. Especally for scleroderma and dermatomyosts, t can be predcted that threeband cameras ndcate a vsually apparent error n color reproducton. It s also demonstrated that the applcaton of the spectral color enhancement technque s useful for nvestgatng spectral features of abnormal parts even f the normal parts have color varatons. Specfc wavelength components of the dfferental spectral reflectance can be also employed for the quantfyng or gradng of skn dsease. In some types of skn dseases, rregular spectral features are observed as the optcal propertes of tssue changes from normal skn. Spectral mages hold not only the nformaton on dye dstrbuton but also the hstologcal structure of skn dsease. Such spectral features have not been revealed yet, such as lght scatterng n the layered structure. Although further nvestgaton s expected, multspectral magng s an effectve tool for capturng the nformaton useful for dermatology. Although an MSC wth a rotatng flter wheel s not sutable n practce, recently, a 6-band MSC for movng mages has also been developed [4]. For practcal applcaton of the presented technque n dermatologcal dagnoss, compact and user-frendly devces for multspectral mage capture are necessary. In ths aspect, a hghperformance MSC wth smaller number of bands s requred. When examnng the reducton n the number of bands, 6-band multspectral mages of abnormal skns are also useful for evaluatng the performance of the magng devce wth smaller number of bands. Dgtal photography wth hgh color reproducblty that enables quanttatve mage analyss s qute benefcal n dermatology and other felds. To put ths technology nto practcal use, the clncal evaluaton s an essental ssue for future work n addton to the development of handy magng devces. Ths work s supported by the Specal Coordnaton Funds for Promotng Scence and Technology of the Mnstry of Educaton, Culture, Sports, Scence and Technology, conducted as a jont project of Tokyo Insttute of Technology, NTT Data Corp., and Olympus Co. n collaboraton wth NVRC of the Natonal Insttute of Telecommuncaton and Communcaton Technology (NICT) and Kagawa Unversty. The authors wsh to acknowledge Toshhko Numahara, MD, for valuable dscussons and Yoshfum Kanno of NTT Data Corp. and Yasuhro Komya of Olympus Co. for techncal support. References. D. J. Eedy and R. Wootton, "Teledermatology: a revew," Brtsh J. of Dermatology, Vol.44 (200) E. A. Krupnsk, B. LeSueur, L. Ellsworth, N. Levne, R. Hansen, N. Slvs, P. Sarantopoulos, P. Hte, J. Wurzel, R. S. Wensten and A. M. Lopez, "Dagnostc accuracy and mage qualty usng a dgtal camera for Teledermatology," Telemedcne Journal, Vol.5, No.3 (999) T. Numahara, "From the standpont of Dermatology," n Dgtal Color Imagng n Bomedcne, H. Tanaka, Y. Myake, M. Nshbor, and D. Mukhophadhyay, eds., Dgtal Bocolor Socety, (200) 67-72, umn.ac.jp/book20002/dn pdf 4. H. Hanesh, T. Hasegawa, A. Hoso, Y. Yokoyama, N. Tsumura, and Y. Myake, System desgn for accurately estmatng the spectral reflectance of art pantngs, Appl. Opt., Vol.39, No.35, (2000) B. Hll, "Color capture, color management and the problem of metamersm: does multspectral magng offer the soluton?" Proc. SPIE, 3963, (2000) P. D. Burns and R. S. Berns: "Analyss of multspectral mage capture," Proc. 4th Color Imagng Conference, (996) 9-22, 7. M. Yamaguch, T. Teraj, K. Ohsawa, T. Uchyama, H. Motomura, Y. Murakam, and N. Ohyama, "Color mage reproducton based on the multspectral and multprmary magng: Expermental evaluaton," Proc. SPIE, vol.4663, (2002) S. Tomats, A. Bono, C. Bartol, M. Carrara, M. Luald, G. Tragn, and R. Marchesn, " Automated melanoma detecton: multspectral magng and neural network approach for classfcaton," Med Phys. Vol. 30, No.2: (2003) N. Tsumura, N. Ojma, K. Sato, M. Shrash, H. Shmzu, H. Nabeshma, S. Akazak, K. Hor, and Y. Myake, "Image-based skn color and texture analyss/synthess by extractng hemoglobn and melann nformaton n the skn," ACM Transactons on Graphcs, Vol. 22, No. 3 (2003) H. Fukuda, T. Uchyama, H. Hanesh, M. Yamaguch, and N. Ohyama, "Development of 6-bands multspectral mage archvng system," Proc. SPIE, Vol.5667 (2005) H. Motomura, K. Ohsawa, N. Ohyama, M. Yamaguch, and H. Hanesh, "Natural Vson mage data fle format for spectral-based color reproducton system," Proc. PICS, (2003) Y. Ohya, T. Ob, M. Yamaguch, N. Ohyama, and Y. Komya, "Natural color reproducton of human skn for telemedcne," Proc. SPIE, Vol.3335, (998) M. Mtsu, Y. Murakam, T. Ob, M. Yamaguch and N. Ohyama, Color enhancement n multspectral mage usng the KL transform, Optcal Revew, Vol.2, No.2 (2005) K. Ohsawa, T. Ajto, H. Fukuda, Y. Komya, H. Hanesh, M. Yamaguch, and N. Ohyama, "Sx-band HDTV camera system for spectrum-based color reproducton," J. Imagng. Sc. and Technol., vol.48, no.2, (2004) 85-92