A MODEL-BASED BOOK BOUNDARY DETECTION TECHNIQUE FOR BOOKSHELF IMAGE ANALYSIS Ej TAIRA, Sech UCHIDA, Hroak SAKOE Graduate School of Informaton Scence and Electrcal Engneerng, Kyushu Unversty Faculty of Informaton Scence and Electrcal Engneerng, Kyushu Unversty 6 10 1 Hakozak, Hgash ku, Fukuoka sh, 812-8581 Japan ABSTRACT Several systems based on bookshelf mage analyss have been studed for automatng bookshelf nspecton n lbrares and bookstores. In conventonal systems, book boundares are frstly detected for extractng ndvdual book by usng some popular lne detecton technque, such as Hough transform. Ther detecton accuraces, however, are sometmes nsuffcent. In ths paper, we propose a novel book boundary detecton technque usng a bookshelf model for hgher detecton accuracy. The model s desgned for descrbng the structural propertes of bookshelves and represented as a fnte state automaton, each of whose states corresponds to a component of bookshelves, such as boundary and ttle. Wth ths model, the book boundary detecton problem s formulated as a model-based optmzaton problem where states and local slant angles at all horzontal postons were varables to be optmzed. The globally optmal soluton s effcently searched for usng a dynamc programmng based algorthm. The effectveness of the proposed technque was shown by several experments. 1. INTRODUCTION Inspecton of bookshelves for managng lbrares and bookstores s a tme-consumng and tedous task because there are enormous books n the bookshelves and many of those books are moved frequently and repeatedly. Bar code systems are often utlzed for semautomatng the bookshelf nspecton. Those systems, however, are stll largely based on manual tasks. Several systems based on bookshelf mage analyss have been studed for full-automatng the bookshelf nspecton [1] [5]. The goal of those systems s to extract ndvdual book nformaton, such as ttle wrtten on the spne of each book, from bookshelf mages. For Ths work was supported n part by the Mnstry of Educaton, Culture, Sports, Scence and Technology n Japan under a Grant-n-Ad for Scentfc Research No.14780293. ths goal, the followng four steps are assumed n the conventonal bookshelf mage analyss systems; () detecton of book boundares, () segmentaton of ndvdual book spne regons, () extracton of ttle characters on each book spne regon, and (v) recognton of book ttles. Among those processes, the book boundary detecton process () s the essental front end and therefore ts performance s crucal. In conventonal bookshelf mage analyss systems, book boundares are detected by some popular lne detecton technque. For example, Hough transform on the edge map of a bookshelf mage s employed for detectng contguous and straght edges correspondng to book boundares. Unfortunately, the detecton accuraces of the conventonal technques are sometmes nsuffcent. Ths s because book boundares are often mssed due to the lack of boundary edges caused by ll lghtng condtons and the contguty of books of the same color. In addton, spurous boundares are often detected due to the edges around ttle characters (or llustratons) on book spne regons. In ths paper, we propose a model-based book boundary detecton technque. For hgher detecton accuracy, a fnte state automaton (FSA) model s newly employed, whch descrbes the structural propertes of bookshelves. Roughly speakng, each state of ths FSA model corresponds to a component of bookshelves, such as boundary and ttle. Wth ths model, the book boundary detecton problem s expressed as a model fttng problem to dentfy the state (.e., the component) at each horzontal poston. Fnally, the horzontal postons dentfed as the boundary state are detected as the postons where book boundares exst. For the practcal boundary detecton, t should also be taken nto account that books have ther ndvdual slants and therefore ther boundares are also slanted non-unformly. For copng wth the non-unformly slanted books, the slant angle at each horzontal poston should be estmated optmally. Consequently, the book boundary detecton problem s expressed as
(a) (a) p1 p pm (b) 1 M (b) lne segment at horzontal poston left boundary rght boundary Fg. 2. (a) Non-unformly slanted books. (b) A sequence of lne segments representng the slants of books. left spne (c) ttle rght spne background 2. MODEL-BASED BOOK BOUNDARY DETECTION TECHNIQUE (d) q1 background rght boundary q2 q3 rght spne q4 q5 q6 ttle 2.1. Problem formulaton Fgure 1(a) shows an example of an orgnal bookshelf mage. As shown n Fg. 1(b), we use the mddle part (M (wdth) N (heght)) of the orgnal mage for dsregardng bookshelf boards and the dfference n the heghts of the books. As shown n Fg. 1(c), there are the followng structural propertes n bookshelf mages. left boundary left/center spne Fg. 1. (a) An orgnal bookshelf mage (HSI color mage actually). (b) The mddle part of (a). (c) The structural propertes of (b). (d) An FSA representaton of (c). a model-based optmzaton problem of both the state and the local slant angle at each horzontal poston. The globally optmal soluton s effcently searched for usng a dynamc programmng (DP) based algorthm. Recently, bookshelf nspecton systems usng rado frequency dentfcaton (RFID) tags s developed. Those systems can be also realze full-automatc and contact-free nspecton. Incorporaton of bookshelf mage analyss systems nto the RFID nspecton system s promsng because bookshelf mage analyss systems wll compensate several drawbacks of the RFID tags, such as nterference between tags, sheld by metals, and narrow scope (due to weak wave power restrcted by some law). A book spne regon always les between two parallel neghborng boundares. A book ttle regon les around the center of each book spne regon. A bookshelf background regon sometmes les between two neghborng books. Thus, t seems that the order of the components of bookshelves,.e., boundary, spne, ttle, and bookshelf background, s naturally governed by some rules. In fact, as shown n Fg. 1(d), the rules can be represented by an FSA model composed of 6 states (Q = {q 1,..., q 6 }), each of whch corresponds to one of those components. Wth ths model, the book boundary detecton problem s expressed as a model fttng problem,.e., a model-based optmzaton problem of the sequence s 1,..., s,..., s M where s Q denotes the state dentfed at horzontal poston. For the practcal boundary detecton, t should also be taken nto account that books have ther ndvdual slants and therefore ther boundares are also slanted
non-unformly (Fg. 2(a)). For copng wth the nonunformly slanted books, the slant angle at each horzontal poston should be estmated optmally. Ths estmaton problem can be expressed as an optmal estmaton problem of the sequence p 1,..., p,..., p M where p s the varable to represent the local slant angle at horzontal poston. As shown n Fg. 2(b), p [1, 2,..., M] denotes the horzontal poston of the top end of the lne segment whch passes through the center of the th column. For practcal smplcty, we use p nstead of some real-valued angle. Fnally, for the detecton of the book boundares, we should consder a model-based optmzaton problem of both s and p at each horzontal poston. Specfcally, ths problem s formulated as the followng maxmzaton problem, maxmze w.r.t. M f(p, s ), =1 p, s ( = 1, 2,..., M), (1) where f(p, s ) s a crteron functon to evaluate p and s at horzontal poston. The detals of f(p, s ) wll be descrbed n Secton 2.3. The maxmzaton problem (1) s solved under three constrants. The frst constrant s a condton that the state sequence s 1,..., s M s governed by the state transton rule of the FSA model (Fg. 1(d)). The second constrant s employed to lmt maxmum slant angle and defned as p W (2) where W s a postve nteger specfyng the range of compensable slant angles. The thrd constrant s employed to lmt the nterval between p and p 1 and defned as p 1 + 1 f s {q 4, q 5, q 6 } p = or s 1 {q 4, q 5, q 6 }, (3) p 1 + {0, 1, 2} otherwse. Wth ths constrant, angle fluctuatons are not allowed f s {q 4, q 5, q 6 } or s 1 {q 4, q 5, q 6 }. Ths means that a book spne regon has ts fxed slant. On the other hand, at book boundares and bookshelf background regons, the angle fluctuatons are allowed, whle ts degree s lmted. 2.2. Soluton by DP Consderng a sequental optmzaton process of p and s ( = 1,..., M), the process has the Markov property. Ths s because under the defnton of the maxmzaton problem and the three constrants of Secton 2.1, /* Intalzaton: */ 1 for all s 1 Q do 2 for p 1 := 1 W to 1 + W do 3 g(p 1, s 1 1) := f(p 1, s 1 1) /* DP Recurson: */ 4 for := 2 to M do 5 for all s Q do 6 for p := W to + W do begn 7 g(p, s ) := f(p, s ) + max g(p 1, s 1 1) p 1,s 1 8 bp(p, s ) := (p 1, s 1 1) whch gves the maxmum at step 7 9 end /* Backtrackng: */ 10 (p opt M) := argmax g(p, s M) M, sopt M p,s 11 for := M downto 2 do 12 (p opt 1, sopt 1 1) := bp(popt, s opt ) Fg. 3. DP algorthm. only two values p 1 and s 1 are necessary for determnaton of p and s, and the other past values (e.g., p 2 and s 2 ) are not necessary. It s well known that the optmzaton problem wth the Markov property can be solved effcently usng DP. Fgure 3 shows a DP algorthm for the optmzaton problem formulated n Secton 2.1. The value g(p, s ) s the maxmum (.e., optmal) cumulated value of f(p k, s k k) up to k =. Step 7 s so-called DP recurson, and ts two varables p 1 and s 1 are restrcted by the FSA model (Fg. 1(d)) and the constrants (2) and (3). The optmzed p and s, denoted as p opt and s opt respectvely, are obtaned by the backtrackng procedure Step 10 12. Fnally, f s opt equals to q 2 or q 3, the lne segment at horzontal poston s detected as a book boundary, whch s slanted at the angle specfed by p opt. Note that the DP algorthm n the proposed technque requred about 2.4 sec on a PC (Xeon 1.7GHz) for an mage of sze 640 100. 2.3. Desgn of crteron functon The crteron functon f(p, s ) s desgned based on the followng observatons. Long contguous edges wth near-vertcal drectonal feature are often detected around book boundares. Most edges wth near-horzontal drectonal feature are detected around the ttle characters and the llustratons on book spne regons.
(a) (b) (c) (d) orgnal mage conventonal technque proposed technque Fg. 4. Examples of book boundary detecton. For each mage, the orgnal mage (left), the result by the conventonal technque A (mddle), and the result by the proposed technque (rght) are shown. The book boundares detected are shown as black lnes, and the bookshelf background regons detected by the proposed technque are shown as horzontal hatchngs. In (b) and (c), the boundares correctly detected only by the proposed technque are ndcated by arrows. In (d), the boundary mssed and the spurous boundares detected by the proposed technque are ndcated by arrows.
Edges are rarely detected n bookshelf background regons and book spne regons. Thus, the crteron functon for the book boundary (.e., f(p, q 2 ) and f(p, q 3 )) are desgned to take larger value f there are many and/or long near-vertcal edges on the lne segment at horzontal poston. Snce long contguous edges show the book boundary more clearly than other edges, each edge s gven a weght accordng to the number of the edges connected to t. The crteron functon for the ttle (.e., f(p, q 6 )) s desgned to take larger value f there are many nearhorzontal edges on the lne segment. The crteron functon for the bookshelf background (.e., f(p, q 1 )) s desgned to take smaller value f there are many edges on the lne segment. The crteron functon for the book spne (.e., f(p, q 4 ) and f(p, q 5 )) are desgned to take smaller value f there are many near-vertcal edges on the lne segment. For suppressng the effect of the llustratons, near-horzontal edges are dsregarded. 3. EXPERIMENTAL RESULTS 3.1. Sample mages Experments for evaluatng the proposed technque qualtatvely and quanttatvely were conducted. For the experments, 60 color (HSI) bookshelf mages of sze 640 100 were prepared, each of whch was clpped from an orgnal mage of sze 640 480 as noted n Secton 2.1. Then the followng edge detecton procedure was performed on each bookshelf mage. Frstly, Canny edge detector was performed on three color components (.e., hue, saturaton and ntensty) ndependently. Then the edge map of each component was bnarzed, and the three edge maps were combned to form one edge map. 3.2. Book boundary detecton technque usng Hough transform [5] The proposed technque was compared wth the followng conventonal technque [5]. 1. Hough transform s performed on the edge map prepared n Secton 3.1. At the votng to Hough parameter space, near-horzontal edges, whch may not correspond to any book boundary, are dsregarded and a weghtng operaton for emphaszng long contguous near-vertcal edges s employed. 2. A thresholdng operaton s performed to fnd local maxma n the Hough parameter space. Then the lne correspondng to each local maxmum s detected as a book boundary. Ths conventonal technque has two versons; n the frst verson (hereafter called conventonal technque A ), the threshold to fnd local maxma s adaptvely decded at a value dependng on the result of votng to the Hough parameter space. In the second verson (hereafter called conventonal technque B ), the threshold s fxed at a manually determned value regardless of the result of votng. 3.3. Qualtatve Evaluaton Fgure 4 shows several results by the conventonal technque A and the proposed technque (W = 50). The book boundares detected are shown as black lnes. Fgures 4(a), (b), and (c) show that the proposed technque (rght column) can detect most book boundares correctly. Especally n (b) and (c), t s shown that the proposed technque can detect all boundares of slanted books whle the conventonal technque A fals. As shown n Fg. 4(d), the proposed technque sometmes msses some boundares (false negatve) and detects spurous boundares (false postve). The former s mostly because the colors of adjacent books are smlar each other and therefore boundary edges are lacked entrely. The latter mostly resulted from contguous edges around ttle characters and areas reflectng some lght. It s worth to note that the proposed technque not only can detect book boundares but also can segment a bookshelf mage nto ndvdual book spne regons and bookshelf background regons, whereas the conventonal technque can only detect book boundares. For example, the bookshelf background regons dentfed by the proposed technque are shown as horzontal hatchngs n Fg. 4. Furthermore, snce the proposed technque can also detect ttle regons, t s expected that the proposed technque can be extended to the next steps,.e., extracton of ttle characters and recognton of book ttles. 3.4. Quanttatve Evaluaton In order to ensure the superorty of the proposed technque aganst to the conventonal technques, quanttatve evaluatons were conducted usng the 60 bookshelf mages of Secton 3.1. Fgure 5 shows the number of false negatves and the number of false postves. In the graph, and show the results by the proposed technque and the conventonal technque A, respectvely. The evaluaton of the conventonal technque B was conducted whle changng the threshold to fnd local maxma, and therefore ts result s shown as a curve lne n the graph.
Number of false negatves 300 250 200 150 100 50 proposed technque conventonal technque A conventonal technque B 0 50 100 150 200 250 300 Number of false postves Fg. 5. Number of false postves and false negatves. Table 1. Book detecton rate (%) mage(#books/#mages) detecton rate no slant (231/20) 90.0 unform slant (197/20) 88.8 non-unform slant (182/20) 91.2 total (610/60) 90.0 The number of false negatves by the proposed technque was 3.2% (= 23/727) of all book boundares and far less than 13.6% (= 99/727) of the conventonal technque A. Ths number was less than the half of that by the conventonal technque B even when 300 false postves were allowed. On the other hand, the number of false postves by the proposed technque was slghtly more than that by the conventonal technque A. The reducton of the false postves s remaned as future work. Fnally, we measured the number of books whose spne regons could be correctly dentfed by the proposed technque. Table 1 shows the rate of correctly detected books. In the table, no slant means that all books stand straght closely (lke Fg. 4(a)), unform slant means that all books have the same slant closely (lke Fg. 4(b)), and non-unform slant means that books have ther ndvdual slants (lke Fg. 4(c)). Ths evaluaton shows that the slants of books do not nfluence the book detecton accuracy. Needless to say, the man cause of msdetecton was the falure of boundary detecton. was represented as a fnte state automaton based on the structural propertes of bookshelves. The book boundary detecton problem was formulated as a modelbased optmzaton problem where states and local slant angles at all horzontal postons were varables to be optmzed and then solved effcently usng a dynamc programmng based algorthm. Expermental results show that the proposed technque has superor detecton performance than the conventonal technques. The results also show that the reducton of spurous boundares s the remanng problem. 5. REFERENCES [1] Z. Otake, T. Nagao, T. Agu and M. Nakajma, Extracton of character regons from bookshelf mages, Bulletn of the Japanese Socety of Prntng Scence and Technology, vol. 29, no. 1, pp. 38 45, Jan. 1992. (n Japanese) [2] Y. Ishkawa, M. Ohkura, M. Shono and R. Hashmoto, A study of extracton of character regons from bookshelf mages, Bulletn of the Okayama Unversty of Scence, vol. 32A, pp. 163 171, March 1997. (n Japanese) [3] Y. Akyama and M. Ito, Book recognton from color mages of book shelves, Proceedngs of the IAPR Workshop on Machne Vson Applcatons (MVA 98), Nov. 1998, pp. 106 110. [4] M. Sawak, H. Murase and N. Hagta, Recognton of characters n bookshelf mages usng automatc dctonary selecton based on estmated degradaton, Journal of the Insttute of Image Informaton and Televson Engneers, vol. 54, no. 6, pp. 881 886, June 2000. (n Japanese) [5] K. Hro, M. Fujyama, T. Mura and M. Ito, Book boundary extracton from book shelf mages, Proceedngs of the Meetng of Image Recognton and Understandng (MIRU2000), vol. I, July 2000, pp. 271 276. (n Japanese) 4. CONCLUSIONS A novel book boundary detecton technque usng a bookshelf model was proposed n ths paper. The model