Model Research on the Optimized and Improved Design of Lucene Search Engine Based on Big Data

Transcription

1 Model Research on the Optmzed and Improved Desgn of Lucene Search Engne Based on Bg Data Shaoyu Lang 1, Syn Lang 2 1 Guangzhou Huashang Vocatonal College, Guangdong , Chna 2 Guangdong Huashang Techncal School, Guangdong , Chna Abstract Lucene open source search tool, as the current frequently used search tool, has many problems n the nfnte expanson of data nformaton. Due to the fact that the search operaton does not reach the optmzed drecton, the data volume thus lmts the search effectveness of effectve nformaton n the search process so that the tmelness of search engne s not rapdly presented. Therefore, ths study adopted two processng modes of ndex ncrement and ndex update, and optmzed the nternal search mode, thus achevng the advantage of smplfyng the search process so as to reduce the declnng probablty of the man ndex mergng optmzaton frequency and acheve the tmelness of search. At the same tme, the ndex acceleraton analyss method of sngle ndex fle was used to optmze the desgn envronment, and correspondng mergng strateges for the man ndex local optmzaton of mult-ndex fle storage was desgned, thus optmzng and mprovng the Lucene search engne and acheve the ultmate effect of lftng effcency through the use of bg data support search functon. Keywords: Data Volume, Lucene, Search Engne, Vector Space, Full Text Informaton Retreval. 1. RESEARCH BACKGROUND 1.1 Lterature revew In the era of bg data, the acquston and search of huge amount of nformaton have become the necessary way to obtan nformaton at present and the utlty of search engnes has also been verfed n many felds (Wu et al., 2016). However, Lucene open source search tool, as the current frequently used search tool, has many problems n the nfnte expanson of data nformaton (Ma and Ca, 2015). The search operaton does not reach the optmzed drecton n the search process, thus lmtng the search effectveness of effectve nformaton so that the tmelness of search engne s not rapdly presented (Zhang et al., 2015). Ths stuaton has long plagued developers n the optmzaton of the mplementaton model of nformaton search functon. However, Lucene tself s only the toolkt of search engnes. Therefore, there s stll a certan computatonal dlemma for truly accurate analyss of large amount of data n the face of massve nformaton. For ths purpose, ths study addressed ths ssue and amed to mprove the applcablty of the Lucene open source search tool and provde the functon presentaton mode for the ndex nformaton after the expanson of data base. 1.2 Research purposes In vew of the many problems exstng n the Lucene, ths study manly makes a dstncton between energy effcency of bg data search. On the one hand, the sngle engne cachng technology s optmzed, so as to expand the multplexng functon of memory ndex and the search acceleraton s also developed (Fu et al., 2015). On the other hand, based on the tmelness of mult-fle storage, t can be used as the man ndex layout to conduct optmzaton, and thus the advantage of promptness of searchng for fragmented nformaton can be hghlghted. The soluton to reduce the hgh operatng cost under the expanson of ndex base s proposed (Ren and Zhang, 2014). In addton, two processng modes of ndex ncrement and ndex update are adopted to optmze the nternal search mode, achevng the advantage of smplfyng the search process so as to reduce the declnng probablty of the man ndex mergng optmzaton frequency and acheve the tmelness of search. Consderng the real-tme requrements n practcal applcaton and the problems of tradtonal real-tme search solutons n the case of large amount of data, ths research proposed a new search soluton and acheved the 393

2 effectve exctaton of the tmeless of full text search engne wth the ndex and search optmzaton solutons, recevng better feedback nformaton n actual projects. 2. SEARCH ENGINE CORE TECHNOLOGY OF LUCENE OPEN SOURCE CODE 2.1 Bref analyss of Lucene technque Lucene s a separate subproject set up by the Apache Software Foundaton n the Jakarta project group. Although ts full text search engne toolkt of open source code does not have a complete full-text search engne, t s equpped wth the full-text search engne functon, provdng a relatvely complete query servce and ndex functon. The logcal arrangement of specal characters, Englsh as well as German can be conducted n some of ts textual engne analyss, thus greatly smplfyng the character search effect of search engne n the applcaton process (Zhang and Hu, 2015). The desgn goal of Lucene s to provde users wth a smple toolkt paradgm to facltate the full text search functon, or to mprove the multple functons of full text search engne on ths bass. Especally n the full text search and search open source lbrary, ts applcaton nterface s more smplfed, whch can bascally acheve the techncal support of full text ndex and search. Especally n the Java development envronment, Lucene can be regarded as the relatvely mature free open source tool. Therefore, the use effcency and economc benefts are relatvely hgh and can be appled as a unversal tool. 2.2 Full text nformaton retreval structure Full-text retreval system s a comprehensve nformaton retreval functon based on bg data technology. Wth the support of the full text database, all the query nformaton can be completely presented n the nformaton source, and the converson of all the content s taken the data collecton of computer dentfcaton and nformaton processng unt (Wang and L, 2014). Therefore, the full text database not only stores a large amount of applcable nformaton, but also possesses the substantve functon for full-text data retreval. The retreval target nvolves vocabulary, words, paragraphs, etc., and deeper edtng and processng. In all full text databases, the massve nformaton can serve as a functonal support for the nformaton volume n the database so as to acheve the unstructured data mplementaton system management of full text database. In the system structure of underlyng RDBMS, the nformatonal advantage of managng large amount of unstructured data are hghlghted and the base rate for queryng massve unstructured data s enhanced, thus achevng the probablty of effcent processng of unstructured data through full text retreval technology (Sun and Hong, 2013). The structure of full text nformaton retreval process s shown n Fgure 1. Fgure 1. Full-text Informaton Retreval Process Structure 3. OPTIMIZATION DIRECTION OF LUCENE SEARCH ENGINE UNDER BIG DATA SUPPORT 3.1 Lucene open source ndex functon ndcator The ndex functon of the Lucene open source tool durng the development stage s to realze the quck retreve query nformaton of search engne. In the process of contnuously accumulatng database nformaton, ts nformaton retreval functon needs to accurately access to the coverage n the full text database and reorganze 394

3 of the nherent mplcaton of the query nformaton n order to obtan more effectve nformaton. Therefore, Lucene's functonal appeal s manly reflected n the followng aspects, whch s shown n Fgure 2. Fgure 2. Lucene Open Source Tool Index Feature Gude Frstly, the full text database tself contans the orgnal features of the nformaton, so the next retreval process needs to objectvely evaluate the orgnal text n the database nformaton to acheve the objectvty of the unfed ndcator n the ntal data. Secondly, n the nformaton retreval envronment, the objectve ndex exsts n the authentcty of the content of the retreval. If the nner connecton of any characters or words cannot be found, the retreval tself tends to be weakened, so Lucene's functonal characterstcs must also reflect thoroughness (Guo and Cao, 2013). Ths thoroughness s a rgd ndcator to mprove the coverage of search nformaton, as well as a necessary condton to evaluate the tmelness of search engnes. Thrdly, although the search language used s a collecton of German and Englsh, t must bear a hgher naturalness n ts language paradgm and ths feature s based on the search request to reduce manual ndexng. If we can acheve the retreval functon of natural language by matchng mode such as truncaton or adjacency, the requred documents wll quckly reflect the locaton, thus mprovng the retreval effcency. And ths optmzaton drecton s also an mportant feature that dstngushes t from the tradtonal keyword retreval method. Fourthly, the stablty of data nformaton s the fundamental ndex of whether Lucene tools can exert the retreval ablty. In the search process of extensve nformaton, the full text database s also close to the closed state. If the data s not updated, when reachng the functonal ndctors of fast ndex, the stablty of the effectveness of data nformaton acquston must be acheved so that the sequental dsorder of ndex nformaton n the tme dmenson can be avoded (Y and Chen, 2014). In addton, the accuracy of the search results and the unstructured nature of the data structure are fundamental functons that Lucene tools already had. These two fundamental functons should not be weakened n combnaton wth other optmzaton crtera, or else the search effectveness of output data nformaton wll also declne. 3.2 Technology development goal of search drecton Currently, there are many ways of nformaton retreval appled n Lucene, among whch the four types are the most common, namely: Boolean logc method, fuzzy logc method, probablty retreval method and vector space representaton method. In the practcal applcaton of techncal lnk, all search engnes must fully consder the exstence of vector space, as well as the actual role of Boolean logc. The basc reason s that although the Boolean logc algorthm can ndeed mprove the search base n the nformaton query and the advantage of ths method s that the retreval nformaton can be quckly presented, yet the sortng functonalty of the Boolean logc algorthm s not 395

4 suffcent. If the sortng functonalty can be mproved, t then can be used n retrevng nformaton. The descendng sort of data nformaton wth hgh smlarty of ths method has not yet completed. Therefore, most of the researches propose to objectvely descrbe the essental content of the retreved nformaton combned wth the manfestaton of vector space (Wu and Y, 2012). For Lucene, the most commonly used search model s also a combnaton model of spatal vector model and Boolean model. Frst of all, the vector space model s an egenvector expresson mode that reflects the textual nformaton through the spatal effect. The nformaton search method s to create an effectve search space n the database so as to acheve the scope defnton of a specfc threshold. Then t wll be able to provde entry ndex agan and objectvely descrbe the mportant nformaton such as functon range and keyword. The vector space functon model s: V D) [ T, W ( D);... T, W ( )] (1) ( 1 1 N N D In ths formula, the base parameter of the entry search tem s represented by T 1 ( = 1,2,3 N), whle W (D) the weght rato of the space vector n T and D. In other words, the objectve functon appearng n T and D can be regarded as the frequency functon equvalent to TF (D). The theoretcal model s: WN( D) [ TF( D)] (2) The known parameters expressed n the above formulas are the techncal ndexes that must be realzed by data nformaton n the retreval functon. Snce the method takes the weght rato of the entry as the extenson condton, the advantage of precse dockng of search tems can be acheved durng the calculaton (Hu and Lu, 2011). The computng method of the weght rato of the entry s: N TF( D) log n (3) In the parsng process of ths functon model, N needs to set as the database fle base, and at the same tme n s used to express the search entry base contaned n the document. Then, the poston defnton of unknown nformaton can be obtaned through ths functon model, and the functon model can be used for dervaton and search analyss of TF-IDF functon (Zheng and Ln, 2010). The analytcal model s: W( D) N TF ( D)log( 0.1) n 2 N tf( d) log 2( 0.1) n n (4) Ths model can be used to fully express the functon TF-IDF and s also the best scheme to summarze the concentraton of document nformaton. In the document nformaton search entry for a large amount of ntegraton, the changes lead to the ncrease of document base. And ths knd of condton s also the nevtable manfestaton of attrbute weakenng n the entry partton. The ncrease of weght rato can reduce the probablty of ths stuaton. 4. ACCELERATION PROGRAM OF LARGE DATA VOLUME INDEX 4.1 Acceleraton Analyss and Desgn of Sngle Index Fle The reducton of the dsk IO operaton has long been the bggest bottleneck of dsk IO, and the reason why large data volume ndex faled to mprove tmely s also based on such factors. If such occason, the tmely adopton of SSD accumulaton can ncrease the dsk IO speed, but ths stuaton s a contngency plan. In practcal applcaton, there s an unnecessary rsk of waste of resources. Therefore, the ntegrated optmzaton of desgn structure s the optmzaton drecton based on the mprovement of ndex effcency. In the search process of sngle ndex fle, f techncal ndcators are used for classfcaton, t can also be regarded that two manstream 396

5 methods are used to solve the bottleneck of dsk IO problems: on the one hand, through the tradtonal reducton of IO operatonal process and the ncrease of cache and parallel ndex, the expected goal s thus acheved. On the other hand, the method of ncreasng the cachng mprove the functon realzaton of the vocabulary ndex, and the multplex sngle Index Wrter object can also be used (L and Geng, 2010). Ths mplementaton method can at least realze the optmzaton of phased ndex, whch, n common occason, can usually prevent the ndex fle from beng locked and then throwng an excepton, leadng to ndex nterrupton. Moreover, the extra memory and tme consumpton and can be reduced by frequent loadng. The code of specfc sngle case Index wrter s: publc classlucenemanager{ prvateindex Wrter { synchronzed(lock_w) { ndex Wrter = null; prvateobject lock_w= new Object(); f(ndex Wrter == null){ //Flelock publc Index Wrter get Index Wrter(Drectory drectory, Index Wrter Confgwconfg)throwsCorrupt Index Excepton, throwscorrupt Index Excepton,Lock Obtan Faled Excepton,IOExcepton f(index Wrter.s Locked(drectory)){ Index Wrter.unlock( drectory) ;}//Lockedfle ndex Wrter = new Index Wrter(drecto ry wconfg); return ndex Wrter;//return Indexwrter //creat e Index Wrter object; } }; It can be seen from the above program code that whether the fle lock exsts n ndex fle needs to be judged at sngle case Index wrter. If such stuaton can be found, t shows that the effectve retreved nformaton may not appear n abnormal stuatons. Conversely, prevous addton of ndex fle can avod the possblty of abnormal stuatons. At ths tme, the ndex fle that has not been unlocked may exert mpact on fle operaton, and thus the sngle case unlockng wll defntely have an adverse effect. Therefore, when t s regarded as the judgment object, the rate of progress between the search can be accelerated, thus obtanng the unlockng mplementaton operaton effect of ndex operaton. In short, the abnormal fle legacy amng at the unlockng operaton s formed n the techncal ndex, thereby mprovng retreval effcency. 4.2 Man ndex local optmzaton mergng strategy of mult-ndex fle storage Currently, Lucene only provdes a sngle ndex fle mergng strategy, but does not menton the mergng strategy of mult-fle storage ndex. In the process of ncreasng data volume, t s obvous that sngle ndex fle cannot quckly satsfy the search needs of large data volume and the ntroducton of mult-fle ndex storage method s a necessary way to deal wth the ncrease n data volume. However, after the ntroducton of mult-fle ndex, the ncrement and mergng operaton of the ndex are more complex. At the same tme, the soluton of ndex fragmentaton and real-tme search have to make numeral changes, so the local optmzaton scheme s more effectve. The local optmzaton mergng s the optmzaton model to obtan local mergng n the man ndex, and such method has not been provded n Lucene n the prevous desgn. Therefore, n order to acheve local mergng effect, t s necessary to adopt a completely new soluton. It can be acheved by the method of sub-fle nternal optmzaton and gradually addng to the man ndex fle and ts partal mergng effect has also acheved the expected effect at ths tme (Ba and Geng, 2016). The man steps of the method are as follows: Frstly, the content of the document set s extracted and analyzed to output the text data stream and at the same tme, the nformaton retreval s conducted n Key1, Key2, Key3. Secondly, all the key words n the data stream are respectvely nserted nto the correspondng sequence table so as to clarfy the vocabulary proporton and reference category of the retreved tem. After that, the processed ndex n the memory s uploaded to the correspondng fle n the dsk ndex to support the real-tme dockng of secondary retreval nformaton. Fnally, the sequence algnment s conducted through the sub-fles of the dsk ndex fle so that the output effect of nternal mergng can be optmzed based on the set strategy. In addton, the addton of project condtons need to be conducted for the optmzed dsk ndex fle so as to support the correspondng functon mplementaton n the man ndex set. The step-by-step elaboraton can be used to depct the mergng optmzaton of man ndex set n ths lnk. Through the step-by-step operaton of ths process, the managng of all search nformaton needed can be completed, whch s also the necessary measure to optmze the tmelness of Lucene search 397

6 engne n current stage. 5. CONCLUSION In summary, the acceleraton analyss of sngle ndex fle ndex s used to mprove the effcency of data nformaton retreval and reduce the adverse effect of repeated appearance of mult phrases. The man ndex local optmzaton mergng strategy adoptng mult ndex fle storage s also a drect way to mprove the effectveness of Lucene search engne. Although ths research studed the mproved desgn model for the above two groups, but they have not been gone through a wde range of applcaton effect comparson. Therefore, t s necessary to further ncrease the data volume n the applcaton process so as to mprove the applcaton effect of ths model. REFERENCE Ba K., Geng G.H. (2016). Research and applcaton of vertcal search engne based on Lucene/Hertrx, Computer Applcaton and Software, 26 (01), Fu J.S., Xu L.L., Ln W.B. (2015). Research and mplementaton of dstrbuted full-web job search engne, Computer Technology and Development, 25 (05), 6-9. Guo Q., Cao J.K. (2013). Research and mplementaton of lterature retreval system of non-ferrous metallurgcal nformaton servce platform based on Lucene.net, Journal of Jangx Unversty of Technology, 34 (01), Hu Q., Lu P.Z. (2011). Usng Solr to buld a search engne for bg database, Software, 32 (06), L Y.C and Geng X.T. (2010). The research and applcaton of the full-text search of Lucene, Computer Technology and Development, 20 (02), Ma Y., Ca B. (2015). Research on the performance optmzaton of Lucene n bg data envronment, Journal of Nanjng Unversty of Technology, 39 (03), Ren S.A., Zhang S.H. (2014). Analyss and optmzaton of LUCENE search algorthms, Lbrary Journal, 33 (12), Sun X.F., Hong Y.R. (2013). Desgn of data persstence structure of an ERP system, Mcroelectroncs and Computer, 30 (04), Wang Y.B., L J.K. (2014). Data mnng of commercal banks n the bg data era, Fnancal Electroncs, (01), Wu W.Z., Y P. (2012). Applcaton of MapReduce n dstrbuted search engne, Computer System Applcaton, 21 (02), Wu Y.B., Da W., Ma Y.H. (2016). Research on network publc opnon management and gudance n the era of bg data, Informaton Securty Research, 2 (04), Y T.P., Chen Q. (2014). Based on Lucene's Chnese analyzers, comparatve study, Computer Engneerng, 38 (22), Zhang S., Hu J.J. (2015). The concept ndex of support for large-scale text retreval n MapReduce envronment, Computer Engneerng, 41 (07), Zhang W.B., Sh W.H., L G.R. (2015). A full-text search engne Lucene system model and applcaton research, Software Gude, 14 (06), Zheng B.Y., Ln S.P. (2010). Based on Lucene's Chnese nverted ndexng technology, Computer Technology and Development, 20 (03),