A Novel Fuzzy Chinese Address Matching Engine Based on Full-text Search Technology

Transcription

1 Based on Full-text Search Technology 12 Institute of Reote Sensing and Digital Earth National Engineering Research Center for Reote Sensing Applications,Beijing,100101, China E-ail: Xiang Li Institute of Reote Sensing and Digital Earth, Beijing,100101, China E-ail: Ling Peng Institute of Reote Sensing and Digital Earth National Engineering Research Center for Reote Sensing Applications,Beijing,100101, China E-ail: Tianhe Chi 3 Institute of Reote Sensing and Digital Earth National Engineering Research Center for Reote Sensing Applications,Beijing,100101, China E-ail: chith@126.co The ability to locate addresses is one of the ost iportant features in an urban geographic inforation syste. Since Chinese geocoding proble cannot be handled by the European geocoding ethod, soe Chinese scholars did special researches on Chinese geocoding. Current researches all focus on address standardizations and odels, and pay less attention to the user input and result control. We designed a novel fuzzy Chinese address atching engine to give a freedo of user input and result control. The engine is coposed of an index builder and a retrieval locator based on full-text search. Furtherore, three kinds of fuzzy atch ethods (Searching-Box Fuzzy Single Match, One-to-One Fuzzy Single Match, and Table-For Fuzzy Batch Match) are ipleented. Through ore than 50,000 pieces of address-point data fro eight districts of Beijing for testing, this engine copared to traditional database retrieval syste shows obvious advantages: (1)It has higher atch efficiency for dealing with large data; (2)It offers greater freedo on user input and result control. In the quality test, when the threshold of fuzzy atching degree is higher than 0.75, the accuracy rate reaches to 100%, with atching rate 83% and recall rate 92% respectively. CENet Septeber 2015 Shanghai, China 1 Speaker 2 Corresponding Author 3 This research was supported by the fire control anageent and rescue coand syste based on Internet of things (2011FU125Z24). Copyright owned by the author(s) under the ters of the Creative Coons Attribution-NonCoercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0).

2 1. Introduction The ability to locate addresses is one of the ost iportant features in an urban geographic inforation syste. Geocoding, also called address atch, which uses a geographical description to ap a space position with X and Y coordinates [1], is the key to solve address location proble. European countries start researches on geocoding earlier than China. For exaple, the project of Topologically Integrated Geographic Encoding and Referencing (TIGER) in Aerica [2], the Postal Code Address Data (PCAD) in Canada [3], the Geocoded National Address File (GNAF) in Australia [4] all belong to this category. Besides, lots of coercial organizations such as Google, Yahoo, Microsoft and ESRI, also released their own free geocoding engines for the governent work, city planning and facility location. However, the European geocoding technology cannot be used in China directly because of the reasons: (1) Chinese address standardizations and geocoding database lack integrated planning. For exaple, in 1988, Beijing firstly proposed a series of urban address data collecting rules [5-6] in the Geocoding Project for Beijing. Later, Shanghai, Zhejiang and Hong Kong also did siilar work, and published asses of different standardizations. So far, there still exists an insurountable barrier to build unifor standards about geocoding. (2) Copared to English address expression, Chinese expression has ore freedo and no space. European geocoding engine cannot handle Chinese geocoding proble directly. Therefore any Chinese scholars do researches on Chinese geocoding and present soe practical odels, such as Hierarchical Address Match Models [7, 8], Inforation Retrieval Address Match Models [9] and Huan Intelligence Address Match Models [10]. Nevertheless, current researches all focus on address standardizations and odels, and pay less attention to the user input and result control. On one hand, users always input address ixed with un-noral ters. Liited result is returned if the address construction is overpursued. On the other hand, different volue of input data require different atch accuracy, efficiency and operation styles. In order to solve the above probles, this paper proposes a novel fuzzy Chinese address atching engine coposed of an index builder and a retrieval locator based on full-text search. This engine has three atching odes: Searching-Box Fuzzy Single Match, One-to-One Fuzzy Single Match, and Table-For Fuzzy Batch Match. Since this engine offers siple address atching coponents and gives a factor to adjust the results, it can be easily used to develop coplex geocoding services. The paper is organized as follows. The core algorith of full-text search technology - Correlation Score Algorith is presented firstly in Sect. 2. Section 3 discusses the fraework of fuzzy Chinese address atching engine. We propose the ipleent of the index builder and the retrieval locator. The retrieval locator contains three fuzzy atching functions, which is the ost iport part of the paper. Section 4 presents a series of experients to deonstrate the efficiency and the accuracy of the proposed function. Finally, the study is suarized in Sect Correlation Score Algorith based on Vector Space Model (VSM) Full-text search engine creates index and records the frequency and location for each word in a file, so it s uch faster to pinpoint the user input. Chinese full-text search technology includes invert index, word segentation, inforation retrieval and etc. It s ainly used to atch key word in the website. It s also apt to atch address due to the siilar process. This paper selects Lucene engine to custoize our address atching engine because of its good extendibility and concurrency support. The kernel of Lucene is the correlation score echanis. It calculates the correlation score between the input query and the docuent. The algorith adds adjustable stiulating factors based on VSM [11], which can enhance the flexibility of the atching algorith. The forula is below: 2

3 Score( q, d ) coord( q, d) querynor( q) 2 [ tf ( k in d) idf ( k) k. getboost() nor( k, d)] k in q (2.1) In Eq. (2.1), k eans key word in query q or docuent d. coord( q, d ) stands for the ratio between the atched key words in q and the key words in q; querynor( q ) stands for the noralized factor of q. The longer the query is (it eans q containing ore key words), the saller the factor value is. tf ( k in d ) stands for the frequency of k in d. idf ( k ) stands for the frequency of inversed docuents. The less likely the key word k appears in the whole database, the bigger the value is. k. getboost () stands for the weight of the key word k. nor( k, d ) stands for the noralized factor of d. The longer the docuent is (it eans d containing ore key words), the saller the value is. In conclusion, if a key word appears ore frequent in the active docuent than in others, the score of the key word will be uch higher to the active docuent than to others. In addition, if the atched key words of a query sentence hold a higher percentage in the active docuent than in others, the score of the query will be uch higher as well. Based on the forula, each address in the database can be treated as a docuent. Then after the word segentation and invert indexed process, the above algorith can be used to calculate the atching degree between the user input and each address piece fro the database. This forula is also the theoretical foundation of the fuzzy Chinese address atching engine. 3. The Fraework of Fuzzy Chinese Address Matching Engine The fuzzy Chinese address atching engine has two ain threads: (1) The creation of geography index; (2) Geography searching and location. The fraework is as Fig. 1. Figure 1: The fraework of fuzzy Chinese address atching engine 3.1 Creation of Geography Index Geography index is created for the whole address database. It puts a piece of dictionary clothes on the address database, so it avoids catastrophic traversals during atching. The creation of geography index happens when updating the address database. The whole process is divided into 3 steps: 1. Preprocess the address database. We first filter the non-standard addresses in the original database and noralize the, then encode the addresses according to their hierarchal eleents. After that, the noralized address can by expressed by a series of codes. In order to iprove the atching rate, each noralized address record is supplied with alias, English abbreviation and etc. The ite which can be located should contain fields including address code, address nae, location description (such as standard address and adinistrative area), 3

4 reark (such as abbreviation and alias) and location coordinates (X, Y coordinates). Accordingly, the geographic coordinates table, the adinistrative area table, the street table, the building table are built. Aong the, the geographic coordinates table is the only table storing coordinates and the copleted related codes, so other tables can be connected logically. 2. Create the word segentation dictionary. The address segentation is siilar to the ordinary sentence segentation, but it needs to think about the characters of the address coposition. First, ost of the addresses can be treated as the cobination of different hierarchical place naes, so an address can be expressed as follows: address = place nae 1 + place nae place nae 3. For exaple, a Chinese address expression Beijing City Chaoyang District jianguoenwai Street 1 No. can be divided into Beijing City, Chaoyang District, Jiangguoenwai Street, No. 1. When the adinistrative level goes down, the location specificity increases. This kind of specificity can be evaluated by their occurring probability in the database, so it s necessary to identify these independent place nae units. Second, any studies about address atch shows that, a noralized hierarchical place nae usually consists of coon nae and special nae. For exaple, Beijing City consist of one special nae Beijing and one coon nae City. The special nae can be accurately described as a geography object, serving as the ain atching word. The coon nae can ark single work unit, serving as the cutting word. Using the standard address as an exaple, the best cutting should be Beijing/ City/ Chaoyang/ District/ Jianguoenwai/ Street/ 1/ No.. According to this feature, in order to increase the segentation accuracy of coon nae and special nae, it s needed to put the coon geography segentation words, as well as adinistrative division naes, counity naes, and street naes into the word segentation dictionary. Coon geography segentation words are as follows: Adinistrative area: Province, city, autonoous region, special adinistrative region, district, alliance, autonoous prefecture, village, town, sub-district office etc. Counity: li, plot, park, lane, apartent, house, garden, cottage etc. Street: road, avenue, street, alley, hutong etc. Nuber: nuber, #, building, doritory, roo, hall etc. Geographic object: ansion, square, center, bar, restaurant, hotel, bureau, copany etc. Spatial direction: east, west, south, north, copound direction(such as southeast, northwest) Spatial relation: contain, disjoint, joint, overlay, cross Word segentation dictionary is not only for the creation of the geography index, but also for the user input segentation. Spatial orientation and spatial relation segentation words can provide the correction inforation of a location expression. They ake the result closer to the real geography location description. For exaple, No.1, Jianguoenwai Street, Chaoyang District, Beijing City, 10 east is not a noral location, because there s an offset description 10 east in the sentence. This offset description can be segregated out and used to calculate the real geography location. 3. Build the geography indexer. First, a Docuent object d is created, and the address data are written into the IndexWriter of d. Second, the fields of address code, address nae, location description, reark are selected as geography index fields. An appropriate Chinese tokenizer (for exaple, Paoding Tokenizer), along with the word segentation dictionary, are used to build Inverted Index. At last, the grouping index, caching index and increental index of Lucene are used to optiize the index. 3.2 Geography Searching and Location In practical applications, fuzzy atch is ore widely used than exact atch. Fuzzy atch includes three odes: Searching-Box Fuzzy Single Match, One-to-One Fuzzy Single Match, and Table-For Fuzzy Batch Match. Searching-Box Fuzzy Single Match eans finding the siilar result set fro the database after input a query sentence. The result eleents rank in descending order according to the 4

5 correlation score. Usually, users have the right to re-choose. The first eleent in result set has the highest score, which is called the ost relevant solution. One-to One Fuzzy Single Match eans finding the target object fro the database after input a query sentence. However, just using the ost relevant solution as the result of One-to-One Fuzzy Singe Match is iproper because it ay not be the target solution. For exaple, if a user intend to find Guangtai Anial Pharaceutical Copany by inputting Guangtai Pharaceutical Copany, the ost relevant solution is Guangtai Biology Pharaceutical Copany. Either uncopleted input or insufficient portion of atched words will cause the disagreeent between the target solution and the ost relevant solution. Under this condition, anual interaction is necessary. Table- For Fuzzy Batch Match is the repetition of One-to-One Fuzzy Single Match. In the process of apping non-spatial table-for data, users pay ore attention to how any target solutions can be returned fro the whole table-for data, rather than how any relevant solutions can be returned fro one record of the data The Algorith of Searching-Box Fuzzy Single Match The query fields are the whole geography indexed fields or soe of the, and the returns are the top n results with the ost highest correlation scores. The atching process uses the MultiFieldQueryParser index analyzer to do ulti-field fuzzy atch. The algorith is below. Algorith 1. The Algorith of Searching-Box Fuzzy Single Match (fuzzysinglematch) Input: - fields denotes the query fields - q denotes the query sentence - n denotes for the nuber of results - T Chinese denotes a Chinese Tokenizer Output: - the top n results with the ost highest correlation scores R = { r1, r2,... r n } 1: R 2:Create a MultiFieldQueryParser P assigned TChinese and fields paraeters 3: Use PT, fields to parse q and get a Query Q 4: Create an IndexSearcher SQ to do the query task 5: Fetch the top n Score Docuents by SQ and store the in DscoreDocs = { d1, d2,... dn} 6: for each d in D scoredocs do 7: R = R U{the index result of d} 8: end for 9: return R The Algorith of One-to-One Fuzzy Single Match One-to-One Fuzzy Single Match has three possible situations. First, the engine finds out the target solution, which eans a successful atch. Second, the query fetches several results when the query sentence is uncopleted. The ost relevant solution ay not be the target solution, so anual operation is needed. Third, the engine cannot find the target solution, which eans a failed atch. We need to judge the copleteness of the input firstly to identify the second situation, and judge the threshold to distinguish the first and the third situation. The second step needs a judgent factor. Lucene will ake the score divided by the axiu score if it s greater than 1. Although the noralized factor of query querynor( q ) and the noralized factor of docuent nor( k, d ) are introduced when calculating scores, it akes no sense to copare the scores of the ost relevant solutions fro different users inputs. To solve this proble, we introduce a factor called Fuzzy Matching Degree ( ) based on the correlation score algorith. The forula is as follows: = S1 / S (3.2) 2 5

6 In Eq.(3.2), S 1 denotes the score of the ost relevant solution, S 2 denotes the exact atch score when the current ost relevant solution is re-calculated. ranges fro 0 to 1. r 1 eans a successful atch( r is an epirical threshold given by user). Especially, = 1 eans an exact atch. The algorith (Algorith 2, containing 26 lines) contains two key steps: Keyword strea copleteness detection( line 3-10) After segent the query sentence into key words, we put each key word into BooleanQuery index analyzer assigned MUST paraeter to get the IndexResults including all keywords. If the size is ore than 1, then it needs the user interaction; otherwise it passes the detection and goes to the threshold detection. Keyword strea threshold detection(line 11-28) If the result set passes the copleteness detection, we put each keyword into BooleanQuery index analyzer assigned SHOULD paraeter to get the IndexResults including at least 1 keyword. If the size of the results set is ore than 0 and the of the ost relevant solution is larger than r, then it passes the detection; else if the size is equal to 0, or the size is ore than 0 with the of the ost relevant solution less or equal to r, it fails. Algorith 2. The Algorith of One-to-One Fuzzy Single Match(oneToOneFuzzySingleMatch) Input: - fields denotes the query fields - q denotes the query sentence -r denotes the threshold of - T Chinese denotes a Chinese Tokenizer Output: - the target solution object Ot arg et( rt arg et, f ), where r t arg et indicates the target index result, f indicates the atching situation flag, f = 0 indicates for the third situation, f = 1 indicates for the second, and f = 2 indicates for the first. 1: Ot arget( rt arg et, f ) 2: Create a QueryParser P assigned TChinese and fields paraeters 3: Use PT, fields to parse q and get a BooleanQuery Q 4: Q assigned MUST paraeter for keyword strea copleteness detection 5: Create an IndexSearcher S Q to do the query task 6: Fetch the top 2 Score Docuents by SQ and store the results in set D 1 % In order to save the processing tie, only get the top 2 results% 7: if size( D 1 ) > 1 then 8: Ot arg et. f 1 9: D1 fuzzysinglematch( fields, q,10) 10: Ot arg et. rt arg et the selected node fro D 1 11: else 12: Q assigned SHOULD paraeter for keyword strea threshold detection 13: Fetch the top 2 Score Docuents by SQ and store the results in set D 2 % In order to save the processing tie, only get the top 2 results% 14: if size( D 2 ) = 0 then 15: Ot arg et. rt arg et 16: Ot arg et. f 0 17: else 18: = get(the first node of D2) 19: if > r then 20: Ot arg et. rt arg et the first node of D2 21: Ot arg et. f 2 6

7 22: else 23: t arg et. t arg et 24: O. 0 t arg et f 25: end if 26: end if 27: end if 28: return arg arg O r O ( r, f ) t et t et The Algorith of Table-For Fuzzy Batch Match Rather than regurgitating, Table-For Fuzzy Batch Match is an iteration process of Oneto-One Fuzzy Single Match. 4. Results and Analysis The experient data is used to test our ethods consists of pieces of address-point data fro eight districts of Beijing. We considered the efficiency of the engine rather than the errors caused by the incopleteness of data. Efficiency test and analysis In this experient, we repeated each algorith 5 ties and took the average value. The result is below. tie(t/s) DataVolue(illion) One-to-One Fuzzy Single Match Searching-Box Fuzzy Single Match Database SQL search Table 1: The coparison of our ethods and SQL searching test We can conclude fro the table above that, the atching engine proposed in the paper has obvious higher efficiency than the coon SQL data syste. The reversed index avoids a lot of traversals, so the effect of the basic data aount can be ignored. This advantage is ore rearkable in a batch atch. Quality test and analysis We get a piece of enterprise table-sheet fro governent data syste containing about 3000 ites with enterprise nae and address description fields. We repeated the fuzzy batch atch ethod 5 ties and took the average value. Fig. 1.2 shows the accuracy rate, recall rate and atching rate changes when assigning different threshold. Ration % 95.00% 90.00% Trend chart of evaluation index 0.2, , , , , , , , , , , , % 80.00% 0.2, , , , , % 0.9, , % 0.9, μ threshold(r) 1 Accuracy Rate Matching Rate Recall Rate Figure 2: Evaluation index trend chart of fuzzy batch atch 7

8 There s a negative correlation between accuracy rate and recall rate. With the increasing of r ( threshold), the recall rate and atching rate are decreasing. In practical application, accuracy rate is relatively ore iportant, so we should first guarantee the accuracy rate, then the recall rate and atching rate. In Fig. 1.2, we know that when r is higher than 0.75, the accuracy rate reaches to 100%, with atching rate 83% and recall rate 92% respectively. The threshold eets the needs of practical application. We can also adjust the threshold according to different situations. 5 Conclusion This paper proposes a novel Chinese fuzzy address atching engine based on full-text search with three kinds of fuzzy atch ethods: Searching-Box Fuzzy Single Match, One-to- One Fuzzy Single Match, and Table-For Fuzzy Batch Match. The core idea of this engine is: by creating index for the address segents, we get the appearance frequency of each segent in the address database, so there s no need to analyse the hierarchy roles of the address eleent during atch. In addition, we creates a factor called to control the accuracy rate and atching rate for Fuzzy Batch Match. The result shows that these algoriths, copared to the traditional ethods, have higher atch efficiency and give higher degree of freedo to user input. Chinese address atch is a coplex proble. The atch engine in this paper focuses on the flexibility of the user input and result control. It lacks the consideration of error tolerance, such as house nuber atch, abbreviation, input istake, which will be iproved in our future work. References [1] T. H. Grayson,Address atching and geocoding[r]. Massachusetts Institute of Technology Departent of Urban Studies and Planning (2000). [2] U.S. Census Bureau TIGER/ Line Shapefiles Technical Docuentation[EB/OL]. aps-data /data/ pdfs/tiger /tgrshp2013/tgrshp2013_techdoc.pdf (2013). [3] Canada Postal Guide - Addressing Guidelines[EB/OL] [4] PSMA Australia Liited G-NAF Product Description[EB/OL]. product=g-naf (2012). [5] China's state bureau of surveying and apping[s]. Coding Rules for urban geographical features - City roads road intersections blocks and unicipal pipe lines (GB/T ). Standards Press of China, Beijing (2009). [6] China's state bureau of surveying and apping[s]. Rules of coding for address in the coon platfor for geospatial inforation service of digital city (GB/T ). Standards Press of China, Beijing (2009). [7] H. Yu, Q. Qi, Y. Li, Study on city address geocoding odel based on street[j]. Journal of Geoinforation Science. 15(2), (2013). [8] L. Zhang, S. Wu, Research on place naes and address segentation in geocoding syste[j]. Science of Surveying and Mapping. 35(2), (2010)(In Chinese). [9] C. Zou, G. Zhu, S. Zhao, Research on custoized query of geographic nae and address based on search engine[j]. Bulletin of Surveying and Mapping. (8), 92-94,124 (2014)(In Chinese). 8

9 [10] Z. Song, Address atching algorith based on Chinese natural language understanding[j]. Journal of Reote Sensing. 17(4), (2013). [11] B. Possas, N. Ziviani, W. Meira, B. Ribeiro-Neto, Setbased vector odel: An efficient approach for correlation based ranking[j]. ACM Transactions on Inforation Systes. 23(4), (2005). 9