Project Report on winter

Transcription

1 Project Report on winter Yaxin Li, Xiaofeng Liu October 17, 2017 Li, Liu October 17, / 31

2 Outline Introduction a Basic Search Engine with Improvements Features PageRank Classification Clustering Word2vec & Doc2vec Li, Liu October 17, / 31

3 Introduction Project Summary Developed a basic search engine mainly oriented to academic papers search Basic functions: information arrangement, query processing Improvements: fast search, search for different attributes, more datasets etc. Added PageRank, Naive Bayes classifier, LSI, K-Means, HAC, word2vec, doc2vec in our project Software Configuration Eclipse Neo Release(4.6.0) Java packages Lucene Dom4j La4j Tomcat 8.0 RStudio Latex Li, Liu October 17, / 31

4 Basic Search Engine Data Source How to build the index How to search the index Search Engine: backend: Tomcat + Servlet frontend: JSP + HTML + CSS Li, Liu October 17, / 31

5 Basic Search Engine - Data Sources CiteSeer full text & meta data.txt files ranged from xml files - meta data SIGMOD ICSE VLDB citaion graph & meta data papers, edges Li, Liu October 17, / 31

6 Basic Search Engine - Organization Li, Liu October 17, / 31

7 Basic Search Engine - Build Index Data Scource docid, title, authors, year, conference, fulltext, citation, simdocs Create a writer Directory dir = FSDirectory.open(Paths.get(indexPath)); StandardAnalyzer analyzer = new StandardAnalyzer(); analyzer.setversion(version.lucene 6 3 0); IndexWriterConfig iwc = new IndexWriterConfig(analyzer); iwc.setopenmode(openmode.create OR APPEND); ndexwriter writer = new IndexWriter(dir, iwc); Li, Liu October 17, / 31

8 Basic Search Engine - Build Index Cont. Create a new document Document doc = new Document(); doc.add(new Field("title",PTitle,TextField.TYPE STORED)); doc.add(new Field("pages",pages==null?"":pages,TextField.TYPE STORED));... add a document to the index writer.adddocument(doc); write the documents into the index writer.forcemerge(1); writer.close(); Li, Liu October 17, / 31

9 Basic Search Engine - Search Index Create a searcher Directory dir = FSDirectory.open(Paths.get(indexPath)); DirectoryReader ireader = DirectoryReader.open(dir); IndexSearcher isearcher = new IndexSearcher(ireader); Create a query QueryParser parser = new QueryParser("fulltext", analyzer); String q = "entropy"; Query query = parser.parse(q); Get the Results ScoreDoc[] hits = isearcher.search(query, ).scoredocs; for(int i = 0; i < hits.length; i ++ ){ ireader.document(hits[i].doc).get("id"); ireader.document(hits[i].doc).get("title")); } Li, Liu October 17, / 31

10 Basic Search Engine - Improvements Keywords Highlights (Context Around Keywords) Searching for Different Attributes Searching for Phrases Speed Up Searching Li, Liu October 17, / 31

11 Basic Search Engine - Improvements Cont. Highlight Keywords ScoreDoc[] hits = searcher.search(query, ).scoreDocs; SimpleHTMLFormatter htmlformatter = new SimpleHTMLFormatter("<span><b>","</b></span>"); SimpleFragmenter fragmenter = new SimpleFragmenter(); fragmenter.setfragmentsize(100); Highlighter highlighter = new Highlighter(htmlFormatter, new QueryScorer(query)); highlighter.settextfragmenter(fragmenter); Speed up searching Ranking & Return Limit (Return top 50 results at most for each query. ) Pagination(Get only 10 results for every searching) Optimize the Index Li, Liu October 17, / 31

12 Features PageRank Naïve Bayes Classification Latent Semantic Indexing Clustering K-Means HAC Li, Liu October 17, / 31

13 Features - PageRank Data Source ICSE VLDB SIGMOD citation graph PageRank Calculation Spider trap and dead end Sort papers using PageRank values or Combine PageRank with lucene similarity score, and sorting papers by the combined scores Li, Liu October 17, / 31

14 Sort the Papers with PageRank Changes in Index Building doc.add(new DoubleDocValuesField("pagerank",pagerank)); Changes in Searching //create a sort criterion SortedNumericSortField sf = new SortedNumericSortField("pagerank",SortField.Type.DOUBLE,true); Sort sort = new Sort(sf); //search the index according to the sorting criterion ScoreDoc[] hits = isearcher.search(query, 10000,sort).scoreDocs; Li, Liu October 17, / 31

15 Sort the Papers with Combined Value When searching an index, Lucene gives a similarity score for each returning document according to the text relevance we can also rank the documents by combining the text relevance and link relevance Field titlef = new Field("title",title,TextField.TYPE STORED); titlef.setboost((float) pagerank); doc.add(titlef); Li, Liu October 17, / 31

16 Features - PageRank Combine PageRank with Search Engine Li, Liu October 17, / 31

17 Features - Naïve Bayes Classification Data Sources : ICSE and VLDB Definition of Terms : Bigrams & Unigrams Feature Selection Mutual Information Feature Size: 10, 100, 1000, 10,000, 100,000 χ 2 Feature Selection set p value to 0.01, 0.05, 0.1 and 0.5 Figure: Bi-chi2 Figure: Uni-chi2 Figure: Bi-MI Figure: Uni-MI Li, Liu October 17, / 31

18 Evaluation of Classification 10-fold Cross Validation Results of Each Experiment Table: χ 2 p value <0.01 <0.05 <0.1 <0.5 uni-chi2-norm bi-chi2-norm uni-chi2-norm-rmsw bi-chi2-norm-rmsw Table: Mutual Information FeatureNum , ,000 uni-chi2-norm bi-chi2-norm uni-chi2-norm-rmsw bi-chi2-norm-rmsw Li, Liu October 17, / 31

19 Evaluation of Classification Cont Evaluations of Classification CHI2 Evaluations of Classification Mutual Information F1 Value type bigram chi2 norm bigram chi2 norm+rmsw unigram chi2 norm F1 Value type bigram mi norm bigram mi norm+rms unigram mi norm unigram chi2 norm+rmsw unigram mi norm+rm <0.01 <0.05 <0.1 <0.5 P value Number of Features Li, Liu October 17, / 31

20 Apply Classification on Website Li, Liu October 17, / 31

21 Features - LSI Data Source ICSE + VLDB 1653(words) 997(documents) tf-idf matrix SVD Clustering Li, Liu October 17, / 31

22 LSI - SVD SVD calculation with la4j.jar U: SVD term matrix D: Singular matrix V: SVD doc matrix D k V for clustering the documents k is the first k biggest singular values Li, Liu October 17, / 31

23 D k V, when k = 100 Li, Liu October 17, / 31

24 Features - Clustering K-Means distance measurement: Euclidean distance Normalized Euclidean distance, which is the same as cosine similarity > 10,000 times of calculation Hierarchical Clustering distance measurement: cosine similarity method: single link, complete link, centroid link Table: summary Data Source Method Distance Purity HAC-single Cosine similarity HAC-centroid Cosine similarity Vevtors from SVD HAC-complete Cosine similarity K-Means Euclidean K-Means Normalized Euclidean Vectors from doc2vec K-Means Normalized Euclidean Li, Liu October 17, / 31

25 Hierarchical Clustering data source: the vector from SVD distance: cosine similarity single link - purity: Li, Liu October 17, / 31

26 Hierarchical Clustering data source: the vector from SVD distance: cosine similarity centroid - purity: Li, Liu October 17, / 31

27 Hierarchical Clustering data source: the vector from SVD distance: cosine similarity complete - purity: Li, Liu October 17, / 31

28 K-Means Clustering data source: the vector from SVD distance: euclidean similarity purity: Li, Liu October 17, / 31

29 K-Means Clustering data source: the vector from SVD distance: normalized euclidean similarity, which is equal to cosine similarity purity: Li, Liu October 17, / 31

30 K-Means Clustering data source: the vector from doc2vec distance: normalized euclidean similarity purity: Li, Liu October 17, / 31

31 Features - Word2vec & Doc2vec Word2vec similar words have similar vectors recommand similar queries for users train the data with C code on GitHub, and use the vectors by java Doc2vec similar documents have similar vectors paper recommandation implemented with python::gensim Figure: recommand queries Figure: recommand docs Li, Liu October 17, / 31

32 Demo Li, Liu October 17, / 31