Normalizing Social Media Texts by Combining Word Embeddings and Edit Distances in a Random Forest Regressor

Size: px
Start display at page:

Download "Normalizing Social Media Texts by Combining Word Embeddings and Edit Distances in a Random Forest Regressor"

Transcription

1 Normalizing Social Media Texts by Combining Word Embeddings and Edit Distances in a Random Forest Regressor Rob van der Goot r.van.der.goot@rug.nl Rob van der Goot r.van.der.goot@rug.nl / 31

2 1 Problem 2 Error Detection 3 Generation 4 Ranking 5 Conclusion 6 Future Work Rob van der Goot r.van.der.goot@rug.nl / 31

3 Outline 1 Problem 2 Error Detection 3 Generation 4 Ranking 5 Conclusion 6 Future Work Rob van der Goot r.van.der.goot@rug.nl / 31

4 Problem Adapt Natural Language Processing pipelines to noisy (web) data Rob van der Goot / 31

5 Problem Adapt Natural Language Processing pipelines to noisy (web) data Normalize Rob van der Goot / 31

6 Problem Spelling Correction vs. Normalization spell. corr. normalization Rob van der Goot / 31

7 Problem Spelling Correction abilites teh kingdon abilities the kingdom Rob van der Goot / 31

8 Problem Normalization abilites abilities teh the kingdon kingdom doin doing Bham Birmingham 2 to The There ggggrrrreeeeeaaaaaaattttttt great ur your Rob van der Goot r.van.der.goot@rug.nl / 31

9 Problem Traditional spelling correction framework: Error detection Candidate generation Ranking of candidates Rob van der Goot / 31

10 Problem Train set: 2,577 tweets from (Li and Liu 2014) Test set: LexNorm (Han and Baldwin 2011) 549 tweets Rob van der Goot / 31

11 Outline 1 Problem 2 Error Detection 3 Generation 4 Ranking 5 Conclusion 6 Future Work Rob van der Goot r.van.der.goot@rug.nl / 31

12 Error Detection Spelling correction: Dictionary lookup Rob van der Goot / 31

13 Error Detection Often skipped in normalization methods Here as well, because the goal is to be used in a pipeline All tokens are considered to be a possible error/disfluency Recall = 100% But the original word is always kept! Rob van der Goot r.van.der.goot@rug.nl / 31

14 Outline 1 Problem 2 Error Detection 3 Generation 4 Ranking 5 Conclusion 6 Future Work Rob van der Goot r.van.der.goot@rug.nl / 31

15 Generation Spelling correction: Lexical edit distance Phonetic edit distance (Double Metaphone) Rob van der Goot / 31

16 Generation Spelling correction: Lexical edit distance Phonetic edit distance (Double Metaphone) Good results Rob van der Goot / 31

17 Generation Spelling correction: Lexical edit distance Phonetic edit distance (Double Metaphone) Good results So we use an existing system (Aspell) Rob van der Goot / 31

18 Generation Other disfluencies A more data aware model is necessary Rob van der Goot r.van.der.goot@rug.nl / 31

19 Generation Other disfluencies A more data aware model is necessary Semi-supervised Rob van der Goot r.van.der.goot@rug.nl / 31

20 Generation Other disfluencies A more data aware model is necessary Semi-supervised Word Embeddings Rob van der Goot r.van.der.goot@rug.nl / 31

21 Generation Word Embeddings Model taken from (Godin et al. 2015) Trained on 400 million Tweets 3,039,345 words Use cosine distance to find top-n words in vector-space Rob van der Goot / 31

22 Generation Rob van der Goot / 31

23 Outline 1 Problem 2 Error Detection 3 Generation 4 Ranking 5 Conclusion 6 Future Work Rob van der Goot r.van.der.goot@rug.nl / 31

24 Ranking Spelling correction: Combination of edit distances Rob van der Goot / 31

25 Ranking Previous approaches: Ngram based approaches Combine Ranking with generation Rob van der Goot / 31

26 Ranking My approach: Use features from generation Supplement these features with N-Gram features Google Ngrams 1 & Twitter Ngrams 2 Combine all features in a Random Forest Classifier Default parameters Scikit Learn, except for the number of trees = Brants and Franz Herdağdelen 2013 Rob van der Goot r.van.der.goot@rug.nl / 31

27 Ranking Rob van der Goot / 31

28 Ranking Ranking (ablation) Rob van der Goot / 31

29 Ranking Ranking System top1 top3 top10 top20 upper bound (Li and Liu 2012) (Li and Liu 2014) (Li and Liu 2015) Our system Rob van der Goot / 31

30 Outline 1 Problem 2 Error Detection 3 Generation 4 Ranking 5 Conclusion 6 Future Work Rob van der Goot r.van.der.goot@rug.nl / 31

31 Conclusion Overview Aspell spell. corr. My approach normalization Rob van der Goot / 31

32 Conclusion For the normalization task: Word embeddings complement edit distances well A random forest classifier works very well for ranking This is a simple system, with a reasonable performance Rob van der Goot r.van.der.goot@rug.nl / 31

33 Outline 1 Problem 2 Error Detection 3 Generation 4 Ranking 5 Conclusion 6 Future Work Rob van der Goot r.van.der.goot@rug.nl / 31

34 Future Work Multilingual/multiword embeddings Generation (build own language models) Rob van der Goot / 31

35 Future Work Multilingual/multiword embeddings Generation (build own language models) Parameter tuning, add domain specific information Find candidate with: word.* Rob van der Goot / 31

36 Future Work This system was created for use in a pipeline system Rob van der Goot r.van.der.goot@rug.nl / 31

37 Future Work This system was created for use in a pipeline system Parse a word graph based on the output of this normalization Rob van der Goot r.van.der.goot@rug.nl / 31

38 Future Work Rob van der Goot / 31

A Modern spell(1) Abhinav Upadhyay EuroBSDCon 2017, Paris

A Modern spell(1) Abhinav Upadhyay EuroBSDCon 2017, Paris A Modern spell(1) Abhinav Upadhyay EuroBSDCon 2017, Paris Outline Shortcomings in the old spell(1) Feature Requirements of a modern spell(1) Implementation Details of new spell(1)

More information

Normalisation of SMS Text

Normalisation of SMS Text Normalisation of SMS Text Gurpreet Singh Khanuja, Sachin Yadav {gurpreet,sachinky}@iitk.ac.in Advisor : Dr. Amitabh Mukerjee Department of Computer Science and Engineering, IIT Kanpur, India April 18,

More information

Learning Semantic Entity Representations with Knowledge Graph and Deep Neural Networks and its Application to Named Entity Disambiguation

Learning Semantic Entity Representations with Knowledge Graph and Deep Neural Networks and its Application to Named Entity Disambiguation Learning Semantic Entity Representations with Knowledge Graph and Deep Neural Networks and its Application to Named Entity Disambiguation Hongzhao Huang 1 and Larry Heck 2 Computer Science Department,

More information

Information Retrieval

Information Retrieval Introduction to Information Retrieval Lecture 3: Dictionaries and tolerant retrieval 1 Outline Dictionaries Wildcard queries skip Edit distance skip Spelling correction skip Soundex 2 Inverted index Our

More information

USING RECURRENT NEURAL NETWORKS FOR DUPLICATE DETECTION AND ENTITY LINKING

USING RECURRENT NEURAL NETWORKS FOR DUPLICATE DETECTION AND ENTITY LINKING USING RECURRENT NEURAL NETWORKS FOR DUPLICATE DETECTION AND ENTITY LINKING BRUNO MARTINS, RUI SANTOS, RICARDO CUSTÓDIO SEPTEMBER 20 TH, 2016 GOLOCAL WORKSHOP WHAT THIS TALK IS ABOUT Approximate string

More information

S-MART: Novel Tree-based Structured Learning Algorithms Applied to Tweet Entity Linking

S-MART: Novel Tree-based Structured Learning Algorithms Applied to Tweet Entity Linking S-MART: Novel Tree-based Structured Learning Algorithms Applied to Tweet Entity Linking Yi Yang * and Ming-Wei Chang # * Georgia Institute of Technology, Atlanta # Microsoft Research, Redmond Traditional

More information

From Boolean Towards Semantic Retrieval Models. Speakers : Arpan Gupta, Seinjuti Chatterjee

From Boolean Towards Semantic Retrieval Models. Speakers : Arpan Gupta, Seinjuti Chatterjee From Boolean Towards Semantic Retrieval Models Speakers : Arpan Gupta, Seinjuti Chatterjee 1 About us Leading Machine Learning Platform For Ecommerce Search 120+Customers & Brands 1200+ Global Websites

More information

Basic techniques. Text processing; term weighting; vector space model; inverted index; Web Search

Basic techniques. Text processing; term weighting; vector space model; inverted index; Web Search Basic techniques Text processing; term weighting; vector space model; inverted index; Web Search Overview Indexes Query Indexing Ranking Results Application Documents User Information analysis Query processing

More information

Shrey Patel B.E. Computer Engineering, Gujarat Technological University, Ahmedabad, Gujarat, India

Shrey Patel B.E. Computer Engineering, Gujarat Technological University, Ahmedabad, Gujarat, India International Journal of Scientific Research in Computer Science, Engineering and Information Technology 2018 IJSRCSEIT Volume 3 Issue 3 ISSN : 2456-3307 Some Issues in Application of NLP to Intelligent

More information

IN4325 Query refinement. Claudia Hauff (WIS, TU Delft)

IN4325 Query refinement. Claudia Hauff (WIS, TU Delft) IN4325 Query refinement Claudia Hauff (WIS, TU Delft) The big picture Information need Topic the user wants to know more about The essence of IR Query Translation of need into an input for the search engine

More information

Mahout in Action MANNING ROBIN ANIL SEAN OWEN TED DUNNING ELLEN FRIEDMAN. Shelter Island

Mahout in Action MANNING ROBIN ANIL SEAN OWEN TED DUNNING ELLEN FRIEDMAN. Shelter Island Mahout in Action SEAN OWEN ROBIN ANIL TED DUNNING ELLEN FRIEDMAN II MANNING Shelter Island contents preface xvii acknowledgments about this book xx xix about multimedia extras xxiii about the cover illustration

More information

A Large Scale Ranker-Based System for Search Query Spelling Correction

A Large Scale Ranker-Based System for Search Query Spelling Correction A Large Scale Ranker-Based System for Search Query Spelling Correction Jianfeng Gao Microsoft Research Redmond, WA, USA jfgao@microsoft.com Xiaolong Li Microsoft Corporation Redmond, WA, USA xiaolong.li@microsoft.com

More information

Exam IST 441 Spring 2013

Exam IST 441 Spring 2013 Exam IST 441 Spring 2013 Last name: Student ID: First name: I acknowledge and accept the University Policies and the Course Policies on Academic Integrity This 100 point exam determines 30% of your grade.

More information

CSCI 5417 Information Retrieval Systems! What is Information Retrieval?

CSCI 5417 Information Retrieval Systems! What is Information Retrieval? CSCI 5417 Information Retrieval Systems! Lecture 1 8/23/2011 Introduction 1 What is Information Retrieval? Information retrieval is the science of searching for information in documents, searching for

More information

Web Mining TEAM 8. Professor Anita Wasilewska CSE 634 Data Mining

Web Mining TEAM 8. Professor Anita Wasilewska CSE 634 Data Mining Web Mining TEAM 8 Paper - You Are What You Tweet : Analyzing Twitter for Public Health Authors : Paul, Michael J., and Mark Dredze. Conference : AAAI Publications, Fifth International AAAI Conference on

More information

Exam IST 441 Spring 2014

Exam IST 441 Spring 2014 Exam IST 441 Spring 2014 Last name: Student ID: First name: I acknowledge and accept the University Policies and the Course Policies on Academic Integrity This 100 point exam determines 30% of your grade.

More information

Corpus Acquisition from the Interwebs. Christian Buck, University of Edinburgh

Corpus Acquisition from the Interwebs. Christian Buck, University of Edinburgh Corpus Acquisition from the Interwebs Christian Buck, University of Edinburgh There is no data like more data (Bob Mercer, 1985) Mining Bilingual Text "Same text in different languages" Usually: one side

More information

On Identifying Disaster-Related Tweets: Matching-based or Learning based?

On Identifying Disaster-Related Tweets: Matching-based or Learning based? IEEE Big MM 2017, April 19-21, 2017 On Identifying Disaster-Related Tweets: Matching-based or Learning based? Presented by Dr. Seon Ho Kim Hien To Sumeet Agrawal Integrated Media Systems Center University

More information

Exam IST 441 Spring 2011

Exam IST 441 Spring 2011 Exam IST 441 Spring 2011 Last name: Student ID: First name: I acknowledge and accept the University Policies and the Course Policies on Academic Integrity This 100 point exam determines 30% of your grade.

More information

Mining Social Media Users Interest

Mining Social Media Users Interest Mining Social Media Users Interest Presenters: Heng Wang,Man Yuan April, 4 th, 2016 Agenda Introduction to Text Mining Tool & Dataset Data Pre-processing Text Mining on Twitter Summary & Future Improvement

More information

Jianyong Wang Department of Computer Science and Technology Tsinghua University

Jianyong Wang Department of Computer Science and Technology Tsinghua University Jianyong Wang Department of Computer Science and Technology Tsinghua University jianyong@tsinghua.edu.cn Joint work with Wei Shen (Tsinghua), Ping Luo (HP), and Min Wang (HP) Outline Introduction to entity

More information

Introduction to Information Retrieval

Introduction to Information Retrieval Introduction Inverted index Processing Boolean queries Course overview Introduction to Information Retrieval http://informationretrieval.org IIR 1: Boolean Retrieval Hinrich Schütze Institute for Natural

More information

The Role of Language Evolution in Digital Archives

The Role of Language Evolution in Digital Archives The Role of Language Evolution in Digital Archives Thomas Risse, Helge Holzmann (L3S) Nina Tahmasebi, Uni Gothenburg L3S Research Center Web Science @ L3S Preserving, understanding and shaping the Web

More information

A Novel Categorized Search Strategy using Distributional Clustering Neenu Joseph. M 1, Sudheep Elayidom 2

A Novel Categorized Search Strategy using Distributional Clustering Neenu Joseph. M 1, Sudheep Elayidom 2 A Novel Categorized Search Strategy using Distributional Clustering Neenu Joseph. M 1, Sudheep Elayidom 2 1 Student, M.E., (Computer science and Engineering) in M.G University, India, 2 Associate Professor

More information

Markov Chains for Robust Graph-based Commonsense Information Extraction

Markov Chains for Robust Graph-based Commonsense Information Extraction Markov Chains for Robust Graph-based Commonsense Information Extraction N iket Tandon 1,4 Dheera j Ra jagopal 2,4 Gerard de M elo 3 (1) Max Planck Institute for Informatics, Germany (2) NUS, Singapore

More information

Deep learning for object detection. Slides from Svetlana Lazebnik and many others

Deep learning for object detection. Slides from Svetlana Lazebnik and many others Deep learning for object detection Slides from Svetlana Lazebnik and many others Recent developments in object detection 80% PASCAL VOC mean0average0precision0(map) 70% 60% 50% 40% 30% 20% 10% Before deep

More information

Entity and Knowledge Base-oriented Information Retrieval

Entity and Knowledge Base-oriented Information Retrieval Entity and Knowledge Base-oriented Information Retrieval Presenter: Liuqing Li liuqing@vt.edu Digital Library Research Laboratory Virginia Polytechnic Institute and State University Blacksburg, VA 24061

More information

Linking Entities in Chinese Queries to Knowledge Graph

Linking Entities in Chinese Queries to Knowledge Graph Linking Entities in Chinese Queries to Knowledge Graph Jun Li 1, Jinxian Pan 2, Chen Ye 1, Yong Huang 1, Danlu Wen 1, and Zhichun Wang 1(B) 1 Beijing Normal University, Beijing, China zcwang@bnu.edu.cn

More information

Query Difficulty Prediction for Contextual Image Retrieval

Query Difficulty Prediction for Contextual Image Retrieval Query Difficulty Prediction for Contextual Image Retrieval Xing Xing 1, Yi Zhang 1, and Mei Han 2 1 School of Engineering, UC Santa Cruz, Santa Cruz, CA 95064 2 Google Inc., Mountain View, CA 94043 Abstract.

More information

Natural Language Processing as Key Component to Successful Information Products

Natural Language Processing as Key Component to Successful Information Products Natural Language Processing as Key Component to Successful Information Products Yves Schabes Teragram Corporation Tera monster in Greek 2 40 (=1,099,511,627,766) 10 12 (one trillion) gram something written

More information

Python With Data Science

Python With Data Science Course Overview This course covers theoretical and technical aspects of using Python in Applied Data Science projects and Data Logistics use cases. Who Should Attend Data Scientists, Software Developers,

More information

Outline GIZA++ Moses. Demo. Steps Output files. Training pipeline Decoder

Outline GIZA++ Moses. Demo. Steps Output files. Training pipeline Decoder GIZA++ and Moses Outline GIZA++ Steps Output files Moses Training pipeline Decoder Demo GIZA++ A statistical machine translation toolkit used to train IBM Models 1-5 (moses only uses output of IBM Model-1)

More information

DIGIT.B4 Big Data PoC

DIGIT.B4 Big Data PoC DIGIT.B4 Big Data PoC DIGIT 01 Social Media D02.01 PoC Requirements Table of contents 1 Introduction... 5 1.1 Context... 5 1.2 Objective... 5 2 Data SOURCES... 6 2.1 Data sources... 6 2.2 Data fields...

More information

Reading group on Ontologies and NLP:

Reading group on Ontologies and NLP: Reading group on Ontologies and NLP: Machine Learning27th infebruary Automated 2014 1 / 25 Te Reading group on Ontologies and NLP: Machine Learning in Automated Text Categorization, by Fabrizio Sebastianini.

More information

Introduction to Text Mining. Hongning Wang

Introduction to Text Mining. Hongning Wang Introduction to Text Mining Hongning Wang CS@UVa Who Am I? Hongning Wang Assistant professor in CS@UVa since August 2014 Research areas Information retrieval Data mining Machine learning CS@UVa CS6501:

More information

Enhancing Named Entity Recognition in Twitter Messages Using Entity Linking

Enhancing Named Entity Recognition in Twitter Messages Using Entity Linking Enhancing Named Entity Recognition in Twitter Messages Using Entity Linking Ikuya Yamada 1 2 3 Hideaki Takeda 2 Yoshiyasu Takefuji 3 ikuya@ousia.jp takeda@nii.ac.jp takefuji@sfc.keio.ac.jp 1 Studio Ousia,

More information

PERSONALIZED TAG RECOMMENDATION

PERSONALIZED TAG RECOMMENDATION PERSONALIZED TAG RECOMMENDATION Ziyu Guan, Xiaofei He, Jiajun Bu, Qiaozhu Mei, Chun Chen, Can Wang Zhejiang University, China Univ. of Illinois/Univ. of Michigan 1 Booming of Social Tagging Applications

More information

Introduction to Lexical Analysis

Introduction to Lexical Analysis Introduction to Lexical Analysis Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexical analyzers (lexers) Regular

More information

CIRGDISCO at RepLab2012 Filtering Task: A Two-Pass Approach for Company Name Disambiguation in Tweets

CIRGDISCO at RepLab2012 Filtering Task: A Two-Pass Approach for Company Name Disambiguation in Tweets CIRGDISCO at RepLab2012 Filtering Task: A Two-Pass Approach for Company Name Disambiguation in Tweets Arjumand Younus 1,2, Colm O Riordan 1, and Gabriella Pasi 2 1 Computational Intelligence Research Group,

More information

A Survey on Removal of Duplicate Records in Database

A Survey on Removal of Duplicate Records in Database Indian Journal of Science and Technology A Survey on Removal of Duplicate Records in Database M. Karthigha 1* and S. Krishna Anand 2 1 PG Student, School of Computing (CSE), SASTRA University, 613401,

More information

Introduction to Information Retrieval

Introduction to Information Retrieval Introduction to Information Retrieval Mohsen Kamyar چهارمین کارگاه ساالنه آزمایشگاه فناوری و وب بهمن ماه 1391 Outline Outline in classic categorization Information vs. Data Retrieval IR Models Evaluation

More information

Object Detection Based on Deep Learning

Object Detection Based on Deep Learning Object Detection Based on Deep Learning Yurii Pashchenko AI Ukraine 2016, Kharkiv, 2016 Image classification (mostly what you ve seen) http://tutorial.caffe.berkeleyvision.org/caffe-cvpr15-detection.pdf

More information

A data-driven framework for archiving and exploring social media data

A data-driven framework for archiving and exploring social media data A data-driven framework for archiving and exploring social media data Qunying Huang and Chen Xu Yongqi An, 20599957 Oct 18, 2016 Introduction Social media applications are widely deployed in various platforms

More information

RLAT Rapid Language Adaptation Toolkit

RLAT Rapid Language Adaptation Toolkit RLAT Rapid Language Adaptation Toolkit Tim Schlippe May 15, 2012 RLAT Rapid Language Adaptation Toolkit - 2 RLAT Rapid Language Adaptation Toolkit RLAT Rapid Language Adaptation Toolkit - 3 Outline Introduction

More information

Outline. Eg. 1: DBLP. Motivation. Eg. 2: ACM DL Portal. Eg. 2: DBLP. Digital Libraries (DL) often have many errors that negatively affect:

Outline. Eg. 1: DBLP. Motivation. Eg. 2: ACM DL Portal. Eg. 2: DBLP. Digital Libraries (DL) often have many errors that negatively affect: Outline Effective and Scalable Solutions for Mixed and Split Citation Problems in Digital Libraries Dongwon Lee, Byung-Won On Penn State University, USA Jaewoo Kang North Carolina State University, USA

More information

Detecting ads in a machine learning approach

Detecting ads in a machine learning approach Detecting ads in a machine learning approach Di Zhang (zhangdi@stanford.edu) 1. Background There are lots of advertisements over the Internet, who have become one of the major approaches for companies

More information

Information Retrieval

Information Retrieval Information Retrieval Data Processing and Storage Ilya Markov i.markov@uva.nl University of Amsterdam Ilya Markov i.markov@uva.nl Information Retrieval 1 Course overview Offline Data Acquisition Data Processing

More information

by the customer who is going to purchase the product.

by the customer who is going to purchase the product. SURVEY ON WORD ALIGNMENT MODEL IN OPINION MINING R.Monisha 1,D.Mani 2,V.Meenasree 3, S.Navaneetha krishnan 4 SNS College of Technology, Coimbatore. megaladev@gmail.com, meenaveerasamy31@gmail.com. ABSTRACT-

More information

Sifaka: Text Mining Above a Search API

Sifaka: Text Mining Above a Search API Sifaka: Text Mining Above a Search API ABSTRACT Cameron VandenBerg Language Technologies Institute Carnegie Mellon University Pittsburgh, PA 15213, USA cmw2@cs.cmu.edu Text mining and analytics software

More information

Cunning Plan. Informal Sketch of Lexical Analysis. Issues in Lexical Analysis. Specifying Lexers

Cunning Plan. Informal Sketch of Lexical Analysis. Issues in Lexical Analysis. Specifying Lexers Cunning Plan Informal Sketch of Lexical Analysis LA identifies tokens from input string lexer : (char list) (token list) Issues in Lexical Analysis Lookahead Ambiguity Specifying Lexers Regular Expressions

More information

Rich feature hierarchies for accurate object detection and semantic segmentation

Rich feature hierarchies for accurate object detection and semantic segmentation Rich feature hierarchies for accurate object detection and semantic segmentation BY; ROSS GIRSHICK, JEFF DONAHUE, TREVOR DARRELL AND JITENDRA MALIK PRESENTER; MUHAMMAD OSAMA Object detection vs. classification

More information

Understanding the Query: THCIB and THUIS at NTCIR-10 Intent Task. Junjun Wang 2013/4/22

Understanding the Query: THCIB and THUIS at NTCIR-10 Intent Task. Junjun Wang 2013/4/22 Understanding the Query: THCIB and THUIS at NTCIR-10 Intent Task Junjun Wang 2013/4/22 Outline Introduction Related Word System Overview Subtopic Candidate Mining Subtopic Ranking Results and Discussion

More information

Module 3: GATE and Social Media. Part 4. Named entities

Module 3: GATE and Social Media. Part 4. Named entities Module 3: GATE and Social Media Part 4. Named entities The 1995-2018 This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs Licence Named Entity Recognition Texts frequently

More information

Ghent University-IBCN Participation in TAC-KBP 2015 Cold Start Slot Filling task

Ghent University-IBCN Participation in TAC-KBP 2015 Cold Start Slot Filling task Ghent University-IBCN Participation in TAC-KBP 2015 Cold Start Slot Filling task Lucas Sterckx, Thomas Demeester, Johannes Deleu, Chris Develder Ghent University - iminds Gaston Crommenlaan 8 Ghent, Belgium

More information

Representation Learning using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval

Representation Learning using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval Representation Learning using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval Xiaodong Liu 12, Jianfeng Gao 1, Xiaodong He 1 Li Deng 1, Kevin Duh 2, Ye-Yi Wang 1 1

More information

Create an Account... 2 Setting up your account... 2 Send a Tweet... 4 Add Link... 4 Add Photo... 5 Delete a Tweet...

Create an Account... 2 Setting up your account... 2 Send a Tweet... 4 Add Link... 4 Add Photo... 5 Delete a Tweet... Twitter is a social networking site allowing users to post thoughts and ideas in 140 characters or less. http://www.twitter.com Create an Account... 2 Setting up your account... 2 Send a Tweet... 4 Add

More information

Guiding Semi-Supervision with Constraint-Driven Learning

Guiding Semi-Supervision with Constraint-Driven Learning Guiding Semi-Supervision with Constraint-Driven Learning Ming-Wei Chang 1 Lev Ratinov 2 Dan Roth 3 1 Department of Computer Science University of Illinois at Urbana-Champaign Paper presentation by: Drew

More information

Corpus Acquisition from the Internet

Corpus Acquisition from the Internet Corpus Acquisition from the Internet Philipp Koehn partially based on slides from Christian Buck 26 October 2017 Big Data 1 For many language pairs, lots of text available. Text you read in your lifetime

More information

Stanford-UBC at TAC-KBP

Stanford-UBC at TAC-KBP Stanford-UBC at TAC-KBP Eneko Agirre, Angel Chang, Dan Jurafsky, Christopher Manning, Valentin Spitkovsky, Eric Yeh Ixa NLP group, University of the Basque Country NLP group, Stanford University Outline

More information

Semantic image search using queries

Semantic image search using queries Semantic image search using queries Shabaz Basheer Patel, Anand Sampat Department of Electrical Engineering Stanford University CA 94305 shabaz@stanford.edu,asampat@stanford.edu Abstract Previous work,

More information

Microsoft FAST Search Server 2010 for SharePoint for Application Developers Course 10806A; 3 Days, Instructor-led

Microsoft FAST Search Server 2010 for SharePoint for Application Developers Course 10806A; 3 Days, Instructor-led Microsoft FAST Search Server 2010 for SharePoint for Application Developers Course 10806A; 3 Days, Instructor-led Course Description This course is designed to highlight the differentiating features of

More information

Lexical Analysis. Finite Automata

Lexical Analysis. Finite Automata #1 Lexical Analysis Finite Automata Cool Demo? (Part 1 of 2) #2 Cunning Plan Informal Sketch of Lexical Analysis LA identifies tokens from input string lexer : (char list) (token list) Issues in Lexical

More information

Providing link suggestions based on recent user activity

Providing link suggestions based on recent user activity Technical Disclosure Commons Defensive Publications Series May 31, 2018 Providing link suggestions based on recent user activity Jakob Foerster Matthew Sharifi Follow this and additional works at: https://www.tdcommons.org/dpubs_series

More information

CHAPTER 7 CONCLUSION AND FUTURE WORK

CHAPTER 7 CONCLUSION AND FUTURE WORK CHAPTER 7 CONCLUSION AND FUTURE WORK 7.1 Conclusion Data pre-processing is very important in data mining process. Certain data cleaning techniques usually are not applicable to all kinds of data. Deduplication

More information

Recap of last time CS276A Information Retrieval

Recap of last time CS276A Information Retrieval Recap of last time CS276A Information Retrieval Index compression Space estimation Lecture 4 This lecture Tolerant retrieval Wild-card queries Spelling correction Soundex Wild-card queries Wild-card queries:

More information

Real-time Collaborative Filtering Recommender Systems

Real-time Collaborative Filtering Recommender Systems Real-time Collaborative Filtering Recommender Systems Huizhi Liang 1,2 Haoran Du 2 Qing Wang 2 1 Department of Computing and Information Systems, The University of Melbourne, Victoria 3010, Australia Email:

More information

Linking FRBR Entities to LOD through Semantic Matching

Linking FRBR Entities to LOD through Semantic Matching Linking FRBR Entities to through Semantic Matching Naimdjon Takhirov, Fabien Duchateau, Trond Aalberg Department of Computer and Information Science Norwegian University of Science and Technology Theory

More information

Search Engine Architecture II

Search Engine Architecture II Search Engine Architecture II Primary Goals of Search Engines Effectiveness (quality): to retrieve the most relevant set of documents for a query Process text and store text statistics to improve relevance

More information

GUIDE Quickstart Guide. A Guide to Setting Up Your Hootsuite Dashboard

GUIDE Quickstart Guide. A Guide to Setting Up Your Hootsuite Dashboard GUIDE Quickstart Guide A Guide to Setting Up Your Hootsuite Dashboard QuickStart Guide Hootsuite Helps You Organize and Manage Your Social Media Hootsuite is a social media management dashboard. You can

More information

Preventing Unraveling in Social Networks: The Anchored k-core Problem

Preventing Unraveling in Social Networks: The Anchored k-core Problem Preventing Unraveling in Social Networks: Kshipra Bhawalkar 1 Jon Kleinberg 2 1 Tim Roughgarden 1 Aneesh Sharma 3 1 Stanford University 2 Cornell University 3 Twitter, Inc. ICALP 2012 Outline 1 The Problem

More information

Enhancing Automatic Wordnet Construction Using Word Embeddings

Enhancing Automatic Wordnet Construction Using Word Embeddings Enhancing Automatic Wordnet Construction Using Word Embeddings Feras Al Tarouti University of Colorado Colorado Springs 1420 Austin Bluffs Pkwy Colorado Springs, CO 80918, USA faltarou@uccs.edu Jugal Kalita

More information

Keywords. Data Validation, Data Sanitisation, Authentication, Maintainability, Comments, Indentation. Robust Programs. Defensive Design Consideration

Keywords. Data Validation, Data Sanitisation, Authentication, Maintainability, Comments, Indentation. Robust Programs. Defensive Design Consideration Keywords Data Validation, Data Sanitisation, Authentication, Maintainability, Comments, Indentation Robust Programs Defensive Design Consideration Starter Understand the elements of defensive program design.

More information

Hashing with Graphs. Sanjiv Kumar (Google), and Shih Fu Chang (Columbia) June, 2011

Hashing with Graphs. Sanjiv Kumar (Google), and Shih Fu Chang (Columbia) June, 2011 Hashing with Graphs Wei Liu (Columbia Columbia), Jun Wang (IBM IBM), Sanjiv Kumar (Google), and Shih Fu Chang (Columbia) June, 2011 Overview Graph Hashing Outline Anchor Graph Hashing Experiments Conclusions

More information

Sentiment Analysis on Twitter Data using KNN and SVM

Sentiment Analysis on Twitter Data using KNN and SVM Vol. 8, No. 6, 27 Sentiment Analysis on Twitter Data using KNN and SVM Mohammad Rezwanul Huq Dept. of Computer Science and Engineering East West University Dhaka, Bangladesh Ahmad Ali Dept. of Computer

More information

Object detection using Region Proposals (RCNN) Ernest Cheung COMP Presentation

Object detection using Region Proposals (RCNN) Ernest Cheung COMP Presentation Object detection using Region Proposals (RCNN) Ernest Cheung COMP790-125 Presentation 1 2 Problem to solve Object detection Input: Image Output: Bounding box of the object 3 Object detection using CNN

More information

CSCI 5417 Information Retrieval Systems Jim Martin!

CSCI 5417 Information Retrieval Systems Jim Martin! CSCI 5417 Information Retrieval Systems Jim Martin! Lecture 4 9/1/2011 Today Finish up spelling correction Realistic indexing Block merge Single-pass in memory Distributed indexing Next HW details 1 Query

More information

INF5820/INF9820 LANGUAGE TECHNOLOGICAL APPLICATIONS. Jan Tore Lønning, Lecture 8, 12 Oct

INF5820/INF9820 LANGUAGE TECHNOLOGICAL APPLICATIONS. Jan Tore Lønning, Lecture 8, 12 Oct 1 INF5820/INF9820 LANGUAGE TECHNOLOGICAL APPLICATIONS Jan Tore Lønning, Lecture 8, 12 Oct. 2016 jtl@ifi.uio.no Today 2 Preparing bitext Parameter tuning Reranking Some linguistic issues STMT so far 3 We

More information

Fast Algorithms for Top-k Approximate String Matching

Fast Algorithms for Top-k Approximate String Matching Fast Algorithms for Top-k Approximate String Matching Zhenglu Yang # Jianjun Yu Masaru Kitsuregawa # # Institute of Industrial Science, The University of Tokyo, Japan {yangzl, kitsure}@tkliisu-tokyoacjp

More information

Ngram Search Engine with Patterns Combining Token, POS, Chunk and NE Information

Ngram Search Engine with Patterns Combining Token, POS, Chunk and NE Information Ngram Search Engine with Patterns Combining Token, POS, Chunk and NE Information Satoshi Sekine Computer Science Department New York University sekine@cs.nyu.edu Kapil Dalwani Computer Science Department

More information

Using Machine Learning to Identify Security Issues in Open-Source Libraries. Asankhaya Sharma Yaqin Zhou SourceClear

Using Machine Learning to Identify Security Issues in Open-Source Libraries. Asankhaya Sharma Yaqin Zhou SourceClear Using Machine Learning to Identify Security Issues in Open-Source Libraries Asankhaya Sharma Yaqin Zhou SourceClear Outline - Overview of problem space Unidentified security issues How Machine Learning

More information

Realtime & Personalized

Realtime & Personalized Realtime & Personalized Notifications @Twitter @pathak_s @lamgary March 8 2017 I was following it on Twitter, I didn't actually see it live. I kept on refreshing my notifications, I saw people were tweeting

More information

3 Data, Data Mining. Chengkai Li

3 Data, Data Mining. Chengkai Li CSE4334/5334 Data Mining 3 Data, Data Mining Chengkai Li Department of Computer Science and Engineering University of Texas at Arlington Fall 2018 (Slides partly courtesy of Pang-Ning Tan, Michael Steinbach

More information

Comparing Sentiment Engine Performance on Reviews and Tweets

Comparing Sentiment Engine Performance on Reviews and Tweets Comparing Sentiment Engine Performance on Reviews and Tweets Emanuele Di Rosa, PhD CSO, Head of Artificial Intelligence Finsa s.p.a. emanuele.dirosa@finsa.it www.app2check.com www.finsa.it Motivations

More information

Social Bookmarks. Blasting their site with them during the first month of creation Only sending them directly to their site

Social Bookmarks. Blasting their site with them during the first month of creation Only sending them directly to their site Hey guys, what's up? We have another, jammed packed and exciting bonus coming at you today. This one is all about the "Everyone knows Everybody" generation; where everyone is socially connected via the

More information

Random Walk Inference and Learning. Carnegie Mellon University 7/28/2011 EMNLP 2011, Edinburgh, Scotland, UK

Random Walk Inference and Learning. Carnegie Mellon University 7/28/2011 EMNLP 2011, Edinburgh, Scotland, UK Random Walk Inference and Learning in A Large Scale Knowledge Base Ni Lao, Tom Mitchell, William W. Cohen Carnegie Mellon University 2011.7.28 1 Outline Motivation Inference in Knowledge Bases The NELL

More information

3-1. Dictionaries and Tolerant Retrieval. Most slides were adapted from Stanford CS 276 course and University of Munich IR course.

3-1. Dictionaries and Tolerant Retrieval. Most slides were adapted from Stanford CS 276 course and University of Munich IR course. 3-1. Dictionaries and Tolerant Retrieval Most slides were adapted from Stanford CS 276 course and University of Munich IR course. 1 Dictionary data structures for inverted indexes Sec. 3.1 The dictionary

More information

TISA Methodology Threat Intelligence Scoring and Analysis

TISA Methodology Threat Intelligence Scoring and Analysis TISA Methodology Threat Intelligence Scoring and Analysis Contents Introduction 2 Defining the Problem 2 The Use of Machine Learning for Intelligence Analysis 3 TISA Text Analysis and Feature Extraction

More information

real-time delivery architecture

real-time delivery architecture real-time delivery architecture @raffi uc berkeley - 27 august 2012 designing twitter what are the goals? evolve from being solely a web stack ROUTING PRESENTATION LOGIC STORAGE & RETRIEVAL T-Bird T-Flock

More information

Introduction to Information Retrieval (Manning, Raghavan, Schutze)

Introduction to Information Retrieval (Manning, Raghavan, Schutze) Introduction to Information Retrieval (Manning, Raghavan, Schutze) Chapter 3 Dictionaries and Tolerant retrieval Chapter 4 Index construction Chapter 5 Index compression Content Dictionary data structures

More information

Detect Rumors in Microblog Posts Using Propagation Structure via Kernel Learning

Detect Rumors in Microblog Posts Using Propagation Structure via Kernel Learning Detect Rumors in Microblog Posts Using Propagation Structure via Kernel Learning Jing Ma 1, Wei Gao 2*, Kam-Fai Wong 1,3 1 The Chinese University of Hong Kong 2 Victoria University of Wellington, New Zealand

More information

Digital Libraries: Language Technologies

Digital Libraries: Language Technologies Digital Libraries: Language Technologies RAFFAELLA BERNARDI UNIVERSITÀ DEGLI STUDI DI TRENTO P.ZZA VENEZIA, ROOM: 2.05, E-MAIL: BERNARDI@DISI.UNITN.IT Contents 1 Recall: Inverted Index..........................................

More information

Author profiling using stylometric and structural feature groupings

Author profiling using stylometric and structural feature groupings Author profiling using stylometric and structural feature groupings Notebook for PAN at CLEF 2015 Andreas Grivas, Anastasia Krithara, and George Giannakopoulos Institute of Informatics and Telecommunications,

More information

DATA COLLECTION. Slides by WESLEY WILLETT 13 FEB 2014

DATA COLLECTION. Slides by WESLEY WILLETT 13 FEB 2014 DATA COLLECTION Slides by WESLEY WILLETT INFO VISUAL 340 ANALYTICS D 13 FEB 2014 WHERE DOES DATA COME FROM? We tend to think of data as a thing in a database somewhere WHY DO YOU NEED DATA? (HINT: Usually,

More information

Natural Language Processing

Natural Language Processing Natural Language Processing Information Retrieval Potsdam, 14 June 2012 Saeedeh Momtazi Information Systems Group based on the slides of the course book Outline 2 1 Introduction 2 Indexing Block Document

More information

HYDRA Large-scale Social Identity Linkage via Heterogeneous Behavior Modeling

HYDRA Large-scale Social Identity Linkage via Heterogeneous Behavior Modeling HYDRA Large-scale Social Identity Linkage via Heterogeneous Behavior Modeling Siyuan Liu Carnegie Mellon. University Siyuan Liu, Shuhui Wang, Feida Zhu, Jinbo Zhang, Ramayya Krishnan. HYDRA: Large-scale

More information

A BFS-BASED SIMILAR CONFERENCE RETRIEVAL FRAMEWORK

A BFS-BASED SIMILAR CONFERENCE RETRIEVAL FRAMEWORK A BFS-BASED SIMILAR CONFERENCE RETRIEVAL FRAMEWORK Qing Guo 1, 2 1 Nanyang Technological University, Singapore 2 SAP Innovation Center Network,Singapore ABSTRACT Literature review is part of scientific

More information

CAP 6412 Advanced Computer Vision

CAP 6412 Advanced Computer Vision CAP 6412 Advanced Computer Vision http://www.cs.ucf.edu/~bgong/cap6412.html Boqing Gong April 21st, 2016 Today Administrivia Free parameters in an approach, model, or algorithm? Egocentric videos by Aisha

More information

Detecting Malicious Web Links and Identifying Their Attack Types

Detecting Malicious Web Links and Identifying Their Attack Types Detecting Malicious Web Links and Identifying Their Attack Types Anti-Spam Team Cellopoint July 3, 2013 Introduction References A great effort has been directed towards detection of malicious URLs Blacklisting

More information

Named Data Networking

Named Data Networking Named Data Networking Ronald van der Pol Outline! Goal of this project! Shortcomings of current internet! NDN basics! NDN forwarding! Naming example! Security! Personal opinion Goal of this

More information

Project 3 Q&A. Jonathan Krause

Project 3 Q&A. Jonathan Krause Project 3 Q&A Jonathan Krause 1 Outline R-CNN Review Error metrics Code Overview Project 3 Report Project 3 Presentations 2 Outline R-CNN Review Error metrics Code Overview Project 3 Report Project 3 Presentations

More information