NExT++: Center for Extreme Search

Transcription

1 NExT++: Center for Extreme Search A Joint Center between NUS, Tsinghua and Southampton University CHUA Tat-Seng National University of Singapore

2 OUTLINE Welcome Key Research Focuses Summary

3 Users at Center of Social Media Environment Public Personal -- Friends

4 NExT++: NUS-Tsinghua Centre on Extreme Search o We were among the first to examine live big multi-channel data analytics in 2010 o Phase I: May 2010 to Sep 2016 with a Grant of S$11 million Emphasis: Technology for unstructured data analytics o Phase II: Oct 2016 to Sep 2021 with a Grant of S$12 million Emphasis : Deep unstructured data analytics o Additional Collaborator: Southampton University M3KG (Multilingual Multimodal Knowledge Graph) o Funded by NRF-NSFC

5 Key Management Team NUS: Co-Director: Chua, Tat-Seng Program Director: Seah, Choon Meng Admin Director: Tan, Tekmin Tsinghua: Co-Director: Sun, Maosong Deputy Director: Li, Juanzi Southampton: Co-Director: Hall, Wendy Deputy Director: Tiropanis, Thanassis International Coordinator: Luan, Huanbo

6 Key Research Focuses Key research focuses: 1) Learning from Multi-Channel Data 2) Rich Media Analytics 3) Multimedia Multimodal Knowledge Graph 4) Recommendation 5) Big data wellness Analytics

7 OUTLINE Welcome Key Research Focuses Summary

8 Key research Wellness profile Diabetes, Obesity, Gout, Depression,.. User profile Mobility profile Movement Patterns, Venue Preferences Demographic profile Age, Gender, Interests, Personality, Occupation,.. User Demography Detection Community Profiling Wellness (diabetes, obesity, depression) Communities Detection Sentiment Analysis Gaming (addiction prediction)

9 plus domain info: Fintech: (+ pricing models for asset price estimation) Business Intelligence: (+ proprietary data) Academic Channel Fusion of knowledge from different types of info networks: Eg. User centric vs. Info centric Recommend relevant items of info domains to users in social domain Integrate info from both domains deep learning & Graph propagation Chatbot for unstructured data

10 Entity extraction is well researched Accuracy on 1,000 concepts on 1-million ImageNet Dataset has progressed well in last few years Next research frontier on relation extraction At image level and video level Some initial results at image level (Zawlin) (see Demo link:

11 Aim: find all (meaningful) triplets of {Sub, Pre, Obj} in videos Object trajectory detection Object recognition Video relation inference Bring level of video semantics to that of language Joint analysis of video & text Video Question-Answering Vide Description Multimodal Knowledge Graph

12 Starting with a Video Video SLAM 3D Point cloud 3D semantic segmentation Seg m point cloud 3D Modeling with Pose 3D Models Applications: Augmented Reality PointNet CRF-RNN End-to-end network INPUT: point cloud OUTPUT: segmented point cloud Dataset: S3DIS Dataset (CVPR2016) [ ScanNet Dataset (CVPR2017) [ Towards 3D model matching with poses, for AR Appl.

13 On a given topic with multimodal and multilingual data, generate M3KG There are often great variety of taxonomies maintained by different groups: Ecommerce taxonomies Others: culture, legal, entertainment, healthcare, food, etc. Take ecommerce example: Different taxonomies by different retailers Entities (instances) organized in different taxonomies with different multimodal descriptions

14 Key tasks: Multimodal entity linking Multimodal taxonomy normalization Integration of external knowledge Search and Recommendation: across taxonomies, organizations, countries & cultures A taxonomy manually constructed by Lizi Related Research: Multimodal knowledge Graph Multimodal Chatbot

15 [Zhang et al KDD CKE.] User behaviors are affiliated with rich side information: User demographics Item attributes Textual reviews Various contexts Key research: Fusion of all side info together to better predict user preference? Automatically learn feature interactions, rather than manually design cross features!

16 Learning explainable rules from decision trees, and selecting sound rules for explaining user-item interactions via attention. Provide sound & explainable reasons (rules) on why the product is suitable choice for the user. Reasons for Recommending The River Café: User being <User-City: Florida, User-Style: Nightlife Seeker Luxury Traveler, User-Age: 50-64> would like to visit Item being <Item-Price: $$$$, Item-Attr: Foie Gras, Lobster> Embed the explainable rules on certain features from decision trees

17 Diabetes Depression Arthritis Asthma Cancer Heart Disease Treatment & Tests Eating behavior Diabetes Mgmt Exercise

18 USERS Apps Obesity Diabetes Gout Depression, etc Knowledge Wellness Knowledge User Data Food Intake & Habits Activities (Physical & Cyber) Others: Treatment & Test; Environment Data

19 Our Application: DietLens Food recognition Recognize a dish photo through deep learning tech Food and acitivty diary Easy to record your everyday diet, extract daily activity from your smart phone..more convenient Analysis report Give users weekly or monthly report about their diet Bespoke recommendation Combining medical knowledge graph, personal profile to customize recommendation and encouragement Gamification Small incremental task and targets to help earn badges and achievements milestone

20 DietLens 125K 250+ Build on around 125k food images and growing Now support 250+ local food recognition Adding more as we progress

21 Tech & system to gathering big lifestyle data Diet, Activities (online & offline), Medical test results The big wellness data analytics Inference of trends in food and exercise Incorporation of wellness knowledge Prediction of likely longer-term illnesses at individual and group level Recommendation and influence of positive lifestyle changes

22 OUTLINE Welcome Key Research Focuses Summary

23 Discuss these 5 areas of research: o o o o o Real-time Unstructured Big Data Analytics Video Processing Multimodal Multilingual Knowledge Graph Recommendation Wellness Underlying Theme: o o o Extraction, Use & Structuring of Knowledge Towards explainability Towards deeper insights

24 Thanks Visit our Web Observatory: