Applying deep learning to multimodal data in social media

Transcription

1 Applying deep learning to multimodal data in social media Tuomo Hiippala 1,2 Christoph Fink 2 Vuokko Heikinheimo 2 Henrikki Tenkanen 2 Tuuli Toivonen 2 1 Department of Languages 2 Digital Geography Lab University of Helsinki, Finland

2 Social media data is inherently multimodal Most communicative situations are multimodal, that is, they involve combinations of different expressive resources (Bateman et al. 2017) Face-to-face interaction: spoken language with gestures, gaze and posture Social media posts: written language, images, emojis, diagrammatic elements, layout, drawings, etc. PHOTOGRAPH LANGUAGE EMOJI Bateman, Wildfeuer & Hiippala (2017) Multimodality: Foundations, Research and Analysis. Berlin: De Gruyter.

3 Multimodality and geo-texts Implications of multimodality for geo-text analytics? Answers to questions about who, what and why may be provided multimodally Finding answers requires combining techniques from natural language processing, computer vision, etc. Di Minin, Tenkanen and Toivonen (2015) Prospects and challenges for social media data in conservation science. Front. Environ. Sci. 3(63): doi: /fenvs

4 Case study: activity detection Detecting activities in Instagram posts associated with urban green areas in Helsinki, Finland: jogging, dog walking, cross-country skiing, etc. Manually annotated as a part of a larger research project on using social media data for conservation not designed with machine learning in mind 2602 images with activities (24%) 8339 images with no activities (76%) Heikinheimo, V. et al. (2017) User-Generated Geographic Information for Visitor Monitoring in a National Park: A Comparison of Social Media Data and Visitor Survey. ISPRS Int. J. Geo-Inf. 6(3): doi: /ijgi

5 Multimodal classification Duong et al. (2017) identify several approaches to multimodal classification 1. Late fusion training separate classifiers for images and text, averaging the classification results 2. Early fusion concatenating features for both images and text, before feeding them to a single classifier 3. Joint fusion learning a joint representation and feeding it to a single classifier Duong et al. (2017) Multimodal classification for analysing social media. arxiv:

6 Late fusion IMAGE CAPTION NASNet-L pre-trained on ImageNet fasttext word embeddings trained on all available captions Hidden layer with 300 neurons 300-dimensional feature vector Dense layer with 64 neurons Dense layer with 64 neurons Dense layer with a sigmoid activation Dense layer with a sigmoid activation Averaged predictions

7 Early fusion IMAGE CAPTION NASNet-L pre-trained on ImageNet fasttext word embeddings trained on all available captions Hidden layer with 300 neurons 300-dimensional feature vector Concatenated feature vector with 600 dimensions Dense layer with 64 neurons Dense layer with a sigmoid activation

8 Joint fusion IMAGE CAPTION NASNet-L pre-trained on ImageNet fasttext word embeddings trained on all available captions Hidden layer with 300 neurons 300-dimensional feature vector Averaged feature vector with 300 dimensions Dense layer with 64 neurons Dense layer with a sigmoid activation

9 Training process Training, validation and testing splits 8861 samples for training 865 for validation 1065 for testing Trained for 40 epochs Optimized using Adam (Kingma & Ba 2014) Implemented using Keras / TensorFlow Kingma & Ba (2014) Adam: A method for stochastic optimization. arxiv:

10 Results Monomodal Multimodal Captions only Images only Joint fusion Early fusion Precision Recall F1-score Activity Not activity Average Activity Not activity Average Activity Not activity Average Activity Not activity Average Results are averaged over five random training / testing splits

11 Take-home message When designing and training classifiers, consider how users communicate on the studied platform CNN features and word embeddings offer a solid baseline for multimodal classification Easy to implement with little added cost Code for this presentation is available at: