Multi-task Emoji Learning

Transcription

1 Muli-ask Emoji Learning Francesco Barbieri Luís Marujo Pradeep Karuuri William Brendel Large Scale Tex Undersanding Sysems Lab, TALN, UPF, Barcelona, Spain Snap Inc. Research, Venice, California, USA Absrac Emojis are very common in social media and undersanding heir underlying semanics is of grea ineres from a Naural Language Processing poin of view. In his work, we invesigae emoji predicion in shor ex messages using a muli-ask pipeline ha simulaneously predics emojis, heir caegories and sub-caegories. The caegories are eiher manually predefined in he unicode sandard or auomaically obained by clusering over word embeddings. We show ha using his caegorical informaion adds meaningful informaion, hus improving he performance of emoji predicion ask. We sysemaically analyze he performance of he emoji predicion ask by varying he number of raining samples and also do a qualiaive analysis by using aenion weighs from he predicion ask. 1 Inroducion Emojis are a popular se of ideograms creaed in he lae 1990s o enrich wrien communicaion by adding nonverbal expressive power o digial communicaion. These symbols can be used by human readers o convey emoions and informaion in a condensed form. As Snapcha, Twier and oher social media plaforms have become popular, so has he usage of emojis. Despie heir populariy, here is very lile research work in predicing emojis. Copyrigh c 2018 held by he auhor(s). Copying permied for privae and academic purposes. In: S. Wijerane, E. Kiciman, H. Saggion, A. Sheh (eds.): Proceedings of he 1 s Inernaional Workshop on Emoji Undersanding and Applicaions in Social Media (Emoji2018), Sanford, CA, USA, 25-JUN-2018, published a hp://ceur-ws.org Over he pas few years, he ineres in emoji research has increased and several sudies has been published in he area of disribuional semanics [BRS16, ERA + 16, WBSD17b, WBSD17a, BCC18], senimen analysis [NSSM15, HGS + 17, KK17, RPG + 18] and mulimodal sysems [CMS15, CSG + 18, BBRS18]. In he pas year researchers also focused on he possibiliy of predicing emojis in a ex message [BBS17, FMS + 17]. The emoji predicion ask consiss in predicing he original emoji presen in a wee (or snap capion) given only he non-emoji exual conen. Prior exploraions of emoji predicion ended o focus on less han 2% of he oal number (2653) of emojis in Unicode 6 sandard 1 emojis. Anoher limiaion of hose papers was ha emoji predicion could be ambiguous. For example, when he model predics he correc label o be, emojis like,,, or can also be valid predicions. In his work, we exended he emoji predicion ask o 300 emojis in order o sudy a larger number of emojis along wih heir unicode sandard caegories, sub-caegories, and he new semanic clusers ha we creaed. We are no aware of any previous research work focused on eiher predicing a large number of emojis (300), or using a muli-ask approach o predic emojis or emoji caegories. We also do a sysemaic analysis of how he number of raining samples affec he performance of he emoji predicion ask. To miigae he problem of emoji ambiguiy, we concenrae on broad emoji caegory predicion in addiion o ha of individual emoji predicion. We grouped emojis in wo differen ways. The firs one was defined by he Unicode consorium 2, which groups emojis ino seven caegories (e.g., Smileys & People, Naure ) and 74 sub-caegories (e.g., face-posiive, face-negaive ). The main caegories are commonly found on mobile phone keyboards as shown in Figure 1). Alernaively

2 Figure 1: Screensho of Apple Emojis keyboard including a subse of Smileys & People caegory emojis, and buons o access he remaining caegories. we also creaed semanic clusers using embeddings. We use a Muli-ask approach o combine he asks of emoji and caegory predicion. Muli-ask approaches [Car, Car97, Ben12, CW08] improves generalizaion by ransferring informaion across differen asks and improving each ask individually. In paricular, muli-ask learning wih simulaneous raining on muliple asks has demonsraed promising resuls [CW08][FMS + 17][Ben12]. Our work performs muli-ask learning by raining a single model wih muliple oupus (he daase is annoaed wih muliple labels) and we evaluae using our gold sandard creaed from Twier and Snap public poss as described in he Daases secion. The subjeciviy of emoji inerpreaion makes emoji predicion a very challenging ask. Neverheless our work shows ha simulaneously predicing emojis, heir caegories, and sub-caegories in a muliask framework improves he overall resuls. I no only improves emoji predicion, bu i also helps wih he idenificaion of emoji caegories, which can be paricularly more relevan when he emoji predicion model is less precise. The remainder of his work is organized in he following way: The nex secion describe he daases used in our experimens. We hen presen he Deep Learning Models explored o solve our research problem, Finally, we discuss he experimens, resuls and hen conclude wih fuure research direcions. 2 Daases In his sudy we explore emoji predicion for wo differen daases: Twier and Snapcha capions. We selec documens (wees and snaps) ha conain a single emoji, and a leas hree okens apar from he emoji. We resric o documens conaining a single emoji so as o minimize he inerference of he presence of oher emojis in he emoji predicion ask. We also consider only he documens ha include he mos frequen 300 emojis in each daase. We resric o he op 300 emojis only due o lack of meaningful number of examples beyond ha. A subse of he mos frequen emojis for each daase is repored in Table 1. Noe ha we remove skin color from he emojis 3 o avoid generaing very similar labels. Table 2 includes saisics on he wo daases. We can see ha Snap capions are shorer han wees, while average word lengh is similar. Anoher imporan difference beween he wo daases is he mos frequen emojis used. Table 1 shows he 60 mos frequen emojis in each daase (on he op Twier and on he boom Snap daa), along wih he number of documens ha include each emoji. In boh daases he frequency is very unbalanced: 16% of wees and 25% of snaps include one of he hree mos frequen emojis,,. Therefore we use a balanced daase in our experimens, in order o give same imporance o each emoji, independen of he frequency of usage. We subsample he mos frequen emojis in order o mach he same number of examples of he leas represened emoji (1,500 examples for Twier and 3,000 for Snap daa). We show ha using fewer han 1,500 examples per emoji leads o a drasic decrease in accuracy of he emoji deecion (see Figure 3). We focus our experimens on 300 emojis because we do no have more han 1,500 wees per emoji beyond he op 300 emojis in our Twier daase. For our experimens we randomly chose 80% of he documens for raining, 10% for validaion and 10% for esing. 2.1 Twier Daase The Twier daase conains 50 million wees rerieved using Twier API. Twees were posed beween January 2016 and April 2017 and were geolocalized in he Unied Saes. We removed hyperlinks from each wee, and lowercased all exual conen in order o reduce noise and sparsiy. Since Twier daa includes a large percenage of bo daa, we filer noise as much as possible, removing repeaed wees (or very similar ones) and seleced a maximum of five wees per user. From his daase, we seleced wees including anyone of he 300 mos frequenly occuring emojis and a leas hree okens (wihou he emoji), resuling in a final daase composed of 2,862,505 wees. 2.2 SnapCapion SnapCapion is an in-house Snapcha inernal daase conaining only Snapcha capions. A capion is he exual overlay componen of a snap. These capions were colleced exclusively from snaps submied o public and crowd-sourced sories (as known as Live Sories or Our Sories). Examples of such public crowd-sourced sories are New York Sory or Thanksgiving Sory. All capions were posed in one year period and do no conain any image or any 3 E.g., are mapped o one single label

3 Table 1: 60 mos frequen emoji for he Twier (op) and Snap (boom) daases oher associaed informaion. 30,004,519 capions. This daase conains 2.3 Caegories and Clusers of Emojis We also consider broader classes of emojis, such as unicode caegories and semanic clusers. The unicode consorium defines a se of 7 caegories caegories and 74 sub-caegories. The problem wih Unicode caegories and subcaegories is ha hey fail o accuraely capure semanically relaed emojis. Emojis like and are boh in he sub-caegory neural faces even hough hey clearly indicae differen emoions. Anoher example is and ha are semanically similar, bu hey appear in differen caegories ( Smiling Faces and Emoions ) even hough hey address nearly idenical meanings. To overcome his limiaion, we propose a second approach o auomaically organize emojis by clusering hem using pre-rained word embeddings similar o emoji2vec [?]. These clusers have he advanage of beer capuring he semanic informaion of emojis. For example and are in he same cluser. These clusers are an imporan aspec o consider because hey are based on how emojis cooccur in shor ex messages from wees and capions of public snaps. We prerained wo differen ses of skip-gram embeddings [MLS13] for Twier and Snap. The firs skip-gram model was rained on a daase of abou 70 million wees and he second skip-gram model was rained on abou 100 million Snap capions. Using he embeddings of he 300 mos frequen emojis of each daase, we creaed wo ses of 30 clusers using a k-means algorihm. The number of clusers was defined based on qualiaive analysis (clusers ha seemed o beer organize emojis by semanics). In addiion, he number of clusers was seleced such ha each cluser has a similar number of emojis ha are usually displayed on a mobile keyword. As a resul, we would be able o jus provide an icon o access direcly each cluser in a similar way as he Figure 1 shows for he op caegories. The resuling clusers will group semanically similar emojis (like in [BKRS16] where 11 cluser are creaed for 100 emojis), grouping love, sad faces, hand/gesures, animals, food, drinks, paries, Chrismas, and so on. Table 2: Average, sandard deviaion and Median lengh of words and characers of he wo daases. Words Chars Daase Avg. Sd. Median Avg. Sd. Median Twier Snap Models Our main archiecure, illusraed in Fig. (2), sars wih our characer and word embedding modules whose oupus are fused by our feaure aenion uni and he word aenion uni. Finally he fully conneced layers and he sofmax play he role of he final muli-ask classifier. Previous approaches [BBS17, FMS + 17] have successfully learned LSTM models for emoji predicion asks. We experimened differen plain LSTMs, sacked LSTMs [FMS + 17], and differen word represenaions before solidifying on our final model archiecure Fig. (2). In addiion, we explored single ask models and muli-ask models. In he case of he muli-ask models, he enire nework is shared and he specializaion only occurs a he final sage o predic specific labels of each ask. This specializaion is accomplished hrough specific linear ransformaions. Finally we used a cross enropy loss funcion for all classificaion asks. In he case of muliask learning, he final loss is he sum of each single loss 4. In he following subsecions, we deail each sage of our main archiecure. 3.1 Word Represenaion The word embeddings are learned ogeher wih he updaes o he model. For ou-of-vocabulary words (OOVWs), we used a fixed represenaion ha is handled as a separae word. In order o rain he fixed represenaion for OOVWs, we sochasically replace (wih p = 0.5) each word ha occurs only once in he raining daa. When we use pre-rained word embeddings, ha are concaenaed wih he learned vecor. 4 We also experimened weighed sum, wih various weighs, bu he bes resuls are obained wih a simple sum of he losses.

4 3.4 Feaure Aenion Figure 2: Final archiecure of our model 3.2 Char Represenaion In addiion, we use a characer based embedding [LLM + 15,?] sacked wih a B-LSTM [GS05], producing a characer-based word embedding ha focuses on word spelling varians. Indeed, he characer-based word embedding learned similar represenaions for words ha are orhographically similar, and hus are expeced o handle differen alernaives of he same word ypes ha normally occur in social media. 3.3 Bi-direcional LSTMs Our bi-direcional LSTM modules, [GS05] named B- LSTM in Fig. (2), consiss of a forward LSTM ha processes an inpu message from lef o righ, while he backward LSTM processes i in he reverse direcion. As a resul, he message represenaion s is based on boh he forward and backward LSTM encoding: s = max {0, W[h fw ; h bw ] + d} where W is a learned parameer marix, fw is he forward LSTM encoding of he message, bw is he backward LSTM encoding of he message, and d is a bias erm, and we use a componen-wise ReLU as he nonlinear uni. We use B-LSTM modules for boh word and senence represenaions, namely Char B-LSTM and Words B-LSTMs in our archiecure Fig. (2). Char B-LSTM akes a sequence of characers and oupus a word embedding vecor. This oupu is mixed wih anoher word represenaion via our feaure aenion module. Then, he sacked Words LSTMs receive sequences of word represenaions from he aenion module, and oupu senence embedding vecors. The feaure aenion module aims o linearly fuse muliple inpu signals insead of simply concaenaing hem. In our archiecure, his module learns a unified word represenaion space, i.e. i produces a single vecor represenaion wih aggregaed knowledge among our muliple inpu word represenaions, based on heir weighed imporance. We can moivae his module from he following observaions. Prior work, [BBS17] combines boh word represenaion x (w) and characer-level represenaion x (c) by simply concaenaing he word and characer embeddings a each LSTM decoding sep h = LSTM([x (w) ; x (c) ]). However, his naive concaenaion resuls in inaccurae decoding, specifically for unknown word oken embeddings, e.g., an all-zero vecor x (w) = 0 or a random vecor x (w) = ɛ U( σ, +σ), or even for ou-of-vocabulary words. While his concaenaion approach does no cause significan errors for well-formaed ex, we observe ha i induces performance degradaion for our social media pos daases which conain a significan number of slang words, i.e., misspelled or ou-of-vocabulary words. As a resul, we use a feaure aenion module, ha adapively emphasizes each feaure represenaion in a global manner a each decoding sep. This process produces a sof-aended conex vecor x as an inpu oken for he nex sacked B-LSTM ha akes care of he senences embedding. [RCP16] inroduced a similar approach, where he characer embeddings are weighed wih an aenion module. We use he following mehod: [a (w), a (c) ] = σ ( W m [x (w) ; x (c) ) ] + b m α (m) = where α = [α (w) exp(a (m) ) exp(a (m ) m {w,c} ) m {w, c} (1) ; α (c) ] R 2 is an aenion vecor a each decoding sep, and x is a final conex vecor a ha maximizes informaion gain for x. Noe ha his feaure aenion requires each feaure represenaion o have he same dimension (e.g. x (w), x (c) R p ), and ha he ransformaion via W m essenially enforces each feaure represenaion o be mapped ino he same unified subspace, wih he oupu of he ransformaion encoding weighed discriminaive feaures for classificaion of emojis. 3.5 Word Aenion No all he words have he same imporance in he represenaion of a documen. We use he aenion

5 mechanism inroduced in [YYD + 16]: u i = anh(w w h i + b w ) α i = exp(u i u w) exp(u i u w) ; d i = α i h i (2) where he final documen represenaion d i is a weighed average of he hidden represenaion h i of he LSTM. The weighs α i are learned by he use of a Muli-Layer Percepron (linear ransformaion W and biases b) wih anh as non-linear operaion, and a sofmax o compue he probabiliy of each word. 4 Experimens And Resuls We use wo main variaions for experimens: Single- Task Predicion of emojis, unicode caegories, and emoji clusers, and Muli-Task Predicion, where we combine he single asks in one single model. We also evaluae he impac of our differen modules including he combinaion of word/char LSTMs and he word aenion uni. Finally we invesigae he influence of he number of layers for he LSTMs. 4.1 Single-Task Predicion We explore hree differen asks: (i) he emoji predicion ask proposed by [BBS17], (ii) predicion of unicode emoji caegories (he emoji in he ex belong o he faces, animal, objecs) and sub-caegories (posiive faces, animal-mammal), and (iii) predicion of auomaic clusers ha we previously generaed using pre-rained word embeddings Predicing Emojis Given a se of documens, each documen conaining only one emoji class, he ask consiss of predicing he emoji from he ex. For his ask, we esed he influence of he number of emoji classes and he number of examples per class. More precisely, for each experimen, we exrac a balanced daase of N class emoji classes, and N daa examples per class, wih N class = {20, 50, 100, 200, 300 and N daa = {100, 500, 1000, 1500, 2000, 2500, 3000}. N class and N daa are esed independenly: when we vary N class, we fix N daa o 3000, and when we vary N daa we fix N class o 300. Figure 3 shows our experimens wih he Snapcha daase. I is clear ha using more examples per class improves our model by around 1% absolue poin from 1500 o 3000 examples. For >2000 examples he sysem converges o is opimum. From Figure 4, we observe ha Twier daa is easier o model han Snap daa. In he 300 emoji predicion ask he bes accuracy a op 5 (a@5) on Twier daa is 40.05% while on Snap daa i is 34.25% (see Top 5 Accuracy lsm 28 lsm + a 26 2 lsm + a char + 2 lsm + a 24 char + lsm char + lsm + a ,000 1,500 2,000 2,500 3,000 Training examples per class Figure 3: Acc@op 5 of he same algo. bu variable nr. of raining insances per class (from 100 o 3000 examples for each emoji) on SnapCapions. Tes and validaion se are fixed for each experimen. Table 3). There are several reasons ha could explain his difference in resuls. One reason is he lengh of he ex messages, since in Twier here are on average welve words per message, while on Snap has only five (see Table 2). Anoher reason could be he missing visual conex of Snap poss 5, while only a small percenage of wees is complemened wih a visual conen. For his reason, wees conain ypically less semanic ambiguiy. Table 3 highlighs he bes performing sysems on he emoji predicion ask. For he wo daases sae of he ar sysems are ouperformed by he combinaion of addiional componens. For example, adding a word aenion module improves he baseline of [BBS17]. Finally, here is an imporan difference when predicing 20 and 300 emojis. We plo on he lef of Figure 3 he accuracy of same model archiecure (Char + 2 LSTM + word aenion) on he emoji predicion ask for differen numbers of emojis (20 o 300). Bes accuracy a op 5 (a@5) drops from 20 o 100, and hen remains consan. We observe he same drop using F1 (ha only considers wheher an emoji is prediced as firs opion), however, having more han 100 classes resuls in improvemen. This is probably due he ype of he more rare emoji classes added afer he mos 100 frequen ones, ha are more specific (like,, or ) hence easier o predic Predicing Unicode Emoji Caegories and Sub-caegories We predic Unicode emojis caegories and subcaegories using he ex message ha includes only 5 Snap ex messages are capions of images and videos posed by Snapcha users, see Daases secion.

6 Table 3: Emoji predicion resuls using muliple number of emojis (20 o 300) and differen models. We use he original implemenaion of [FMS + 17], while we implemen [BBS17] Daase Models F1 a@5 F1 a@5 F1 a@5 F1 a@5 F1 a@5 Twier LSTM LSTM + A LSTM + A. [FMS + 17] Char + LSTM [BBS17] Char + LSTM + A Char + 2 LSTM + A Top 5 Accuracy Snap LSTM LSTM + A LSTM + A. [FMS + 17] Char + LSTM [BBS17] Char + LSTM + A Char + 2 LSTM + A Top 5 acc: Twier Top 5 acc: Snap Macro F1: Twier Macro F1: Snap Number of labels Figure 4: F1 and op 5 for he model Char + 2 LSTM + Word A. on Twier and Snap daa. one emoji as we did in he emoji predicion ask. Table 4 shows he predicion resuls using macro- F1 and a@5 evaluaion merics. In he firs wo blocks (main and sub lines), we predic he main caegory and sub-caegory respecively. The hird block deails he clusers evaluaion resuls, and he las block presens he emoji predicion resuls. In he firs line of each block are he single-ask resuls and he remaining lines include he ones using a muli-ask framework Predicing Clusers Given a ex message conaining an emoji e we predic he cluser ha emoji e belongs o. Cluser creaion is described in he daase secion. Cluser resuls are repored in Table 4, in he lines corresponding o Semanic Clusers. The resuls are beer on Snap han Twier for broader classes and our clusers capure beer semanics han he caegories and Macro F1 Table 4: Resuls for single and muli-ask predicion of emojis including main unicode caegories, subcaegories, and clusers. Pred. Twier Snap Task Loss F1 A@5 F1 A@5 Main Main Caegory Main + Sub Main + Emoji Main + sub + Emoji Sub Sub Caegory Sub + Main Sub + Emoji Sub + Main + Emoji Semanic Clusers Clusers Clusers + Emoji Emoji Emoji Emoji + Main Emoji + Sub Emoji + Main + Sub Emoji + Clusers sub-caegories of Unicode Sandard. 4.2 Muli-Task Predicions In Table 4 we show he muli-ask predicion resuls. We considered muliple muli-ask combinaions. Learning more han one objecive ask simulaneously helps in he main caegory predicion, as macro F1 improves from 46.56% o 48.52% (4.2% relaive improvemen) when adding also sub-caegory and emoji losses. Sub-caegories predicion also improves when i is learned ogeher wih main caegories and emojis. On Snap daa, caegory and sub-caegory predicion asks do no improve using a muliask approach in erms of macro F1, bu we obain a relaive improvemen of 8.67% and 21.14% using a@5. The clusers predicion asks also benefi from muli-ask learning when combined wih he emoji predicion. However, emoji predicion seems no o improve much in a muli-ask seing for Twier. Emoji

7 G: P: 0.26, 0.15, 0.13, 0.10, 0.04 we are having a ca pary in room 135 #ausinblueroos #behedifference we are having a ca pary in room 135 #ausinblueroos #behedifference G: P: 0.75, 0.02, 0.02, 0.01, 0.01 Feeling slighly pregnan bu i was worh i Feeling slighly pregnan bu i was worh i G: P: 0.97, 0.01, 0.002, 0.001, I s official, I have found my #unicorn! Look a his geomeric ribal prin! I s official, I have found my #unicorn! Look a his geomeric ribal prin! Figure 5: Word and Feaure aenion visualizaion. The firs line highlighs in blue word aenion, while he second line shows he feaure aenion. Uncolored words mean almos zero aenion over hem. Table 5: Top and boom 10 emojis wih bes accuracy on he Twier (op) and on Snap (boom) predicion on Snap improves from 33.86% o 34.27% or 1.21% relaive improvemen in erms of a@5 when i is learned ogeher wih Unicode sub-caegories. 4.3 Qualiaive Analysis We analyzed in deail our emoji predicion approach (char + 2 LSTM + aenion) based on he bes performing sysem described in he previous secion. This analysis enumeraes he emojis ha are easier and harder o predic. We also include some visualizaion examples of where he aenion module obains more informaion. These examples provide us wih a beer undersanding of he imporance of he characer and word feaures in our resuls Wha emoji is difficul o predic? Table 5 shows a lis of he op and boom 10 emojis based on he predicion accuracy. We invesigaed wha emojis are difficul o predic, and we found ineresing paerns. As expeced, he emojis ha are easier o predic describe specific objecs wihou muliple meanings (such as, and ) or opics (e.g., and ). These emojis, as suggesed in [BRS16], could easily be replaced by a word, such as by key), or are used when specific words occur in a ex message including Chrismas for and ). In boh daases, subjecive emojis including and obained lowes accuracy values. These subjecive emojis describe emoional informaion, and hey can be inerpreed differenly among differen users and based on he surrounding conex. Hence, hese emojis do no seem o have a specific meaning and become difficul o model Feaure and Word Aenion We previously described he wo ypes of aenion explored. The Feaure Aenion approach gives more imporance o eiher he characer or word represenaion of a word. The Word Aenion approach increases he imporance of more discriminaive words, for example he word key o predic he emoji. Figure 5 visualizes he weighs of each of hese wo aenion modules using hree example messages. For each of hem, we lis he gold label ( G ) and he prediced labels ( P ), along wih heir predicion probabiliy. i.e. he oupu of he sofmax layer. The inernal weighs of he wo aenion modules are visualized using ex highlighs. Darker color indicaes more aenion over word (α i from Formula 2 of each word in he message). In second line of each message he red highligh shows he weighs of he feaure aenion (α of Formula 1). Bold ex formaing indicae he ou of vocabulary words. Based on he hree examples, and some addiional ha we manually evaluaed, we verified how hese wo aenion approaches work. The Word Aenion module (blue highligh) give us insighs on he recogniion of emojis. In he firs example he mos imporan word is ca and he predicions are indeed abou cas, apar from he fifh prediced emoji. This emoji is riggered (probably) because of he presence of he oken 135 as he word aenion module also focuses on his oken. In he second example, he aenion goes o he word pregnan, bu in his case his word misleads he nework ha incorrecly predics baby emojis. However, he correc emoji is hen prediced as fourh opion. In he las example, he nework correcly classifies he emoji, based on he hashag #unicorn. Regarding he Feaure Aenion over he word or characer represenaion of each oken in a message, we observed ha he characer represenaion seems o gain imporance on long and less frequen okens, namely numbers, hashags, and as expeced, ou of

8 vocabulary words ( 135 and #ausinblueroos ). 5 Conclusion In his paper, we explored emoji predicion in wo social media plaforms, Twier and Snapcha. We exended he emoji predicion ask o a large number of emojis and showed ha he predicion performance drasically drops beween 50 and 100 emojis, while he addiion of more emojis keeps he accuracy of he model somehow consan (even if i has o predic more emojis). We aribue hese resuls o he specificiy of he less-used emojis. We also proposed a novel ask ha predics broader classes of emojis, grouping emojis in auomaic clusers or predefined caegories, as defined by he Unicode consorium. These new asks allow us o beer evaluae he predicions of he model, since plain emoji predicion may be ambiguous. We also carried ou an exensive qualiaive analysis in order o undersand he imporance of he characer encoding of words in noisy social media ex, he number of raining examples, and he difficulies in modeling specific emojis. Finally, we proposed a muli-ask approach o predic emojis and emoji group affiliaion a he same ime. We showed ha he model obains significan improvemens in he Twier daase, while more invesigaion is needed for he Snapcha daase. Acknowledgmens This work was done when Francesco B. inerned a Snap Inc. Francesco B. acknowledge suppor also from he TUNER projec (TIN C5-5-R, MINECO/FEDER, UE) and he Maria de Maezu Unis of Excellence Programme (MDM ). References [BBRS18] [BBS17] [BCC18] Francesco Barbieri, Miguel Balleseros, Francesco Ronzano, and Horacio Saggion. Mulimodal emoji predicion. In Proceedings of NAACL: Shor Papers, New Orleans, US, Associaion for Compuaional Linguisics. Francesco Barbieri, Miguel Balleseros, and Horacio Saggion. Are emojis predicable? In Proceedings of he 15h Conference of he European Chaper of he Associaion for Compuaional Linguisics: Volume 2, Shor Papers, pages , Valencia, Spain, April Associaion for Compuaional Linguisics. Francesco Barbieri and Jose Camacho- Collados. How Gender and Skin Tone [Ben12] [BKRS16] [BRS16] Modifiers Affec Emoji Semanics in Twier. In Proceedings of he 7h Join Conference on Lexical and Compuaional Semanics (*SEM 2018), New Orleans, LA, Unied Saes, Yoshua Bengio. Deep learning of represenaions for unsupervised and ransfer learning. In ICML Workshop, Francesco Barbieri, German Kruszewski, Francesco Ronzano, and Horacio Saggion. How Cosmopolian Are Emojis? Exploring Emojis Usage and Meaning over Differen Languages wih Disribuional Semanics. In Proceedings of he 2016 ACM on Mulimedia Conference, pages , Amserdam, Neherlands, Ocober ACM. Francesco Barbieri, Francesco Ronzano, and Horacio Saggion. Wha does his emoji mean? a vecor space skip-gram model for.emojis. In LREC, [Car] R. Caruana. Muliask learning: A knowledge-based source of i.b. In ICML 93. [Car97] [CMS15] [CSG + 18] Rich Caruana. Muliask learning. Mach. Learn., 28(1):41 75, July Spencer Cappallo, Thomas Mensink, and Cees GM Snoek. Image2emoji: Zero-sho emoji predicion for visual media. In Proceedings of he 23rd ACM inernaional conference on Mulimedia, pages ACM, Spencer Cappallo, Sacey Svelichnaya, Pierre Garrigues, Thomas Mensink, and Cees GM Snoek. The new modaliy: Emoji challenges in predicion, anicipaion, and rerieval. arxiv preprin arxiv: , [CW08] R. Collober and J. Weson. A unified archiecure for naural language processing: Deep neural neworks wih muliask learning. In ICML. ACM, [ERA + 16] Ben Eisner, Tim Rockäschel, Isabelle Augensein, Mako Bosnjak, and Sebasian Riedel. emoji2vec: Learning emoji represenaions from heir descripion. In Proceedings of The Fourh Inernaional Workshop on Naural Language Processing for Social Media, pages 48 54,

9 Ausin, TX, USA, November Associaion for Compuaional Linguisics. [FMS + 17] Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. Using millions of emoji occurrences o learn any-domain represen. for deecing senimen, emoion and sarcasm. In EMNLP, [GS05] [HGS + 17] [KK17] [LLM + 15] [MLS13] [NSSM15] [RCP16] Alex Graves and Juergen Schmidhuber. Framewise phoneme classificaion wih bidirecional lsm and oher neural nework archiecures. Neural Neworks, 18, Tianran Hu, Han Guo, Hao Sun, Thuyvy Thi Nguyen, and Jiebo Luo. Spice up Your Cha: The Inenions and Senimen Effecs of Using Emoji. Proc. of ICWSM 2017, Mayu Kimura and Marie Kasurai. Auomaic consrucion of an emoji senimen lexicon. In Proceedings of he 2017 IEEE/ACM Inernaional Conference on Advances in Social Neworks Analysis and Mining 2017, pages ACM, W. Ling, T. Luís, L. Marujo, R.F. Asudillo, S. Amir, C. Dyer, A.W. Black, and I. Trancoso. Finding funcion in form: Composiional characer models for open vocabulary word represenaion. EMNLP, Tomas Mikolov, Quoc V Le, and Ilya Suskever. Exploiing similariies among languages for machine ranslaion. arxiv preprin arxiv: , Pera Kralj Novak, Jasmina Smailović, Boru Sluban, and Igor Mozeič. Senimen of emojis. PloS one, 10(12):e , Marek Rei, Gamal KO Crichon, and Sampo Pyysalo. Aending o characers in neural sequence labeling models. In Coling, [RPG + 18] David Rodrigues, Marília Prada, Rui Gaspar, Margarida V Garrido, and Diniz Lopes. Lisbon emoji and emoicon daabase (leed): Norms for emoji and emoicons in seven evaluaive dimensions. Behavior research mehods, pages , [WBSD17a] Sanjaya Wijerane, Lakshika Balasuriya, Ami Sheh, and Derek Doran. Emojine: An open service and api for emoji sense discovery. Inernaional AAAI Conference on Web and Social Media (ICWSM 2017). Monreal, Canada, [WBSD17b] Sanjaya Wijerane, Lakshika Balasuriya, Ami Sheh, and Derek Doran. A semanics-based measure of emoji similariy. Web Inelligence, [YYD + 16] Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical aenion neworks for documen classificaion. In Proceedings of he 2016 Conference of he Norh American Chaper of he Associaion for Compuaional Linguisics: Human Language Technologies, pages , 2016.