How do people tag pictures? A study with Facebook application

Transcription

1 COMP3750 Final Report How do people tag pictures? A study with Facebook application Author: Victor Hartanto Wibisono, U Supervisor: Dr. Lexing Xie 1 June 2012

2 Acknowledgements This research would not have been possible without the guidance and the help of several individuals who in one way or another contributed and extended their valuable assistance in the preparation and completion of this study. First and foremost, my utmost gratitude to Dr. Lexing Xie, my research supervisor whose sincerity and encouragement I will never forget. Dr. Xie has been a steady pillar of support as I hurdle all the obstacles in the completion this research work. I would also like to thank Mr. Khoi-Nguyen Tran for his support and guidance in the implementation of the Facebook App. I would also like to thank Dr. Weifa Liang the course coordinator, for his guidance in the course. I am grateful for the assistance of fellow students Amy Zhang and Justin Ho who have provided valuable suggestions in the writing of this report. Last but not least, I also thank the ANU Academic Skills and Learning Centre for providing us ANU students with a comprehensive guide to write a research report. Page 2 of 19

3 Abstract This report documents the implementation of Tag Expert a tool that helps researchers carry out investigations into how being a Facebook friend will affect the words people use to tag. Tag Expert is a Facebook app which allow people to tag Flickr pictures of their friends and strangers. These tags are stored in a database which are then analysed to distinguish what kinds of tags come from the friends of the owner of the pictures, and what kinds come from strangers. While the development of the software is not completed due to time constraint, this report outlines some suggestions on how the application can be improved and how the data gathered can be analysed. The application leverages on Facebook s Graph API, Flickr s Developer API, MySQL database and PHP in order to gather and store the data. Page 3 of 19

4 1 Introduction Facebook is currently the largest social networking platform on the Internet. As of May 2012, Facebook has over 900 million active users, more than half of them are on mobile devices. Every day, there are 200 million photos uploaded on Facebook. Other photo sharing websites such as Flickr receives about 50 million pictures from its users every month. Every time a user uploads a picture, he or she will be asked to put a tag on the picture. An image tag is a word that describes the image. According to Morgan Ames, people tag pictures on photo sharing websites like Flickr in order to let others know about the picture. In the taxonomy of people s motivation for tagging, research has identified that people tag pictures because they are keen to use these keywords to organise and communicate the description of the pictures to themselves and for others. The question of why people tag pictures is addressed by the myriad of research efforts that look into the motivations that moves people to annotate their online multimedia content. In this aspect, research (M. Ames, et al, 2011) has established that the primary motivations for multimedia annotations are for a person s own personal use as well as for social purposes. While there has been a lot of research done in the area of people s tagging motivation and behaviour answering the questions of why people tag there is a lack of research done to answer the question of how people tag pictures. In addressing this question, we would like to know how people s choice of words is dependent on their knowledge of the picture and its owner, as well as where they are coming from their background information. This background information can only come from the personal attributes of the person such as their nationality, gender and age group. For example, in tagging a picture of a place in Australia, an Australian will tag it differently from an Indonesian because of the lack of information that the Indonesian might have of the place. The contributions of this research are two folds. Firstly, this research effort produces a tool in the form of a Facebook App for researchers to investigate the words people use to tag pictures in relation to their personal attributes as mentioned above. We do this by collecting people s tags on pictures as well as their personal information from Facebook. Using this tool, we will also be able to see how friendship on Facebook will affect a person s tag most notably the contextual tags that the person would produce. Page 4 of 19

5 2 Background At the beginning of this research, we hypothesised that being a friend on Facebook will provide a person with some needed information in order to provide a user with some contextual tags to a picture. Contextual tags are tags that provide information on when, where, how and why a photo was taken (Xie, et al, 2010). This is unlike the visual tags that are generic words to describe the relevant picture. This difference is illustrated in the diagram below. Figure 1: Social Network Diagram In the diagram above, we can see how different people will tag a particular picture in different ways. Victor, being an owner of the picture, describes the picture with the words girl, Mary and Italy. Being an owner of the picture, Victor knows that the picture is a painting of his friend Mary that he saw in Italy. Another person who decided to put his tag on the picture was John. John is a friend of Victor and is thus aware that it is a picture of Mary. This eventually causes him to tag the picture with the words girl, Mary and pretty. As we can see in the diagram above, John s tags are very different from that of Sam who is a stranger to this picture. The word stranger here simply means that Sam is not in any way related to the owner of the picture or the picture itself. In the diagram, we saw how Sam tagged the picture with words girl and photo. Sam s tags are purely visual tags because he does not have any contextual knowledge of this picture the fact that the person in the picture is Victor s friend, Mary. Page 5 of 19

6 The above description shows the hypothesis of this research. Through a Facebook app, we aim to prove using empirical data that being a Facebook friend of the owner of the picture will give a person more knowledge than purely visual cues that allow him to produce contextual tags for a picture. Page 6 of 19

7 3 Related Work Problems related to image annotation and descriptions have been tackled from different ends by many communities. The image retrieval and computer vision community has created many algorithms and systems for performing image annotation and object recognition; the web, humancomputer interaction communities have been engaged in studying user intention and usage of social tagging systems; information from social sharing, media content and ontology are then used to improve and revise photo tags (Xie et al, 2010). In the process of finding inspiration for this Facebook app, several apps related to image tagging were explored. Most notable among those explored was the ESP game, which solves the problem of the many untagged pictures online. The ESP game (von Anh, L, 2006) was developed in order to solve the problem of having the many images on the web that are not annotated. Since it is a very engaging game, the user ends up playing it for prolonged periods of time, thus annotating many images. We also looked at the Name-It-Game (C. Snoek and J. Steggink), which adds semantic annotation into the picture tagging game. The Name-It-Game extends the work of ESP game by adding semantic meanings to the words that are used. This is useful because many words can have several different meanings and adding semantic information will improve tagging information. Looking at the above two previous works in this area has helped the research efforts in two areas. Firstly, looking at the applications provides some ideas as to what makes online application popular. The game and score and achievement element of Name-It-Game shows the importance of having similar attributes of scores and players in our app. Secondly, one of the factors that influence the success of the ESP is its unintended social dimension. Even though the two players are not aware of the details of the other, the fact that they would have to agree to a certain tag before they can both move on to the next picture provides them with an element of mystery and surprise that someone they know not who thinks the same way as them. In our application, we would like to leverage Facebook s social networking facilities in order to ensure that there is a social element. This will be explained further in later sections. Page 7 of 19

8 4 Method Step 1: Software design and Human Ethics approval Step 2: Gathering users attributes from Facebook Step 3: Gathering users public photos from Flickr Step 5: Data analysis Step 4: Gathering users tags Figure 2: Research process The diagram above shows the steps taken to conduct the research. As mentioned earlier, the goal of this endeavour is to create a tool, which will allow researchers to investigate how being a friend on Facebook will provide users with more information and lead to the providing of contextual tags. 4.1 Step 1: Software design and approval from the Human Ethics Committee The first step in the research process is to design the app both in the front- and back-end such that it can carry out its mission. As depicted in the diagram below, the app consists of one main screen where the game is played. The idea is to do the data collection in an asynchronous manner using AJAX. Page 8 of 19

9 Figure 3: Application mock design In addition to designing the user interface, we also looked at the data that had to be collected by the application. In addition to that, the tools and technologies used were considered. In the view that the tools have to integrate with Facebook s API (the URL address in the reference section), we decided to use PHP as our programming language in order to leverage on Facebook s PHP SDK. We also decided to use JQuery in order to facilitate the AJAX processes and MySQL as our back-end database. Apart from technical considerations, we also considered the information that we are going to collect from the users. As a standard practice at the university, it is imperative that we seek approval from the Human Ethics Committee before collecting any data. In the application, the pieces of information that we required from the users are: (1) Name (2) Gender (3) Location (4) Hometown (5) Age Page 9 of 19

10 (6) (7) List of their friends Facebook ID There was a bit of a problem with the approval for the list of the users friends Facebook ID. However, it is imperative that we collect this piece of data because it is the only way to know the friendship between the users of the app. Step 1: Allow Tag Expert to access his / her data on Facebook Step 2: Enter his / her Flickr NSID Step 3: Start tagging! Figure 4: Users process when using the app The app is also designed with users simplicity in mind. As shown in the diagram above, to use the app the users have to only undergo a three-step process. For the user, the app itself only consists of one usable screen, though we have other pages to store other information on the app such as the privacy policy, terms and conditions and contact information of the developers. 4.2 Step 2: Gathering user attributes from their Facebook information. The main reason for making it a Facebook app is the consequential ability of the app to extract information about its user from Facebook. These pieces of information are crucial in our analysis of the users personal attributes in relation to their tags. In addition to the personal attributes, the app also gathers the Facebook userid of the friends of the friends of the user. This information is stored in a directed graph. Before storing the information, the app will ask for authorisation, which is a standard practice for all Facebook apps. This is done to ensure that the user has fully consented in getting their personal data captured by the application. The diagram below shows the MySQL database schema that is used in capturing the data from the application. The database schema consists of four tables that stores information about: (1) the users, (2) the friendships of the users, (3) picture details (4) tags Page 10 of 19

11 Table user friend photo tag Field id username name location_name gender hometown_name flickr_nsid flickr_username Purpose to keep the data of the users and their personal attributes to store data about the friends of the users to store data about the photo details to store data of picture tags Table: user Description Users Facebook userid Users Facebook username Users name Users current location Users gender Users hometown Users Flickr NSID Users Flickr username Table: friend Field person1_facebook_id person2_facebook_id Table: photo Field id (auto-incremented) owner_facebook_id owner_flickr_nsid picture_flickr_farm_id picture_flickr_server_id picture_flickr_photo_id picture_flickr_secret picture_type picture_facebook_id picture_facebook_source Figure 5: Database Schema We should note here that before gathering all the information, there would be a standard Facebook authorisation dialog box, which would ask the user if he would authorise the App to extract the relevant data. Due to the lack of Facebook development experience, the author took a long time to learn about Facebook s authorisation system. In addition, Facebook also requires that the server used by the application supports HTTPS functionalities and is configured in a specific manner, which also took quite a while to grasp. Facebook also requires that the app would provide the users with the terms and conditions and privacy policies. Page 11 of 19

12 Figure 6: Facebook s JSON response 3. Request for iframe Client (the User) 1. Initial user request Facebook Server 4. API call request App Server (at ANU) 2. HTML response 5. API response 6. iframe response Figure 7: Architectural Link between Tag Expert, Facebook, and App Server Page 12 of 19

13 In the technical spectrum, we can see in the diagram above how the app will communicate with the servers of Facebook and the client. The application is hosted on our own server and it receives the client s requests. Then it will request to Facebook via the Facebook Graph API and receive a JSON response. This JSON response will then be processed and stored into the app database on our server. 4.3 Step 3: Gathering users public photos from Flickr Facebook Graph API request Flickr API request Facebook Tag Expert Flickr JSON response JSON response Figure 8: How Tag Expert, Facebook and Flickr communicate The next step is to gather the public photos of the user through Tag Expert Facebook App. For the current first version of the application, the user have to enter his / her Flickr NSID (ref. Flickr API). With the NSID, the app will then request to Flickr the public photos of the user. Figure 9: How picture details are stored in the database In the diagram above, we can see how the several pieces information such as the farm id, server id, Flickr picture id and secret of the picture are stored. Using these information, we will be able to retrieve the picture from Flickr s URL which is in the form of (Flickr URL documentation). Page 13 of 19

14 As seen in the diagram above, one of the information stored about the picture is the Facebook userid of its owner. This is done so that when the application starts up, it will look for pictures belonging to the people who are the Facebook friends of the user. 4.4 Step 4: Gathering user tags Figure 10: App design with user tagging facilities As of the current iteration of the app, the tagging facility is under development. As shown in the app design picture above, the aim is to allow the users to insert 4 tags for each picture. Each tag can only be made with one word. Using a PHP function, the server will only take in the first word of each tag. This is only for the first iteration and we will definitely look into improving this function in later releases of the app. Page 14 of 19

15 Field id picture_id owner_id tagger_id tag_type Table: tag Description Tag ID (auto-incremented primary key) ID (generated by Tag Expert) of the picture Facebook ID of the owner Facebook ID of the tagger Either friend or stranger Figure 11: Table to store tag data We can see in the picture above how the app stores the picture tags from the users. As we can see in the above diagram, in addition to the tag-word, the app also stores the Facebook userid of the tagger, the owner of the picture, the picture id and the status of the tagger. This status field is used to indicate whether or not the person who tags is a stranger or a friend of the owner of the picture. This field will make it really easy later on for us to separate the tag data into those coming from the friend and non-friend of the owner. 4.5 Step 5: Data analysis Due to the limited time after development, we were not able to analyse the data gathered through the app. However, the design of the database was such that data analysis was made easier because there would not be too much need for too many SQL inner-joins in order to make queries to the database. A proposed suggestion on conducting the data analysis would be for each picture, we would list down the tags gathered some of them would be purely visual while others will be contextual. We would then only look at the contextual tags and see how the percentage of these tags that comes from friends of the owner of the picture and the percentage that comes from non-friend of the owner of the picture. Page 15 of 19

16 Figure 12: A suggestion on how data analysis can be carried out Page 16 of 19

17 5 Conclusion Figure 13: Tag Expert at its current iteration The original aim of this project was to develop a tool that would allow researchers to investigate how people s friendships might affect the contextual tags that they produce. This project has achieve some parts of it but not all. This is mainly due to the time constraints. It was very hard to estimate the amount of time needed in the beginning, which caused our estimation to be over optimistic. However, in this project we have got a starting point in the form of a tool, which will allow us to begin our investigation. Also, this project has also provided a slight user interface improvement from the last project. Page 17 of 19

18 6 Future work Throughout the descriptions above, there is undoubtedly many parts where effort in furthering this is needed. One of the most prominent areas is data analysis. While this Facebook app facilitates data collection, it does not have any mechanism for data analysis. It would be very helpful to have functions in the app that will allow for query to be processed and the result displayed in a presentable manner. Another area in which improvements can be made in this project is to begin investigation on how Facebook friendship affects tags on specific kinds of pictures. In this project, we have selected the pictures at random. It would be very helpful for researchers to have a way to know what specific kinds of pictures would be most greatly affected by contextual knowledge of the taggers. Page 18 of 19

19 References Z. Qin; How do people tag pictures? A pilot study with Facebook application. June L. Li; How do people tag pictures? A pilot study with mobile application. June Facebook Graph API. Available online at Flickr Developer API. Available online at Flickr URL Documentation. Available online at M. Ames and M. Naaman, Why we tag: Motivations for annotation in mobile and online media. In proc. CHI, pages , Wikipedia (2011), Facebook. Available online: http// L. Xie, A. Natsev, M. Hill, et al; The Accuracy and Value of Machine-Generated Image Tags. In proc. CIVR, von Ahn, L.;, "Games with a purpose." Computer, vol.39, no.6, pp.92-94, June C. Snoek and J. Steggink, Adding semantics to image-region annotations with the Name-It-Game, Multimedia Systems, vol. 17, no. 5, pp , October Page 19 of 19