Aggregating services. Christoffer Björkskog. Helsinki February 13, 2008 Master s Thesis Chapter UNIVERSITY OF HELSINKI Department of Computer Science

Transcription

1 Aggregating services Christoffer Björkskog Helsinki February 13, 2008 Master s Thesis Chapter UNIVERSITY OF HELSINKI Department of Computer Science

2 HELSINGIN YLIOPISTO HELSINGFORS UNIVERSITET UNIVERSITY OF HELSINKI Tiedekunta/Osasto Fakultet/Sektion Faculty Laitos Institution Department Faculty of Science Tekijä Författare Author Christoffer Björkskog Työn nimi Arbetets titel Title Department of Computer Science Aggregating services Oppiaine Läroämne Subject Työn laji Arbetets art Level Aika Datum Month and year Sivumäärä Sidoantal Number of pages Master s Thesis Chapter February 13, pages + 0 appendix pages Tiivistelmä Referat Abstract Abstract to be written... Avainsanat Nyckelord Keywords mobile media, context-awareness, web 2.0, aggregating services Säilytyspaikka Förvaringsställe Where deposited Muita tietoja övriga uppgifter Additional information

3 Contents ii 1 Introduction 1 2 Syndication RSS and Atom Aggregating content Screen Scraping 3 4 Mashups The architecture of a mashup site The process of building a mashup User built mashups Conclution 11 References 11

4 1 Introduction An aggregate is a whole formed by several elements [dic]. Combining sources from different web pages into what is called aggregated documents is a common phenomena on the internet today [ES08]. Information from different sources is assembled into a whole. Millions of blogs (online diaries in reversd chronological order) and a huge amount of online newspapers and websites are syndicated, providing their content in a machine readable format by providing feeds that enable users and applications to access the data. The most common way of aggregating syndicated content is through search engines and RSS (Really simple Syndication) readers. A mashups is a new way of aggregating content. Mashups takes elements and information from different sources that are either retrieved through APIs (Application Programming Interface) or collected through other means such as screen scraping or feeds and combines the information so that it creates new and unusual or interesting use of the data. 2 Syndication Syndication is supplying feeds that contain content that others can subscribe to [GMR07]. An example of this is a blog that syndicate its contents via RSS feeds. Other people can then subscribe to this feed using a RSS reader. When the author has written a new entry to the blog the subscribers can be notified of the change. Syndication increases traffic to the web page and it builds a brand awareness for the webpage. Syndication can also enhance the web page s rankings in search engines [Ham05]. Syndication helps to maintain relationships between sites in a community and improves the relationship between the site and the users. Additional technology may enhance syndicated services for instance an application that observes a syndicated feed and notifies a subscriber via instance messaging. Syndication adds a richness to internet services and encourages a reuse of data while pushing the continuous development of semantic technology forward. Bandwidth is saved when screen scraping is reduced. This can be achieved by providing syndication feeds.

5 2 Figure 1: Certain web browsers have got a built in RSS 2.1 RSS and Atom RSS is a collection of XML (Extensible Markup Language) file formats that summarizes the contents of a webpage [BGS06]. The concept emerged in 1997 and the first version of RSS emerged in 1999 [GMR07]. Several versions of RSS exist due to forks in the development of the standard. The common versions that are widely used, but each a bit different from each other are RSS 1.0, RSS 2.0 and Atom. These have their own specific uses as well as their own advantages and disadvantages for feed publishers. RSS is especially used by sites where new content is added regularly, for instance news sites and weblogs. The amount of sites that support RSS is increasing rapidly [BGS06]. RSS and Atom are widely established within the weblogging community [Ham05]. There are millions of weblogs, personal online diaries, being written worldwide and most of these produce a syndication feed such as RSS or Atom.

6 3 2.2 Aggregating content Syndicated content can be aggregated, brought together into one place using for instance RSS readers. Desktop, web based and mobile RSS readers exist enabling an overview of syndicated content. The earliest ways of aggregating and reading RSS feeds were through web-based readers. Web based RSS readers provide a convenient way of staying up to date with the content of the web pages of interest without having to sit at the own personal computer. A popular web-based aggregator is bloglines [Blo08] Google has released an RSS reading service called Google reader that has gained popularity. 3 Screen Scraping If the content providers do not provide an API or a feed to access the data, mashup developers may need to resort to screen scraping in order to retrieve certain information [Mer06]. This means that content that was originally intended for human consumption can be parsed and analyzed in order to find sematic data structures representing it. This process is called Scraping. The information and data structures accuired can then be used to create a mashup. Screens scraping as data acquisition is used by a handful of mashups, especially when retrieving information from the public sectors this may be needed. An example of a mashup project that uses screen scraping is XMLTV which is a set of tools that aggregates TV program listings from all over the world. Screen scraping has atleast two drawbacks. Unlike APIs which are interfaces of how to access data, there is no specific programmatic contract between content-provider and content-consumer for screen scraping. The scrapers have to analyze the content model of the site to be scraped and design their tools to comply with that. If the content provider change their way of representing the content the tools of the scrapers are likely to stop working. Websites periodically update their look-and-feel in order to remain fresh and stylish and this causes maintenance headaches to the scrapers. Another issue is that it does not exist any screen-scraping toolkit software that is sophisticated and reusable emough, sometimes called scrapis. Why these APIs and toolkits does not exist is mainly because each scraping tool have extremely application-specific needs. Designers are forced to reverse-engineer content, parse and aggreagate raw data and create data models of the content.

7 4 Mashups 4 Web users today have transformed from being content consumers into content providers [RT08]. Today you can without knowing HTML create professional looking websites and blogs. That has not always been the case. A lot of services exist where users can contribute with content. The newest kinds of web tools combines, or enable users to combine content from different web sources into unique services. These are called mashups. A mashup is a web application that combines data from multiple web sources to provide a unique service. Mashups have become popular because of the emphasis on interactive user experience and the union of data provided by different services[mer06].there are a lot of different kind of handcrafted mashup solutions [AKTV07] and a few different classes of pupular mashups being developed [Mer06]. Examples of mashups are AdSense where ads related to the content of the page is embedded into the webpage [AKTV07]. See Fig. 4. Mashups aggregate and combine third-party data and takes advantage of data sources that usually comes from another site/service [Mer06]. The name comes from the popular music scene where the word mashup is used for the combination of the vocal and music tracks from two, usually belonging to different genres, songs. The most popular types of mashups are mapping mashups, video and photo mashups, search and shopping mashups, and news mashups. Mapping mashups One factor why mapping mashups, where data are presented graphically using maps, became popular was the release of Google Maps API, which let mashup builders present all kinds of data onto maps. After that, Yahoo, Microsoft and AOL released their own map APIs aswell. People are collecting and creating vast amount of data, a lot of these data have got some location indentifier associated with them. This means that they can be displayed on a map. There exists a lot of mapping mashups, some are collected and can be found from One example is the ChicagoCrime.org website which displays crime activity, fetched from the Chicago police s database, in Chicago onto a Google Map. Users can interact with the mashup site, having it for instance to display a map of south Chicago with the details of all recent burglary crimes shown graphically as pushpins on the map. See Fig. 3. Even thought the concept and the representation are simple it becomes visually powerful to combine crime and map data. The Google Maps API can be found from

8 5 html and describes how the developer can use Google Maps as part of a mashup. The following example code outputs a map, that is zoomed to the level 15, around the location on longitude and latitude (Which is the Kumpula campus of the University of Helsinki) and puts a marker there, see Fig. 2. <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" " <html xmlns=" <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"/> <title>google Maps JavaScript API Example</title> <script src=" type="text/javascript"></script> <script type="text/javascript"> //<![CDATA[ function load() { if (GBrowserIsCompatible()) { var map = new GMap2(document.getElementById("map")); // , is the latitude and longitude of kumpula var location = new GLatLng( , ); var kumpula = new GMarker(location); map.setcenter(location, 15); map.addoverlay(kumpula); } } //]]> </script> </head> <body onload="load()" onunload="gunload()"> <div id="map" style="width: 500px; height: 300px"></div> </body> </html>

9 6 Figure 2: The example code outputs this map This example is a modification of the simple example found from google.com/apis/maps/documentation/examples/index.html Video and photo mashups Some photo hosting services such as Flickr provide APIs to access shared photos [Mer06]. This has led to the emergence many interresting mashups. The content providers have metadata associated with the images that are hosted there. These metadata can i.e. be who took he picture, what the picture is of, when and where it was taken etc. Mashup developers can then use photos along with other information and match the metadata of the photos with other content. One example is a combination of the lyrics of a song with images related to the words in the song, or render the text of a webpage in photos by matching the metadata of the pictures with the words. Search and shopping mashups Before Web APIs were common and before the term masup was conied for combining web content, search and shopping mashups already existed. Services such as MySimon, BizRate, PriceGrabber and Google s Froogle aggregate price data that can be compared. They used a combination of screens scraping and business-to-buisness technologies to aggregate the data. Consumer marketplaces such as ebay and Amazon provide APIs to make it easier to create mashups and interresting web services. For instance Amazon enables mashup creators to get revenues from their mashups. For instance a mashup developer might

10 7 Figure 3: ChicagoCrime.org presents crime data on a map use Amazons API to enable users to create wishlists of things they would like to buy or recieve as gifts. If then someone via the site buys an item from Amazon, the mashup developer can get a certain percent of the purchase in revenue from Amazon. News mashups Certain news sites have been syndicating their content according to topics since Theses are for instance BBC, New York Times and Reuters, they have been using either RSS or Atom as means of syndication. Aggregatign mashups of syndicated feeds can create a personalized newspaper from the users preferred feeds accordig to the persons interrests. One example of a news feed mashup is Diggdot.us that mashes up feeds form Digg.com, Slashdot.org and del.icio.us. 4.1 The architecture of a mashup site API/Service provides The content providers may be unaware of that the content it is providing is used

11 8 Figure 4: Ads related to the text in a page can be embedded into a webpage via Google AdSense service. to create web mashups but they may also provide their contend for this purpose among others by providing APIs to access the content. The content may be provided through web protocols, for instance be REST, Web Services or RSS/Atom. Many potential data providers do not yet provide APIs for accessing the data. Mashup developers acquire information from these web sites through screen scraping. The web page ChicageCrime.org has Google and the chicago Police Departments as APIand Service providers. The mashup site The mashup site is where the service is hosted and the mashup logic resides. It is not necessarily where the mashup is executed. The page can be executed through dynamic content generation on the server, but also through client side scripting such as JavaScript or a combination if these which is usually the case. Mashup applications often use data that is provided to them form their own user base, this way at least one set of data is local. Complex queries may reqire processing that is not suitable on the client side, for instance a query that would have the mashup to show the "average purchase price for real estate bought by actors who have costarred in movies with Kevin Bacon". The client s browser The application is rendered graphically in the client s web browser and some processing takes place there. For instance the Google Maps API is intended for access via the browser using JavaScript.

12 9 Figure 5: The process of producing mashups [MMD06] 4.2 The process of building a mashup For complex mashups certain problems need to be solved [RT08]. See Fig. 5. The data source for mashups may vary in format structure and location [MMD06]. The source of the data could be a web page, a feed or even a PDF on a local filesystem. The data needs to be extracted into a manageable form [RT08]. If the data is dynamically extracted, the rules how to extract it needs to be solved. Data might span different locations and sources making it more difficult to manage. The data needs to be organized and classified [MMD06] to make it manageable and to find relationships between existing data sources and new ones [RT08]. The data may need to be transformed and cleansed in order to fix misspellings or to transform the data into the desired format. If the data retrieved has for instance the artist name Norah Jones in the format Jones, Norah, and another dataset has got the artists names in the former manner, a transformation is needed to integrate the new dataset so that it matches our existing one, or an address needs to be transformed into geographical positions. The transformation process may use other services for these transformations as well. In [RT08] a data integration step is presented in the process of creating mashups. That involves combining two or more data sets together. If for instance, one would build a mashup that would lists all the movies performed by this year s Oscar award winners, one would need to retrieve a list of this year s Oscar award winners and merge that using a database join operations on the winners names with a movie database. Data visualization takes the final data and represents it to the user in a certain way.

13 10 Figure 6: Data is gathered using xpath analysis where similar items as an example are found [RT08] 4.3 User built mashups There are different approaches that allow users to create mashups [RT08]. One is based on widgets where the user through widgets can manage different services and elements in the mashup. The other is based on data extracting through DOM (Document Object Model) analyzes. In [RT08] "Building Mashups by Example" a framework for building mashups is presented where users can without programming skills build customized mashups by providing examples that the framework can interpret and gather similar data. The DOM of the page is analyzed and similar items to the suggested one are extracted frm the document using XPath analysis. See Fig. 6. The data is cleaned in a way where the user modifies an example data into the desired form and the system interprets the change and comes up with a transformation how to change all the similar data into the desired format. Other means of creating mashups is by using widgets. Yahoo Pipes is a mashup building tool that lets users build mashups via widgets. These widgets produce an output that can either be entered into another widget as input or directed to the output of the mashup. Even though no programming is required, it may still be difficult for an average user to create a mashup since you need to have an underlying knowledge of certain programming terms and structure to utilize the features. To create a mashup several problems need to be addressed. These problems include gathering data from multiple web sources, cleaning the data and combining it.

14 11 Figure 7: Users can build custom mashups using widgets in Yahoo Pipes [RT08] 5 Conclution References AKTV07 BGS06 Blo08 dic ES08 Ankolekar, A., Krötzsch, M., Tran, T. and Vrandecic, D., The two cultures: mashing up web 2.0 and the semantic web. WWW 07: Proceedings of the 16th international conference on World Wide Web, New York, NY, USA, 2007, ACM, pages Blekas, A., Garofalakis, J. and Stefanis, V., Use of rss feeds for content adaptation in mobile web browsing. W4A: Proceedings of the 2006 international cross-disciplinary workshop on Web accessibility (W4A), New York, NY, USA, 2006, ACM, pages Bloglines, About Bloglines. IAC Search & Media, San Francisco Bay Area, URL: new oxford american dictionary. Evan Schrier, Mira Dontcheva, C. J. G. W. D. S., Adaptive layout for dynamically aggregated documents. IUI 08: Proceedings of the 13th international conference on Intelligent user interfaces, New York, NY, USA, 2008, ACM, pages

15 12 GMR07 Glotzbach, R. J., Mohler, J. L. and Radwan, J. E., Rss as a course information delivery method. SIGGRAPH 07: ACM SIGGRAPH 2007 educators program, New York, NY, USA, 2007, ACM, page 16. Ham05 Hammersley, B., Developing Feeds with Rss and Atom. O Reilly & Associates, April URL deveoprssatom/. Mer06 Merrill, D., Mashups: The new breed of web app, URL http: // html. MMD06 Murthy, S., Maier, D. and Delcambre, L., Mash-o-matic. DocEng 06: Proceedings of the 2006 ACM symposium on Document engineering, New York, NY, USA, 2006, ACM, pages RT08 Rattapoom Tuchinda, Pedro Szekely, C. K., Building mashups by example. IUI 08: Proceedings of the 13th international conference on Intelligent user interfaces, New York, NY, USA, 2008, ACM, pages