Policies to Resolve Archived HTTP Redirection

Transcription

1 Policies to Resolve Archived HTTP Redirection ABC XYZ ABC One University Some city ABSTRACT HyperText Transfer Protocol (HTTP) defined a Status code (Redirection 3xx) that enables the user agent to redirect the requested URI to another destination. The web archive crawler used to crawl the status and keep the redirected URI in its seed list to be crawled later. The current trend in retrieving URI or mementos with HTTP redirection status does not take the redirection into consideration. In this paper, we ran a quantitative study to conclude a set of policies that will help the client to reach the memento in the existence of HTTP redirection. The results indicated that 14% of the current web had HTTP redirection status additional to 7% of the mementos had redirection status. The first proposed policy successfully resolved 17 URIs out of 77 URIs that did not have mementos but had HTTP redirection status. The second proposed policy helped the client to get the nearest memento to the requested date/time especially in the multi-archive environment. client should do extra steps to get the content of this resource. On the web archiving paradigm, the client used to apply the same procedure to retrieve the archived snapshot for the URI even if this URI has a redirection status without a consideration to the temporal nature of the content. For example, URI ( is the generic bit.ly for so it always retrieve 301 Response status code with location to curl -I HTTP/ Moved Location: When the user tried to get the Timemap for this bit.ly URI, it is not archived at all (figure 1). Categories and Subject Descriptors H.3.7 [Information Storage and Retrieval]: Digital Libraries General Terms Design, Experimentation, Standardization Keywords HTTP Redirection, Temporal Retrieval, Memento 1. INTRODUCTION Hypertext Transfer Protocol (HTTP/1.1) [6] Status code defined (Redirection 3xx) as This class of status code indicates that further action needs to be taken by the user agent in order to fulfill the request. The action required MAY be carried out by the user agent without interaction with the user if and only if the method used in the second request is GET or HEAD. A client SHOULD detect infinite redirection loops, since such loops generate network traffic for each redirection. Based on this definition the HTTP Figure 1: bit.ly/1l2iwb Timemap But, if the user tried to follow the redirected link (www. amazon.com), the user could retrieve mementos. curl link/ < / rel="first memento";datetime="sat, 12 Dec :25:32 GMT",

2 < / rel="memento";datetime="mon, 25 Jan :31:56 GMT", The situation became more complicated for the memento with HTTP redirection status. If the user wanted to retrieve the memento for on May 2008, he will discovered that the timemap have only three mementos on (27 Mar 2009), (08 Nov 2010) and (09 Nov 2010). Then, he should select one of these mementos which are far from his interested date. Actually, these three mementos had HTTP redirection to If the user tried to visit the mementos, he could find a set of mementos in his interested date. We can use the Memento protocol to do the content negotiation in the date/time dimension by adding an extra header for the Accept-Datetime. The Memento Timegate will respond with the nearest memento to (Jan 1, 2006) which is (Mar 27, 2009), then it redirects to a memento for http: // on (Mar 27, 2009). curl -I -L -H "Accept-Datetime: Thu, 01 May :00:00 GMT" HTTP/ Moved Temporarily Location: / HTTP/ Moved Permanently Location: / If the client considers the HTTP redirection status for the memento, it could redo the content negotiation in the date/time dimension for the redirected URI. It will bring the nearest memento to (Jan 1, 2008). curl -I -L -H "Accept-Datetime: Sun, 01 Jan :00:00 GMT" HTTP/ Moved Temporarily Location: memento/ / In these two examples, the client s awareness with the redirection status provided a new approach to reach a fine-grain date/time. The goal of this paper is to find a policy rules that will help the client to use the HTTP redirection to get a better estimate to the requested memento. In this paper, we will cover the different cases for the HTTP redirection in the current and the past web (section 3 and 4). Section 5 will describe the experiment with the initial results. Section 6 will discuss the redirection policy for the URI and memento. 2. RELATED WORK 2.1 HyperText Transfer Protocol (HTTP) Hypertext Transfer Protocol (HTTP) is an application protocol for distributed, collaborative, hypermedia information systems [6]. HTTP is the protocol behind of the World Wide Web (www). HTTP depends on Request-Response paradigm in the Client-Server environment. HTTP Request is the message from client to server. The first line is called the Request-Line that begins with a method token, followed by the Request-URI and the protocol version [6]. GET /index.html HTTP/1.1 Accept: image/gif, image/x-xbitmap, image/ jpeg, */* Accept-Language: en-us Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT) Host: example.org Connection: Keep-Alive HTTP Response is the reply message from the server to the client. The response message starts with the status line. Status line starts with the HTTP protocol version, the status code and Reason Phrase. HTTP/ OK Date: Mon, 06 Dec :54:26 GMT Server: Apache/1.3.6 (Unix) Last-Modified: Fri, 04 Oct :06:11 GMT ETag: "2f5cd e1bd6" Accept-Ranges: bytes Content-length: 327 Connection: close Content-type: text/html The status code represents the class of the response. 1xx: Informational - Request received, continuing process. 2xx: Success - The action was successfully received, understood, and accepted. 3xx: Redirection - Further action must be taken in order to complete the request. 4xx: Client Error - The request contains bad syntax or cannot be fulfilled. 5xx: Server Error - The server failed to fulfill an apparently valid request [6]. 3xx Redirection class used when the document has moved and the server is configured to tell the clients where it has been moved. The popular status codes in this category are: 301 (moved permanently) which means the requested URL is no longer used by the server, 302 (Found) (deprecated) and 307 (Moved Temporarily) are used when the requested URL has moved but temporarily [9]. These redirection responses have Location header that specifies the new location. 2.2 Web Archiving Library of Congress defined the Web Archiving [2] as Web archiving is the process of creating an archival copy of a website. An archived site is a snapshot of how the original site looked at a particular point in time. The web archiving workflow started with the selection of the web resources to be archived, then the collection of these resource to be ready for preservation. Preservation is storing the content for future usage. Finally, the delivery is the process to allow the user to access the web archive content [4]. Heritrix crawler [3] is an open source web crawler that is used by Internet Archive 1 to take a periodical snapshots of the Web. Heritrix keeps track of the resource s HTTP status including the HTTP redirection status. It keeps a record with the Location header to be used later in the retrieval process. 2.3 Memento Memento [1] is an extension for the popular HyperText Transfer Protocol (HTTP) to allow the user to browse the past 1

3 web as the current web. Memento extends HTTP content negotiation [5] to be in the datetime dimension, a new HTTP headers proposed a special header Accept-Datetime [7]. GET / HTTP /1.1 Host : odu. edu Accept - Datetime : Sat, 17 Dec :00:00 GMT TimeGate [7] for an Original Resource is a resource that supports negotiation to allow selective, datetime-based, access to an archived copy of this Original Resource. TimeMap for an Original Resource is a resource from which a list of URIs of Mementos of the Original Resource is available. A TimeMap Aggregator [8] (for simplicity Aggregator) harvests and merges TimeMaps; the Aggregator exposes its own TimeGates and TimeMaps. The Memento Aggregator provides TimeGates and TimeMaps inter-archives, with finer datetime granularity after merging the different TimeMaps. 3. TEMPORAL TIMEMAP REDIRECTION A TimeMap for an Original Resource is a resource from which a list of URIs of Mementos of the Original Resource is available. TimeMap does not consider the preserved HTTP status for the memento. If we extend the TimeMap to include the archived HTTP status for each memento, we may get three different categories: All redirection to the same resource The timemap TM for the resource X is a list of the mementos that represent the snapshots for the resource X where each one of these mementos have a redirection status to and only to resource Y. T M(R X ) = {M X (T 1 ) [status : 3xx], M X (T 2 ) [status : 3xx],... M X (T n) [status : 3xx]} M X (T i ) [status : 3xx] M Y (T i ) This category describes this set of URIs that have not changed in the past. For example, bit.ly/xxx URIs do not change through the time. All redirection to different resources The timemap TM for the resource X is a list of the mementos that represent the snapshots for the resource X where each one of these mementos have a redirection status to a set of URIs. T M (R X) = {M X (T 1) [status : 3xx], M X (T 2) [status : 3xx],... M X (T n) [status : 3xx]} M X(T i) [status : 3xx] M Y (T i) or M Z(T i) Timemap with miscellaneous status The timemap TM for the resource X is a list of the mementos that represent the snapshots for the resource X where each one of these mementos may or may not have a redirection status. T M(R X) = {M X(T 1) [status : xxx], M X(T 2) [status : xxx],... M X(T n) [status : xxx]} Figure 2 explains the different categories. 4. URI-MEMENTO REDIRECTION RELA- TION The relation between the the original resource (URI-R) and the memento (URI-M) could be classified into five cases based on the HTTP status for both of URI-R and URI-M. Case 1: URI-R has HTTP status (OK) in the current Web and URI-M has HTTP status (OK) too. Case 2: URI-R has HTTP status (OK) in the current Web and URI-M has HTTP status (Redirection). Case 3: URI-R has HTTP status (Redirection) in the current Web and URI-M has HTTP status (Redirection) and both of them redirected to the same URI. Case 4: URI-R has HTTP status (Redirection) in the current Web and URI-M has HTTP status (Redirection) but both of them redirected to a different URI. Case 5: URI-R has HTTP status (Redirection) in the current Web and URI-M has HTTP status (OK). Table 1 lists these cases. Figure 4 illustrates these cases. 5. EXPERIMENT To be able to conclude the different redirection following policies for each case, we ran a quantitative study to estimate the percentage of happening for each case. We sample 10,000 URI randomly from Open Directory Project (DMOZ) 2. The experiment focused on drawing a complete graph of the HTTP status for the original resources that were extracted from DMOZ, the mementos for each resource, the redirected location for the resources or mementos that carried HTTP status redirection (3xx). We used the algorithm 1 to extract the HTTP status for the resources. The experiment started with URI-R and performing two main steps: HTTP status discovery and Timemap discovery. In the first step, HTTP status discovery (line 2-6), we did HEAD request for the URI-R to get the HTTP status (URI-R[Status]). For URI- R with HTTP status (Redirection), we retrieved the redirection location (URI-R[Redirected]), then get its timemap URI-T[Redirected]. The second step, Timemap discovery (lines 7-13), we retrieved the timemap (URI-T[Original]) for each URI-R. For each memento (URI-M), we did HEAD request to get the HTTP status URI-M[Status]. For the mementos with HTTP redirection, we will get the redirection location URI-M[Location]. Table 2 shows the distribution of HTTP status on the current web for our 10,000 sampled URIs. The result shows that 14.71% of the URIs have redirection status. This percentage could be divided to 8% with status code 301 (Moved Permanently) and 6% with status code 302 (Found). 2

4 (a) Category 1: All the mementos have redirection to the same URI (0.69%). (b) Category 2: All the mementos have redirection to different URI (0.30%) (c) Category 3: The mementos have different HTTP status (27.91%). Figure 2: Timemap Redirection Categories. Case Number URI-R HTTP Status URI-M HTTP Status Case 1 Non-Redirectin Non-Redirection 78.66% Case 2 Non-Redirectin Redirection 2.67% Case 3 Redirection to R x Redirection to R x Mementos 1.31% Case 4 Redirection to R x Redirection to R y Mementos 1.30% Case 5 Redirection Non-Redirection 13.37% Table 1: RedirectionCases

5 (a) Case 1 (78.66%) (b) Case 2 (2.67%) (c) Case 3 (1.31%) (d) Case 4 (1.30%) (e) Case 5 (13.37%) Figure 3: HTTP Status/Code Percentage (10,000 URI-R) OK (200) Redirection (3xx) Not-Found (4xx) 1.18 Others 1.28 Table 2: Sample URI Current HTTP Status 6. RESULTS AND ANALYSIS After retrieving the timemap for the 10,000 URIs from the memento aggregator. We started to quantify the two main steps in algorithm 1. HTTP status discovery: Table 2 shows that 14.71% of the original URIs had HTTP redirection status. For these 1471 URI-R, we followed the redirection location URI-R[Redirected] and retrieve its timemap URI-T[Redirected]. Timemap Discovery: Using memento aggregator, we were able to discovered 894,717 mementos. The HTTP status for the memento list is shown as table 3. The table shows that around 6% of the mementos have HTTP redirection status which means you need to follow this link to get the archived content. HTTP Status/Code Percentage (894,717 URI-M) OK (200) 93.46% Redirection (3xx) 5.69% Not-Found (4xx) 0.26% Others 0.59% Analysis Table 3: Mementos HTTP Status To draw the complete graph between the present and the past, we quantified the different cases that were listed in section 3 and 4. First, We studied the timemap cases (section 3). We resolved the HTTP status for each mementos in the timemap, and group them by status as shown in table 4. Based on these results, we can quantify the different categories in section 3 as the following:

6 Algorithm 1 Memento Redirection Status Algorithm Require: Set of URI-R extracted from DMOZ. 1: for all URI-R do 2: URI-R[Status] Get HTTP Status for URI-R. 3: if URI-R[Status] == 3xx then 4: URI-R[Redirected] Get the Redirected Location for URI-R. 5: URI-T[Redirected] Get the Timemap for URI-R[Redirected] using memento aggregator. 6: end if 7: URI-T[Original] Get the Timemap for each URI-R using memento aggregator. 8: for all URI-M in URI-T[Original do 9: URI-M[Status] Get HTTP Status for URI-M. 10: if URI-M[Status] == 3xx then 11: URI-M[Redirected] Get the Redirected Location for URI-M. 12: end if 13: end for 14: end for 6465 timemaps have all the mementos with HTTP status OK (out of scope of our research) 2791 timemaps have mementos with different HTTP status (Timemap case 3) 99 timemaps have all redirection HTTP status (Timemap case 1 and 2) Timemap Category Percentage All Mementos have OK status 64.65% Mementos have mix status 27.91% Redirection to the same URI 9.97% Redirection to different URI 17.72% All Mementos have Redirect status 0.99% Redirection to the same URI 0.69% Redirection to different URI 0.30% URI has no Mementos at all 6.45% Table 4: URI-Memento relation results In this section, we will put a new policy for resolving the HTTP Status redirection URI. We start with the current behavior in the Wayback Machine. Then, we will give two policies: Policy one, Original resource with a redirection and policy two, memento with a redirection. 7.1 The Current Wayback Machine behavior Wayback Machine is a url search based browser for the web archiving data. The user could start with a URI and the wayback machine could give him the available snapshots through the time. Also, Wayback Machine is able to replay the resource as it appears in the past. When Wayback Machine receives a query of URI that has a HTTP status redirection, it does not take the status of the URI in consideration and starts to search for the mementos for this URI. To replay a memento with HTTP redirection status, Wayback Machine displays a message with Got HTTP 3xx response code at crawl time, then it redirects to a memento for the redirected URI. We compared between the timemap HTTP status (the set of HTTP status of the mementos in the timemap) with the HTTP status of the original resource in the current web (table 2. Table 5 shows the relation between the status on the current web and the status of the timemap. Even we had 1471 URIs have HTTP status redirection in the current web, only 83 timemap had HTTP redirection status for all the mementos, while there were 425 timemaps have HTTP status OK in all the mementos. We can conclude that the HTTP status on the current web could not give us an indication about the status of the timemap because the URI HTTP status could change through the time without any rules. During the experiment, we were not able to conclude a pattern for the URI HTTP status change. The next step was following the memento redirection to compare the original redirection with the memento redirection. Table 1 listed the different cases of the relation between the original resources with the percentage of each case from the experiment. 7. ARCHIVED HTTP REDIRECTION RE- TRIEVAL POLICIES Figure 4: Wayback Machine is replaying a memento with HTTP status redirection 7.2 Policy one: URI-R with a redirection

7 URI Status URI count Timemap Status count(10,000) Mix status 1849 OK 8283 All 200 status 5886 All Redirect status 14 Mix status 880 Redirection 1471 All 200 status 425 All Redirect status 83 Mix status 32 Not-found 118 All 200 status 75 All Redirect status 2 Others 128 All 200 status 79 Mix status 30 Table 5: Timemap status comparing to the URI-R status on the current web. In this case, we have URI-R that redirects to another URI- R[Redirected] (it appeared 1471/10000 URIs). We have three cases: Case three, four and five (see table 1). We could apply the following policy: 1. Starting with URI-R. 2. (a) If the retrieved memento has (HTTP Status OK), then return this memento. (Stop) (b) Else if the retrieved memento has (HTTP Status Redirection), then repeat step one with the new memento original URI. (c) If the retrieved memento has (HTTP Status 404) URI-M[Redirected][Original]. and URI-R i has more redirection, set R as Redirection(URI- R i) then go to step Policy two: URI-M with a redirection Here, we will address the case two (see table 1). Here, we may have two cases: In this example, URI-M[Original] = bit.ly/2eejblwhile URI-M[Redirected][Original] = cnn.com. The client should be able to determine each case based on the comparison between the original resource for each URI- M and URI-M[Redirection]. If they are the same, then it is (Internal redirection), otherwise it is (Archived redirection). We defined different policy for each case. For the first case, internal redirection, the client simply will follow the redirection. For the archived redirection, the client will repeat the content-negotiation in the datetime dimension for the We summarized both policies in algorithm 2. Figure 5 illustrates a flowchart for both policies. Based on this new policy, we were able to retrieve mementos for 17 URIs out of 77 URIs that have no mementos at all but carried redirection status and the redirected URI has mementos. 1. Internal redirection to closer date: Memento redirects to another Memento with the same original. Redirection from a memento for On (Jan 01, 2010) to another date (Jan 21, 2010) In this example, URI-M[Original]=URI-M[Redirected][Original]=cs.odu.edu 2. Archived redirection status: Memento redirects to another Memento with a different original. Redirection from a memento for On (Nov 09, 2010) to another memento for on the same date CONCLUSION In this paper, we discussed the different scenarios for the HTTP Redirection status in the current and the past web. Following the redirection in the past web needs special handling other-than the current web. The paper provides redirection retrieval policy for web archived client when discovering redirection on original or memento. 9. REFERENCES [1] Memento: Adding time to the web. [2] Web archiving faqs. http: // [3] Heritrix Crawler, [4] A. Brown. Archiving websites: a practical guide for information management professionals. Facet, London, first edit edition, [5] K. Holtman and A. Mutz. RFC Transparent Content Negotiation in HTTP, [6] K. Holtman and A. Mutz. RFC Hypertext Transfer Protocol, [7] H. Van de Sompel, M. L. Nelson, and R. Sanderson. HTTP framework for time-based access to resource states. draft-vandesompel-memento/, 2011.

8 Algorithm 2 Memento Redirection Policy Require: URI-R is an original resource that redirects to URI-R[Redirected]. 1: while TRUE do 2: Do the content-negotiation for URI-R in the datetime dimension. 3: if URI-M[Status] == 2xx then 4: Return URI-M [EXIT] 5: else if URI-M[Status] == 3xx then 6: if URI-M[Redirection][Original] == URI-R[Redirection] then 7: Set URI-M URI-M[Redirection], Follow the redirection [Internal Redirection] 8: else 9: Set URI-R as URI-M[Redirection][Original] [Repeat] 10: end if 11: else if URI-M[Status] == 4xx then 12: if URI-R has URI-R[Redirection] then 13: URI-R URI-R[Redirection] 14: elsereturn Resource Not Archived [EXIT] 15: end if 16: end if 17: end while Figure 5: Redirection Retrieval Policy

9 [8] H. Van de Sompel, M. L. Nelson, R. Sanderson, L. Balakireva, S. Ainsworth, and H. Shankar. Memento: TimeMap API for Web Archives. slides/memento_201002_timemap.pdf, [9] C. Wong. Http Pocket Reference. O Reilly & Associates, Inc., Sebastopol, CA, USA, APPENDIX Terminologies URI-R is used to denote the URI of an Original Resource. An Original Resource is a resource that exists or used to exist, and for which access to one of its prior states is desired. URI-R[Redirected] URI-T, TM X is used to denote the URI of a TimeMap. A TimeMap for an Original Resource is a resource from which a list of URIs of Mementos of the Original Resource is available. URI-M, M X is used to denote the URI of a Memento. A Memento for an Original Resource is a resource that encapsulates a prior state of the Original Resource. A Memento for an Original Resource as it existed at time T j is a resource that encapsulates the state that the Original Resource had at time T j. URI-M[Redirected] URI-M[Original] URI-M[Redirected][Original] URI-X[Status] URI-G is used to denote the URI of a TimeGate. A TimeGate for an Original Resource is a resource that is capable of negotiation to allow selective, datetimebased, access to prior states of the Original Resource.