Modeling and Caching of Peer-to-Peer Traffic

Transcription

1 1 Modeling and Caching of Peer-to-Peer Traffic Osama Saleh and Mohamed Hefeeda School of Computing Science Simon Fraser University Surrey, Canada {osaleh, Technical Report: TR Abstract Peer-to-peer (P2P) file sharing systems generate a major portion of the Internet traffic, and this portion is expected to increase in the future. We explore the potential of deploying caches in different Autonomous Systems (ASes) to reduce the cost imposed by P2P traffic on Internet service providers and to alleviate the load on the Internet backbone. We conduct a measurement study to analyze P2P characteristics that are relevant to caching, such as object popularity and size distributions. Our study shows that the popularity of P2P objects in different ASes can be modeled by a Mandelbrot-Zipf distribution, and that several workloads exist in P2P systems, each could be modeled by a Beta distribution. Guided by our findings, we develop a novel caching algorithm for P2P traffic that is based on object segmentation and partial admission and eviction of objects. Our trace-based simulations show that with a relatively small cache size, a byte hit rate of up to 35% can be acheived by our algorithm, which is a few percentages smaller than that of the off-line optimal algorithm. Our results also show that our algorithm achieves a byte hit rate that is at least 4% more and at most triple the byte hit rate of the common web caching algorithms. Furthermore, our algorithm is robust in face of aborted downloads a common case in P2P systems and achieves a byte hit rate that is even higher than in the case of full downloads. I. INTRODUCTION Peer-to-peer (P2P) file-sharing systems have gained tremendous popularity in the past few years. More users are continually joining such systems and more objects are being made available, enticing even more users to join. Currently, the traffic generated by P2P systems accounts for a major fraction of the Internet traffic today [1], and is bound to increase [2]. The increasingly large volume of P2P traffic highlights

2 2 the importance of caching such traffic to reduce the cost incurred by Internet Services Providers (ISPs) and alleviate the load on the Internet backbone. Caching in the Internet has mainly been considered for the web and video streaming traffic, with little attention to the P2P traffic which has become the dominant bandwidth consumer. Many caching algorithms for web traffic [3] and for video streaming systems [4] have been proposed and analyzed. Directly applying such algorithms to cache P2P traffic may not yield the best cache performance, because of the different traffic characteristics as well as the caching objectives. For instance, the main objectives in web and video streaming systems are to reduce user s perceived latency, servers load, and WAN bandwidth consumption. While the objectives in P2P caching would mainly be to reduce the WAN bandwidth consumption, since there are no servers in P2P systems and latency is not a major concern for clients. In addition, unlike web objects, objects in P2P systems are mostly immutable, have large sizes, and their popularity does not follow Zipf-like distribution [1]. Furthermore, although objects in P2P and video streaming systems share some characteristics (e.g., immutability and and large size), streaming systems impose stringent timing requirements which limit the flexibility of the caching algorithms in choosing which segments to store in the cache. Therefore, new caching algorithms that consider the new characteristics and objectives need to be designed and evaluated. In this paper, we, first, study the P2P traffic characteristics via a three-month measurement study on a popular file-sharing system. We use the results obtained from our measurement to develop a deeper understanding of the P2P traffic characteristics that are relevant to caching. Such characteristics include object popularity, object size and popularity dynamics. We, then, design and evaluate a novel P2P caching algorithm for object admission, segmentation and replacement. Specifically, our contributions can be summarized as follows. First, we develop new models for P2P traffic based on our measurement study. We show that the popularity of P2P objects can be modeled by a Mandelbrot-Zipf distribution, which is a generalized form of Zipf-like distributions with an extra parameter. This extra parameters captures the flattened head nature of the popularity distribution near the lowest ranked objects (when plotted on a log-log scale). The flattened head nature has been also observed by a previous study [1], but no specific distribution was given. We also show that P2P traffic is composed of different workloads, each of which could be modeled by a Beta distribution. Second, we analyze the impact of the Mandelbrot-Zipf popularity model on caching and show that relying on object popularity alone may not yield a high hit rate. Third, we design a new caching algorithm for P2P traffic that is based on segmentation, partial admission and eviction of objects.

3 3 We perform trace-based simulations to evaluate the performance of our algorithm and compare it against common web caching algorithms (LRU and LFU) and a recent caching algorithm proposed for P2P systems. Our results show that with a relatively small cache size, less than 1% of the total traffic, a byte hit rate of up to 35% can be achieved by our algorithm, which is a few percentages smaller than the byte hit rate achieved by an off-line optimal algorithm with complete knowledge of future requests. Our results also show that our algorithm achieves a byte hit rate that is at least 4% more and at most triple the byte hit rate of the common web caching algorithms. The rest of this paper is organized as follows. In Section II, we summarize the related work. Section III describes our measurement study, provides models for popularity and object size and analyzes the effect of Mandelbrot-Zipf distribution on the cache hit rate. Our P2P caching algorithm is described in Section IV. We evaluate the performance of our algorithm using trace-based simulation in Section V. Section VI concludes the paper and outlines future directions for this work. II. RELATED WORK Several measurement studies have analyzed various aspects of P2P systems, and a few of them have proposed caching algorithms for P2P traffic. Gummadi et al. [1] study the object characteristics of P2P traffic in Kazaa and show that P2P objects are mainly immutable, multimedia, large objects that are downloaded at most once. The study shows that the popularity of P2P objects exhibits a flattened head and does not follow Zipf distribution, which is typically used to model the popularity of web objects [5]. The study provides a simulation model for generating P2P traffic that mimics the observed popularity curve, but it does not provide any closed-form models for it. Sen and Wang [6] study the aggregate properties of P2P traffic in a large-scale ISP, and confirm that P2P traffic does not obey Zipf s distribution. Their observations also show that few clients are responsible for most of the traffic. Klemm et al. [7] study the query behavior in Gnutella file sharing network and propose using two Zipflike distributions to model query popularity. However, measuring popularity of P2P objects by queries may not be accurate. Queries are issued using keywords, not object IDs, and keywords are generated by independent users. Thus, unique objects may not correspond to unique keywords. Further, queries do not imply downloads, since a query could be abandoned or go unanswered. We use unique reply messages in our measurement. These reply messages correspond to replicas of objects that do exist on hosts, and were downloaded sometime in the past. While the aforementioned studies provide useful insights into P2P systems, they were not explicitly designed to study caching P2P traffic. Therefore, they did not focus on analyzing the impact of P2P

4 4 traffic characteristics on caching. The study in [1] highlighted the potential of caching and briefly studied the impact of traffic characteristics on caching. But the study was performed in only one network domain (a university campus network). Leibowitz et al. [8] study the feasibility of caching P2P traffic. They conclude that P2P traffic is highly repetitive and responds well to caching. However, the study does not provide any algorithm for caching. Wierzbicki et al. [9] propose two cache replacement policies for P2P traffic: MINRS (Minimum Relative Size), which evicts the object with the least cached fraction, and LSB (Least Sent Byte), which evicts the object which has served the least number of bytes from the cache. Our simulation results show that our algorithm outperforms LSB, which is better than MINRS according to their results. Another related problem to caching is P2P traffic identification. Because P2P traffic is not standardized, different protocols use different control messages, and some use encryption and dynamic ports. Sen et al. [1] propose a technique for identifying P2P traffic using application level signatures. Karagianniset et al. [11] propose a non-payload P2P traffic identification at the transport layer based on network connection patterns. III. MODELING P2P TRAFFIC We are interested in deploying caches in different autonomous systems (ASes) to reduce the WAN traffic imposed by P2P systems. Thus, our measurement study focuses on measuring the characteristics of P2P traffic that would be observed by these individual caches, and would impact their performance. Such characteristics include object popularity, popularity dynamics, number of P2P clients per AS, and object size distribution. We measure such characteristics in several ASes of various sizes. A. Measurement Methodology We conduct a passive measurement study of the Gnutella file-sharing network [12]. According to the Gnutella protocol specifications peers exchange several types of messages including PING, PONG, QUERY and QUERYHIT. A QUERY message contains a set of keywords a user is searching for and is propagated to all neighbors in the overlay for a distance specified by the TTL field. A typical value for TTL is seven hops. The QUERY message contains the search keywords, a TTL field and the the address of the immediate neighbor which forwarded the message to the current peer. If a peer has one or more of the requested files, it replies with a QUERYHIT message. A QUERYHIT message is routed on the reverse path of the QUERY message it is responding to, and it contains the name and the URN (uniform resource name) of the file, the IP address of the responding node, and file size. Upon receiving replies

5 5 (a) (b) 1 3 AS1183 Object popularity MZipf(.47, 5) Frequency Rank (c) (d) Fig. 1. Object popularity in P2P traffic. from several peers, the querying peer chooses a set of peers and establishes direct connections with them to retrieve the requested file. We modify an open-source Gnutella client called Limewire [13] by implementing a listener class which logs all messages that pass through it. Limewire implementation has two kinds of peers: ultrapeers, characterized by high bandwidth and long connection periods, and leaf-peers which are ordinary peers that only connect to ultra-peers. We run our measurement in an ultra-peer mode and allow it to maintain up to 5 simultaneous connections to other Gnutella peers. On average, our measurement node was connected to 279 nodes, 63% of which were ultra peers. Our measurement node passively recorded

6 6 1 4 AS143 Object popularity MZipf(1,.7) 1 3 Frequency Rank (a) (b) 1 3 AS95 Object popularity MZipf(.62, 5) 1 3 AS946 Object popularity MZipf(1., 287) Frequency 1 2 Frequency Rank (c) Rank (d) Fig. 2. Object popularity in P2P traffic. the contents of all QUERY and QUERYHIT messages that pass through it without injecting any traffic into the network. The measurement study was conducted between 16 January 26 and 16 April 26. Our measurement peer is located at Simon Fraser University, Canada. But since the Gnutella protocol does not favor nodes based on their geographic locations [7], we are able to observe peers from many ASes. Table I summarizes some statistics about our measurement. All measurement data is stored in several trace files with a total size of approximately 4 giga bytes, and will be made available to the research community at [14].

7 7 TABLE I MEASUREMENT STUDY STATISTICS. Measurement period 16 Jan Apr. 26 Number of QUERY messages 214,264,18 Number of QUERYHIT messages 17,63,419 Number of unique objects 11,449,168 Number of IP addresses 23,148,1245 Number of unique ASes 16,295 Number of ASes with more than 1, replies 392 B. Measuring and Modeling Object Popularity The relative popularity of an object is defined as the probability of requesting that object relative to other objects. Object popularity is critical for the performance of the cache. Intuitively, storing the more popular objects in the cache is expected to yield higher hit rates. Since we are interested in the performance of individual caches, we measure the popularity of objects in each AS, not the global popularity of objects. To measure the popularity of an object in a specific AS, we count the number of replicas of that object in the AS considered. Number of replicas indicates the number downloads that were completed in the past. This means that if a cache were deployed, it would have seen a similar number of requests. This assumes that most of the downloads were supplied by peers from outside the AS, which is actually the case because peers in most current P2P networks have no sense of network proximity and thus do not favor local peers over non-local peers. In fact, previous studies [1] have shown that up to 86% of the requested P2P objects were downloaded from peers outside of the local network even though they were available locally. Notice that measuring popularity using replica count does not account for aborted downloads, i.e., requests that initiated a download but never finished it. This means that the popularity of individual objects is somewhat underestimated. This may make up for the overestimation resulted from assuming that all replicas are downloaded from peers outside of the AS. Note also that, from the cache perspective, it is the relative popularity of objects, not the absolute number of requests of each objects, that influences the cache performance. Thus we opt to measure popularity as the number of replicas that exist for each object instead of the number of requests it receives. To count the number of replicas of a given object, we extract from our trace all QUERYHIT messages

8 8 TABLE II OBJECT POPULARITY IN SEVERAL AS DOMAINS. AS No. unique IPs unique objects MZipf(α, q) % of cacheable traffic , , 122 (.65, 5) , , 184 (.55, 25) , 48 4, 381 (1., 287) , , 416 (.67, 5) , , 81 (.62, 5) , , 396 (.62, ) , 74 (.6, 55) , 198 (.62, 6) , , 73 (.6, 5) , , 89 (.53, 1) , , 259 (.56, 5) , , 78 (.7, 1) , , 72 (.58, 4) , 921 (.6, 4) , , 263 (.62, 8) , , 128 (.47, 5) which contain the unique ID (URN) of that object. Since these QUERYHIT messages contain the IP addresses of the responding nodes that have copies of the requested object, we can determine the number of replicas by counting the number of unique IP addresses associated with a specific object ID. We then map these unique IP addresses to their corresponding ASes by using the GeoIP database [15]. The number of unique IP addresses per AS associated with an object AS gives us the number of replicas of the considered object in that AS. We compute the popularity of each object in each of the top 16 ASes (in terms of the number of QUERYHIT messages sent). These top 16 ASes contribute around 38% of the total traffic seen by our study. We rank objects based on their popularity, and we plot popularity versus rank. Figures 1 and 2 show a sample of our results for eight ASes. Results for other ASes are very similar. As shown in the figures, there is a flattened head in the popularity curve of P2P objects. This flattened head indicates that objects at the lowest ranks are not actually very popular. This flattened head phenomenon could be attributed to two main characteristics of objects in P2P systems: immutability and large sizes. The immutability of P2P objects eliminates the need for a user to download an object more than once. This download at most once behavior has also been observed in previous studies [1]. The large size of objects,

9 9 and therefore the long time required to download them, may make users download only objects that they are really interested in. This is in contrast to web objects, which take much shorter times to download, and therefore, users may download web objects even if they are of marginal interest to them. These two characteristics reduce the total number of requests for popular objects. Figures 1 and 2 also show that, unlike the case for web objects [5], using a Zipf-like distribution to model the popularity of P2P objects would result in a significant error. In log-log scale, the Zipflike distribution appears as a straight line, which can reasonably fit most of the popularity distribution except the left-most part, i.e., the flattened head. In particular, the Zipf-like distribution would greatly overestimate the popularity of objects at the lowest ranks, which are the most important to caching mechanisms because those are the good candidates to be stored in the cache. We propose a new model that captures the flattened head of the popularity distribution of objects in P2P systems. Our model uses the Mandelbrot-Zipf distribution [16], which is a general form of Zipf-like distribution. The Mandelbrot-Zipf distribution defines the probability of accessing an object at rank i out of N available objects as p(i) = H (i + q) α, (1) where H = N i=1 1/(i+q)α, α is the skewness factor, and q is a parameter which we call the plateau factor because it is the reason behind the plateau shape near to the left-most part of the distribution (see Figure 3). Notice that the higher the value of q, the more flattened the head of the distribution is, and when q =, Mandelbrot-Zipf distribution degenerates to a Zipf-like distribution with a skewness factor α. To validate this popularity model, we fit the popularity distributions of the top 16 ASes to Mandelbrot- Zipf distribution using the Matlab distribution fitting tool. Our results, some shown in Figures 1 and 2, indicate that Mandelbrot-Zipf distribution models reasonably well the popularity of P2P objects. Table II summarizes α and q values for the top sixteen ASes. We observe that a typical value for α is between.4 and.7, and a typical value of q is between 25 and 6. We also observe that ASes with a large ratio of number of objects to the number of hosts have a large q value, which corresponds to a more flattened popularity curve. This could be explained by the request at most once behavior of peers, where the number of replicas of a certain object in an AS is bounded by the number of hosts that exist in that AS. We also notice that such ASes are responsible for most of the traffic. For instance, AS number 14832, Figure 1(a), contributed the highest reply rate among all ASes we looked at even though it has

10 Zipf (1) MZipf(1, 2) MZipf(1, 5) MZipf(1, 1) Frequency Rank Fig. 3. Mandelbrot-Zipf, denoted as MZipf(α, q), versus Zipf, denoted as Zipf(α), distributions. only 515 hosts. Those heavy-hitter ASes have also being observed in previous studies [6]. We are aware that the number of unique IP addresses in an AS might not reflect the number of unique hosts due to dynamic IP addressing. To approximate the number of unique hosts, we look at the number of dynamic IP addresses found in a random subset of our trace containing more than 1, unique replies. We identify an IP address as dynamic if two IP addresses reply with the the same object URN and local object index. A local index is included in the query hit replies and it identifies the location of the object on the disk of the responding peer. We find out that 8.3% of the IP addresses in this sample trace were dynamic IP addresses. Although such an approximation might be different depending on the AS, it nontheless points out the existence of dynamic IP addresses that could lead to over-estimating the number of unique peers in an AS. We notice that a large number of hosts in an AS results in a small value of the plateau parameter q, and a large amount of uncacheable traffic (i.e., objects with only once replica). The larger the number of hosts, the higher the chance of popular objects being downloaded more often which results in a less flattened head observed in the loglog plot of the popularity distribution. This is equivalent to a small value of q. A small q value has a positive effect on caching, since storing the popular objects yields higher hit rate. This can be seen in ASes 9548, 223 and 2461 where the number of hosts is in the order of hundreds of thousands, and the value of q is five or less; see table II. On the downside, a large number of hosts means a higher ratio of uncacheable traffic. This could be

11 Hit Rate (%) Zipf(1) 4 MZipf(1,25) MZipf(1, 6) Relative Cache Size (%) Fig. 4. Hit rate loss under Mandelbrot-Zipf popularity distribution. explained by the fact that a larger number hosts means a more diverse user community, which means a higher one-timer population. A high ratio of uncacheable traffic affects caching negatively, because the maximum byte hit rate that could be achieved is bounded above by the amount of cacheable traffic. However, the amount of cacheable traffic is not an indication of a higher byte hit rate under finite size caches. This is because the cache performance depends only on the the most popular objects, assuming that the cache stores them. Cacheable traffic, however, refers to anything that was downloaded more than once; an extreme example is when all objects are downloaded just twice. Thus, the skewness of the popularity distribution is the deciding factor in cache performance. ASes with small values of q and large values of α would benefit the most from deploying caches. This means that the theoretical maximum limits of the cacheability of P2P traffic from previous studies [1], [8] do not necessarily imply a good cache performance. C. The Effect of Mandelbrot-Zipf Popularity on Caching In the previous section we showed that P2P popularity could be modeled using the Mandelbrot-Zipf distribution. In this section, we analyze the impact of this popularity model on the cache hit rate. Let us assume that only the C most popular objects are stored in the cache, that is an LFU policy is used. We also assume that all objects have the same size. Then, the hit rate HR is given by:

12 Hit rate Zipf(1.) 1 mzipf(.5,5) mzipf(.5,4) Relative Cache Size Fig. 5. Hit rate loss under Mandelbrot-Zipf popularity distribution with LRU. HR = C i=1 C p(i) = i=1 C i=1 H (i + q) α H (i + q) α = H ln q + C q + 1. (2) For Zipf-like popularity distribution (q = ), HR = H ln C. Figure 4 plots the hit rate under both popularity distributions as a function of the cache size. Similar hit rate reduction has been observed for LRU as shown in Figure 5. Both figures show that the hit rate under Mandelbrot-Zipf suffers severely, especially under large values of q, and small cache sizes. This is because large values of q mean that popular objects receive less requests, and thus storing them does not yield a high hit rate. Under small cache sizes, the hit rate suffers more because the fraction of objects that could be store in the cache receives only a small portion of the total traffic. This indicates that caching schemes that capitalize on object popularity alone may not yield high hit rates and may not very effective in reducing the evergrowing P2P traffic. D. Popularity Dynamics Since P2P objects are mostly immutable, users need not download them more than once. Once downloaded, popular objects will receive less and less requests and other new popular objects are

13 13 Number of requests 1 x Frequency of requests 1 x Frequency of requests 18 x Week Week Week (a) Top five popularity (b) Top ten popularity (c) Top (11 2) th popularity Fig. 6. Popularity degradation in P2P systems. introduced in the system. To quantify this change in popularity globally, we study popularity dynamics at two levels: across all ASes (globally) and per AS. For global popularity dynamics we pick the top 2 most requested objects during the first month of our measurement and see how their popularity changes over the entire measurement period. We look at the average number of requests received by objects ranked from one to five, from one to ten and from ten to twenty. We observe that the popularity grows as the file is introduced to reach its peak, and then degrades steadily in a matter of weeks after its peak. We cannot make any claims as to when those popular files were introduced, but the pattern of increase then decrease indicates that those top 2 objects are new popular objects in the system. Figure 6 shows how the popularity degrades for the top five, top ten and next top ten most popular objects. As evident from the figure, popularity in P2P systems is dynamic, and changes over time. Similar observations about object lifetime in other P2P systems have been made in [1], [17]. Figure 9 summarizes the percentages of requests received by the top most popular objects. To study the popularity dynamics per AS, we pick the top ten most requested objects seen in the first month of our measurement and see how their popularity changes over time in each AS. We do the experiment for the top eight ASes: the top four ASes, in terms of the number of replies they sent, and for another four random ASes. As Figures 7 and 8 show, the number of requests received by the top ten most popular objects decreases over time for the top four ASes and the random ASes, respectively. Thus popularity of P2P objects is dynamic and changes over time. Such popularity dynamics may have negative effects on caching, since cached objects tend to grow stale after they have built a good popularity profile. Thus, caches that capitalize on access frequency might cache old popular objects unnecessarily. However, we believe that the introduction of new popular

14 14 8 x 14 AS x 14 AS Number of requests Number of requests Week Week (a) (b) 14 AS AS Number of requests Number of requests Week Week (c) (d) Fig. 7. Popularity dynamics of the top five most popular objects in the top four ASes. objects negates the effect of caching old popular objects. That is, the introduction of new popular objects competing for cache space will result in the eviction of older popular objects eventually. E. Measuring and Modeling Object Size Many traditional cache replacement algorithms use object size in their decision to evict objects [3]. Other algorithms, such as Greedy-Dual Size [18], incorporate size with other characteristics to maximize cache performance. Thus, understanding how object size is distributed is critical to developing effective caching schemes.

15 15 7 AS946 6 AS Number of requests Number of requests Week Week (a) (b) 6 AS AS Number of requests Number of requests Week Week (c) (d) Fig. 8. Popularity dynamics of the top five most popular objects in four randomly-chosen ASes. To develop a model for P2P object sizes, we collect information about objects in two popular P2P filesharing systems: Gnutella and BitTorrent. In BitTorrent, there are a number of web servers that maintain metadata,.torrent, files about files in the network. These servers are known as torrent sites. We develop a script which contact four popular torrent sites and downloads random.torrent files. We downloaded 1 thousand.torrent files, 5% of which are unique. Each.torrent file contains information about the shared object, such as object size, number of segments in the object and the size of each segment. For the Gnutella system, we use randomly pick 5 thousand unique objects from the traces we collected during our measurement study and analyze their sizes.

16 Percentage of requests (%) Week Fig. 9. Popularity degradation of the top five most popular objects in the top twenty ASes. The histograms of the size distribution in both BitTorrent and Gnutella, Figures 1(a) and 11(a), show that there exists several peaks where each peak corresponds to a certain workload. It is clear that it is hard to model the entire size range using a single distribution. Therefore, we divide the whole size range (approximately MB - 25, MB) into multiple intervals. The choices of such intervals is not random, but governed by the existence of different types of content. For example, a peak around 7 MB corresponds to most shared CD movies; another peak around 25 MB corresponds to high quality TV video series and some software objects while a peak around few MB corresponds to video clips and audio clips. Using the Chi-Square goodness-of-fit test, we show that the size distribution of each content type (i.e., workload) can can be modeled as a Beta distribution. We also show that the size distribution of objects observed per AS for the Gnutella traffic we collected is also multimodal and contains multiple workloads, which could be modeled using the Beta distribution. A beta distribution is defined as follows: A random variable is said to be beta distributed with parameters α and β (both positive) in the interval [A,B] if the probability density function (pdf) of X is 1 Γ(α + β) P(x) = B A Γ(α)Γ(β) ( x A B A ) α 1 ( ) B x β 1 A x B B A Otherwise When A =, B = 1, the above distribution is called the standard beta distribution and with a pdf ( ) Γ(α + β) P(x) = (1 x) β 1 x α 1 Γ(α)Γ(β) (3)

17 17 Frequency Cumulative probability ( 349) MB Beta(.62, 1.52) Object Size (MB) Normalized Size (a) (b) 1.9 (35 7) MB Beta(.48,.32) 1.9 ( ) MB Beta(.59,.55).8.8 Cumulative probability Cumulative probability Normalized Size Normalized Size (c) (d) 1.9 (14 45)MB Beta(.38,.53) 1.9 (45 25) MB Beta(.61, 2.11).8.8 Cumulative probability Cumulative probability Normalized Size Normalized Size (e) (f) Fig. 1. Object size distribution in BitTorrent.

18 18 where Γ(α) is the gamma function and is defined as: Γ(α) = x α 1 e x dx, x 1 Figure 1 shows the CDF (Cumulative Distribution Function) of object sizes in BitTorrent fitted against the CDF of the standard Beta distribution. Similarly, Figure 11 shows that Beta distribution could be used to reasonably model the object size distribution in Gnutella. We notice that the sizes of objects in BitTorrent are larger than in Gnutella. While objects of less than 1 MB constitute approximately 9% of objects in Gnutella, they only constitute 36% in BitTorrent. Figure 12 shows the distribution of objects into workloads in Gnutella and BitTorrent. The fact that objects in BitTorrent are usually large video objects could be explained as follows. BitTorrent offers higher availability, due to its tit-for-tat mechanism where each peer contributes by uploading to other peers, while a peer in Gnutella can choose to free-ride without running the risk of getting slower downloading rates. Thus peers in Gnutella may not be as willing to share content, which means objects are less available. BitTorrent, on the other hand gives each peer a downloading rate proportional to its uploading rate which punishes free-riders, thus achieving higher availability. Higher availability means less download aborts, and thus faster object retrieval, which in turn promotes sharing of large objects. We should keep in mind that, even though most of the downloaded objects observed by our measurement were small objects, the majority of the traffic was contributed by large sized objects. To study this, we pick the top ten ASes, and look at the fraction of objects with size less than 1 MB and larger than 1 MB. For each of these two categories, we count the number of objects downloaded in the respective AS, and the sum of their sizes, which corresponds to the traffic they contributed. As Figure 13 shows, objects with size larger than 1 MB account for only about 1% of the total number of downloaded, but they are responsible for about 9% of the traffic. This shows that large size of objects is a very important factor to consider for any caching policy which strives to reduce the P2P traffic on the Internet backbone. Knowing the distribution of P2P objects and the fraction of requests for each workload, we can estimate the required cache size (Size) to achieve a certain byte hit rate. n Size = BHR λ i.µ i, where λ i is the fraction of requests received by workload i and µ i is the mean of the respective size distribution in workload i (which is Beta distributed), and is defined as follows α µ = A + (B A) α + β. i=1

19 19 6 x ( 349) MB Beta(.5, 1.42) Frequency Object Size (MB) Cumulative probability Normalized Size (a) (b) 1.9 (35 7) MB Beta(.63,.44) 1.9 (7 14) MB Beta(.29,.57).8.8 Cumulative probability Cumulative probability Normalized Size Normalized Size (c) (d) > 14 MB Beta(.34, 1.26) Cumulative probability Normalized Size (e) Fig. 11. Object size distribution in Gnutella.

20 2 6 5 Gnutella Bittorrent Percentage of objects (%) [,1] [1,1] [1, 7] [7, 14] > 14 Object size (MB) Fig. 12. Object size comparison under BitTorrent and Gnutella. We discuss the the relationship between byte hit rate and cache size further in the evaluation section. F. Summary of the measurement study We designed our measurement study to analyze the P2P traffic that would be observed by individual P2P caches deployed in different ASes. We found that popular P2P objects are not as highly popular as their web counterparts. The popularity distribution of P2P objects has a flattened head at lowest ranks, and therefore, modeling this popularity as a Zipf-like distribution yields a significant error. We also found that a generalized form of the Zipf distribution, called Mandelbrot-Zipf, captures this flattened head nature and therefore is a better model for popularity of P2P objects. Furthermore, we found that the Mandelbrot-Zipf popularity has a negative effect on hit rates of caches that use the common LRU and LFU policies. In addition, our study revealed the dynamic nature of object popularity in P2P systems, which should be considered by the P2P caching algorithms. Finlay, we found that objects in P2P systems have much larger sizes than web objects, and they can be classified into several categories based on content types. Each category can be modeled as a Beta distribution. IV. P2P CACHING ALGORITHM With the understanding of the P2P traffic we developed, we design and evaluate a novel P2P caching scheme based on segmentation and partial caching. Partial caching is necessary because of large objects sizes. Unlike traditional memory systems where a page must be loaded to memory before it is used,

21 21 Percentage of # of downloads / # of bytes (%) % of # of downloads % of number of bytes downloaded AS Rank Fig. 13. Number of downloads for objects with size larger than 1 MB, and the number of bytes they contributed to the total traffic. This is measured for the top 1 ASes. The Figure shows that even though large objects constitute only 1% of the total number of downloads, they are responsible for about 9% of the traffic. objects in web or P2P systems do not have to be cached on every cache miss. In particular, objects in P2P systems are large and the most popular objects may not receive too many requests. Thus storing an entire object upon request may waste cache space. P2P caching should take a conservative approach towards admitting new objects into the cache to reduce the cost of storing unpopular objects. The basic idea of our algorithm is to cache of each object a portion that is proportional to its popularity. That way popular objects will be incrementally given more cache space than unpopular objects. To achieve this goal, our algorithm caches a constant portion of an object when it first sees that object. As the object receives more hits, its cached size increases. The rate at which the cached size increases grows with the number of hits an object receives. To facilitate admission and eviction of objects, our policy divides objects into segments. The smallest granularity that our algorithm caches/evicts is one segment. We use different segment sizes for different workloads because using the same segment size for all workloads may favor some objects over others and introduce additional cache management overhead. For example, if we use a small segment size for all workloads, large objects will have large number of segments. This will introduce a high overhead of managing large number of segments within each object. Such overhead includes locating which segments of an object are cached, and deciding which segments to evict. On the other hand, using a large segment size for all workloads will favor smaller objects since they will have smaller number of segments and get cached quicker. Making segment size

22 22 relative to object size has the same two problems. This is because the range of P2P object sizes extends to few gigabytes. Thus using relative segmentation, as a percentage of an object size, may result in large objects having segments that are orders of magnitude larger than unsegmented smaller object. Consistent with our observations of multiple workloads, we use four different segment sizes. For objects with sizes less than 1 MB, we use a segment size of 512 KB, for objects between 1 MB and 1 MB, a segment size of 1 MB, and for objects between 1 MB and 8 MB a segment size of 2 MB. Objects whose size exceeds 8 MB are divided into segments of 1 MB in size. We call this segmentation scheme variable segmentation scheme. Note that our segmentation procedure is inline with most segmentation schemes used by real P2P systems. Our analysis of more than 1, torrent files shows that the vast majority of Bittorrent objects are segmented into 256 KB, 512 KB, 1 MB and 2 MB segments. E-Donkey uses segments of size 9.28 MB [19]. Depending on the client implementation, segment size in Gnutella can either be a percentage of object size, as in BearShare [2], or fixed at few hundred KBs [12]. For each workload the cache keeps the average of object size in this workload. Denote the average object size in workload w as µ w. Further let γ i be the number of bytes served from object i normalized by its cached size. That is, γ i can be considered as number of times each byte in this object has been served from the cache. The cache ranks objects according to their γ value, such that for objects ranked from 1 to n, γ 1 γ 2 γ 3 γ n. From this point on, we refer to an object at rank i simply as object i. When an object is seen for the first time, only one segment of it is stored in the cache. If a request arrives for an object of which at least one segment is cached, then the cache computes the number of segments to be added to this object s segments as (γ i /γ 1 )µ w, where µ w is the mean of the object size in workload w which object i belongs to. Notice that this is only the number of segments the cache could store of object i. But since downloads can be aborted at any point during the session, the number of segments actually cached upon a request, denoted by k, is given by k = min[missed, max(1, γ i γ 1 µ w )], (4) where missed is the number of requested segments not in the cache. This means that the cache will stop storing uncached segments if the client fails or aborts the download, and that the cache stores at least one segment of an object. The pseudo-code of our algorithm appears in Figure 14. At a high level, the algorithm works as follows. The cache intercepts client requests and extracts the object ID and the requested range. If no segments of the requested range is in the cache, the cache stores at least one segment of the requested range. If the

23 23 entire requested range is cached, the cache will serve it to the client. If the requested range is partially available in the cache, the cache serves the cached segments to the client, and decides how many of the missing segments to be cached using Eq. (4). In all cases, the cache updates the average object size of the workload to which the object belongs and the γ values of the requested object. request(object i, requested range) 1. if object i is not in the cache 2. add one segment of i to cache, evicting if necessary 3. else 4. hit = cached range requested range 5. γ i += hit / cached size of i 6. missed = (requested range - hit)/segment size 7. k = min[missed, max(1, γ i µ w )] γ 1 8. if cache does not have space for k segments 9. evict segments from the least valued object 1. add k segments of object i to the cache 11. return Fig. 14. P2P caching algorithm. The algorithm uses a priority queue data structure to store objects according to their γ i value. When performing eviction, the required segments are removed from the least valued objects. Currently, our algorithm evicts contiguous segments from the least valued objects without favoring any segments over others. This is because P2P downloads are likely to start from anywhere in the file [9]. It remains a future goal of ours to study in-object request patterns and improve our algorithm accordingly. Our algorithm needs to perform O(logN) comparisons with every hit or miss, where N is the number of objects in the cache. But since objects are large, this is a reasonable cost considering the small number of objects the cache will contain. V. EVALUATION In this section, we use trace-driven simulation to study the performance of our P2P caching algorithm, and compare it against two common web caching algorithms (LRU, LFU and GDS) and a recent caching

24 24 algorithm proposed for P2P systems, and the offline optimal policy. A. Experimental Setup We use traces obtained from our measurement study to conduct our experiments. Our objective is to study the effectiveness of deploying caches in several ASes. Thus, we collect information about objects found in a certain AS and measure the byte hit rate that could be achieved if a cache were to be deployed in that AS. We use the byte hit rate as the performance metric because we are mainly interested in reducing the WAN traffic We run several AS traces through the cache and compare the byte hit rate achieved using several caching algorithms. In addition to our algorithm, we implement the Least Recently Used (LRU), Least Frequently Used (LFU), Greedy-Dual Size (GDS) [18] and Least Sent Bytes (LSB) [9], and the off-line optimal (OPT) algorithms. LRU capitalizes on the temporal correlation in requests, and thus replaces the oldest object in the cache. LFU sorts objects based on their access frequency and evicts the object with the least frequency. GDS maintains a characteristic value H i for each object, where H i = cost i /size i +L. The object with the smallest H i value will be evicted next. The cost parameter could be set to maximize hit rate, minimize latency or minimize miss rate. We set the size parameter to be 1 + size/536, which is the estimated number of packets sent and received upon a miss. LSB is designed for P2P traffic caching and it uses the number of transmitted bytes of an object as a sorting key. Objects which have transmitted the least amount of bytes will be evicted next. OPT looks at the entire stream of requests off-line and caches the objects that will serve the most number of bytes from the cache. We measure the byte hit rate under several scenarios. First, we consider the case when objects requested by peers are downloaded entirely, that is, there are no aborted transactions. Then, we consider the case when the downloading peers pre-maturely terminate downloads during the session, which is not uncommon in P2P systems. We also run our experiments under the independent reference model, where entries of the trace file are randomly shuffled to eliminate the effect of temporal correlation and popularity degradation. We do that to isolate and study the effect of object popularity on caching. Finally, we run our experiments on the original trace with preserved temporal correlation between requests to study the combined effect of popularity and popularity degradation. In all experiments, we use the ASes different traffic characteristics, which are representative of P2P traffic observed in other ASes. We concentrate on top ASes which sent the most number of replies to ensure that we have enough traffic from an AS to evaluate the effectiveness of deploying a cache for it. Caches usually strive to store objects that have a higher likelihood of being requested in the near

25 25 future. This likelihood of requesting objects is called temporal locality of requests [21], [22]. Temporal locality has two components: popularity and temporal correlations. Popularity of an object is the number of requests it receives relative to other objects. Temporal correlation of requests means that rquests to the same object tend to be clustered or correlated together in time. To illustrate the difference consider the following two sequences...abcaegfaxy NAACEABA... in which A receives more requests than other object. Thus is A is more popular than other objects. Consider a second sequence... ABBBACCCEAFBBAB... in which requests to object B are C correlated and tend to appear close to each other. Temporal correlation is usually measured in the number of intervening requests between two consecutive requests to the same object. A shorter sequence of intervening requests imply higher temporal correlations. Temporal locality of web request streams has been widely studied in the literature (see [21] [23] and references therein). However, we believe that temporal correlations is not as critical to P2P caching as it is to web caching. This is because web objects are small, which means that the cache can accommodate a long sequence of requests, during which temporal locality will be evident. But in P2P caching, objects are orders of magnitude larger than their web counterparts, which means that a sequence of P2P objects that could be stored in the cache is orders of magnitude shorter than a sequence of web objects. B. Caching under Full Downloads One of the main advantages of our caching policy is that it effectively minimizes the impact of unpopular traffic by admitting objects incrementally. Traditional policies usually cache an entire object upon a miss, evicting other, perhaps more popular, objects if necessary. This may waste cache space by storing unpopular objects in the cache. To investigate the impact of partial versus full caching, we, first, assume that we have a no-failure scenario where peers download the object entirely. We look at all objects found in an AS, line them up in a randomly shuffled stream of requests and run them through the cache. We vary the cache size between and 1 GB. Figure 15 shows the byte hit rate for four representative ASes with different characteristics. These ASes have different maximum achievable byte hit rates, which is defined as the fraction of traffic downloaded more than once, i.e., cachebale traffic, over the total amount of traffic. As shown in the figure, our policy outperforms other policies by a wide margin. For instance, in AS397 (Figure 15(a)) with a cache of 6

26 26 Byte Hit Rate OPT P2P LSB LFU LRU GDS Byte hit rate under AS397 Byte Hit Rate (%) OPT P2P LSB LFU LRU GDS AS Cache Size (GB) Cache Size(GB) (a) 48% of traffic is cacheable (b) 54% of traffic is cacheable Byte Hit Rate OPT P2P LSB LFU LRU GDS AS1161 Byte Hit Rate OPT P2P LSB LFU LRU GDS AS Cache Size (GB) Cache Size (GB) (c) 16% of traffic is cacheable (d) 95% of traffic is cacheable Fig. 15. Byte hit rate for different caching algorithms. No aborted downloads. GB, our policy achieves a byte hit rate of 24%, which is almost double the rate achieved by LRU, LFU, GDS and LSB policies. Moreover, the byte hit rate achieved by our algorithm is about 3% less than that of the optimal algorithm. Our trace shows that the amount of traffic seen in AS397 is around 24.9 tera bytes. This means that a reasonable cache of size 6 GB would have served about 6 tera bytes locally using our algorithm. The reason traditional policies perform poorly for P2P traffic is because of the effect of unpopular objects. For example, one-timer objects are stored entirely under traditional policies on a miss. Under our policy, however, only one segment of each one-timer will find its way into the cache, thus minimizing