The Problem of Broker Placement in Peer-to-Peer Overlay Networks

Transcription

1 The Problem of Broer Placement in Peer-to-Peer Overlay Networs Pawe l Garbaci and Dic H.J. Epema Faculty of Electrical Engineering, Mathematics, and Computer Science Delft University of Technology P.O. Box 5031, 2600 GA Delft, the Netherlands tel , <p.garbaci@ewi.tudelft.nl> Maarten van Steen Department of Computer Science Vrije Universiteit Amsterdam 1081 HV Amsterdam, the Netherlands November 7, 2004 Abstract In large peer-to-peer overlay networs, nodes usually share resources to support all inds of applications. In such networs, a subset of these nodes may assume the role of broer in order to act as intermediaries for finding suitable resources. When some notion of distance between nodes such as the internode latency is defined, broers may be responsible for a group of nodes that are close to each other. In this paper we present a broer-placement algorithm that finds a suitable location for a new broer when some broer is overloaded. With latency as a metric, such a networ can be embedded in a Euclidean space R d, and our algorithm amounts to selecting suitable regions in R d for broer placement. The algorithm is of complexity O(n d+1 ) with n the number of nodes that may be assigned to the placed broer.

2 P. Garbaci, D.H.J. Epema, M. van Steen 1 1 Introduction With the advent of peer-to-peer overlay networs, we are gradually seeing a new generation of fully decentralized Internet applications that rely on resources spread across the Internet and owned by different organizations and users. Following a relatively simple model, resources such as storage and processing capacity are made available by the nodes of such networs. In principle, these networs may consist of tens of thousands to millions of nodes, and so finding and sharing the resources that are suitable for a specific application may turn out to be a formidable tas. A standard solution to this problem is to mae use of special nodes called broers. A broer collects resource information from a collection of nodes so that it can assist other nodes in finding and obtaining resources. In this paper, we assume that broers are just designated nodes that participate with other nodes in maintaining and resource sharing in a peer-topeer overlay networ. Such nodes are also sometimes referred to as superpeers [12]. In order for broerage to be effective and efficient, a broer should have enough associated nodes for which it collects resource information, while at the same time broers should be prevented from becoming bottlenecs, and from becoming overloaded. In this paper, we address the problem of assigning nodes the role of broer in a peer-to-peer overlay networ. We consider this assignment problem for widely distributed systems in which proximity is used as a metric for resource management. As a concrete example, we consider Globule [7], which is a peer-to-peer content delivery networ in which replicas of Web documents are to be placed close to those documents clients. Proximity in Globule is expressed in terms of latency. We have developed a latency-driven replica placement algorithm that allows any node hosting a Web document to identify the regions where it should place replicas to optimally serve clients [10]. Once such a region has been identified, a specific node in that region must be sought for hosting a replica. To this end, a broer for that region is consulted. It has been shown that with internode latency as the notion of distance [4, 6, 11], peer-topeer systems consisting of broers and other nodes can be thought of as being embedded in a d-dimensional Euclidean space R d. Nodes are given locations such that the latency between two nodes can be estimated as the distance between their locations. In our model of such networs, we assume that a node is represented by the broer that is closest to it. As a consequence, the number of nodes assigned to a broer, and the load put on it, depends on its position. In this paper we present an algorithm for placing a new broer when some broer gets overloaded. This algorithm is of complexity O(n d+1 ) with n the number of servers that may be assigned to the new broer, which we show to be optimal up to a linear factor in n. Our main contribution is that broers are placed in such a way that they represent only nodes that are close to each other. As such, our broers allow for proximity-based resource management. The rest of this paper is organized as follows. In Section 2, we present our problem statement and give some definitions. Section 3 contains our broer-placement algorithm, and Section 4 discusses its complexity. We conclude in Section 5 with a brief summary and some remars on the deployment of our algorithm in a real environment. 2 The problem of broer placement 2.1 Motivation Broers play an important role in many distributed systems. For Globule, they currently mainly contribute to monitoring available resources. In our system, nodes are assumed to be

3 2 The Problem of Broer Placement in a Peer-to-Peer Overlay Networs end-user machines that host their owner s Web documents. We use replication to optimize performance taing metrics into account such as consumed bandwidth, response times, number of returned stale documents, etc. From previous experiments, we have observed that by deciding per document how replication should tae place, and by re-evaluating these decisions regularly, it becomes possible to reach near-optimal performance [8, 9]. A prerequisite for such adaptive replication is efficiently finding suitable replica servers. With potentially millions of very different nodes at hand of which many may be expected to often join and leave the system at will, finding replica servers is a formidable tas. To alleviate problems, we use broers. Broers are assumed to be elected from high-available nodes. Their main tas is to monitor the resources in a collection of (non-broer) nodes, which we shall call servers. Monitoring includes eeping trac of relatively static information such as available software libraries, but may also include highly dynamic information such as current dis space usage, CPU utilization, memory consumption, and so on. By-and-large, we expect that most of the load imposed on a broer is due to such monitoring, in combination with search requests for resources. As mentioned, a node is associated with its closest broer. Proximity is determined by placement in a 6-dimensional Euclidean space in which distance (between non-broer nodes) is an estimator for actual latency [11, 10]. We do not address broer election in this paper. However, when a node has been elected as a broer, we can logically position it anywhere in our space such that it can assist in monitoring the resources of the non-broer nodes for the region where it has been placed. 2.2 Terminology In order to describe the broer-placement problem we need to introduce some terminology. We will consider the d-dimensional Euclidean space R d with the distance between two points d x, y given by dist(x, y) = i=1 (x i y i ) 2. The d-dimensional closed (open) ball B r (p) (B r (p)) with radius r 0 and center p is defined as B r (p) = {x : dist(x, p) r} (B r (p) = {x : dist(x, p) < r}). In R 1, a closed ball is a closed interval; in R 2 it is a circle with its interior. A -dimensional ball in R d with < d is defined as a ball in a -dimensional linear subspace of R d. (In this paper, a linear subspace does not have to include the origin.) When we refer to a ball without specifying what ind, we assume that it is a closed ball. The (d 1)-sphere S r (p) with radius r 0 and center p in R d is defined as S r (p) = {x : dist(x, p) = r}. A -sphere with < d 1 in R d is a -sphere in a ( + 1)-dimensional linear subspace of R d. A 2-sphere is an ordinary sphere or a single point, a 1-sphere is a circle or a single point, and a 0-sphere is a pair of points or a single point. A (d 1)-sphere in R d is also called a hypersphere. A set {B i : 1 i n} of d-dimensional closed balls in R d divides the space into regions; a region is defined as n i=1 A i, where A i is either B i or Bi c, with Bc i the complement of the open ball B i. So regions do include their boundaries. Now let s assume that each ball B i has a weight denoted by W (B i ). We define the weight of a region as the sum of the weights of the balls that contain it. That is, for a region R = n i=1 A i, its weight W (R) is given by W (R) = A i =B i W (A i ). A set P R d is a set of characteristic points of a set of regions R if it contains a point in every region in R. An example of a set of regions with weights and characteristic points is shown in Figure 1. Four two-dimensional balls with weights 2, 4, 8 and 16 divide the space into ten regions

4 P. Garbaci, D.H.J. Epema, M. van Steen 3 symbol of region II:2 III:4 weight of region VI:10 VII:12 X:28 VIII:20 characteristic point IV:8 IX:24 V:16 I:0 Figure 1: An example of the regions, the region weights, and a set of characteristic points for a set of four balls in R 2. named I to X. The blac dots represent the characteristic points. 2.3 Problem statement We consider a distributed system consisting of two types of entities: broers and servers. Each server is assigned to exactly one broer, and each broer is responsible for zero or more servers. Each entity (server or broer) is placed in R d, where the distance between server nodes is an estimate for the latency. Server locations are actual locations, they cannot be changed. Broer locations are logical locations, and they can be chosen freely. Of course, broers are nodes so they also have actual locations, but these are irrelevant to our problem. The server-to-broer assignment is described by a simple rule: each server is assigned to the broer closest to it (taing into account logical locations of the broers). Using terminology from computational geometry, we can rephrase this rule by stating that a broer is responsible for all servers situated inside its Voronoi region [5]. The Voronoi region of a broer is defined as the set of points that are closer to this broer than to any other broer in the system. Each server has some demand expressed as a positive real number, and each broer has a capacity described by a positive real number. Both demands and capacities are in terms of some abstract resource unit. We require that the sum of the demands of the servers assigned to a broer does not exceed the capacity of that broer. Now assume that some broer b is overloaded and that a new broer ˆb is available to tae over the responsibility for some of b s servers. The problem that we then need to solve is to compute coordinates of ˆb such that not too few but also not too many servers are assigned to it. More precisely, a solution to this problem has to satisfy two conditions. The first condition is that the sum of the demands of the servers assigned to ˆb may not exceed its capacity. The second condition is that the sum of the demands of the servers re-assigned from b to ˆb must at least be equal to the excess load of b. Of course, the new load of ˆb consists of more than only the load shifted from b to ˆb, as also some of the loads of other broers may be shifted to it. Depending on ˆb s capacity, a solution to this problem may or may not exist. In order to state the problem in more detail, we give the following definitions. The server ball B(s) of server s is the ball with radius equal to the distance between s and its broer

5 4 The Problem of Broer Placement in a Peer-to-Peer Overlay Networs and center the location of s, and the server sphere of server s is the boundary of B(s). The weight of B(s) equals the demand of server s. We define R as the set of regions defined by all server balls in the system, and R(b) as the set of regions defined by the server balls of all servers assigned to broer b. Because R(b) is defined by a subset of the balls that define R, every region in R(b) is entirely contained in exactly one of the regions in R(b). Each region in R and R(b) is assigned a weight as defined in Section 2.2. Server s is assigned to ˆb if and only if ˆb is placed inside B(s). If after placing a new broer a server is at an equal distance from the new broer as from its old broer, it is not re-assigned to the new broer. (a) II:2 III:4 (b) II:2 XII:4 s1 b s2 s1 b s2 VI:10 VII:12 VIII:20 X:28 VI:10 XIV:12 IV:8 s3 IX:24 s4 V:16 s3 s4 b XIII:8 b I:0 XI:0 Figure 2: Division of the space into regions (a) R and (b) R(b). As an example, Figure 2 presents a two-dimensional system that consists of two broers b and b with capacities 9 and 19, and four servers s1, s2, s3, s4 with demands 2, 4, 8 and 16, respectively. The broer of s1, s2 and s3 is b while s4 is assigned to b. Figure 2 (a) shows the set R, which consists of the ten regions (I to X) which are the intersections of balls B(s1) to B(s4). Figure 2 (b) shows the set R(b) of regions created by the intersections of B(s1), B(s2) and B(s3) (the balls of the servers assigned to b). Note that regions II and VI exist in both R and R(b). A region in R represents equivalent placements for the new broer ˆb in the sense that the set of servers assigned to ˆb depends only on the region but not on the specific location for ˆb in it. If we place broer ˆb inside some region in R, then the load produced by the servers assigned to ˆb equals the weight of this region. Similarly, a region in R(b) represents equivalent placements for the new broer ˆb in the sense that the set of servers re-assigned from b to ˆb depends only on the region but not on the specific location for ˆb in it. If we place ˆb inside a region from R(b) then the load produced by the servers removed from b equals the weight of this region. Therefore, an acceptable location of ˆb is a point that belongs both to a region in R with a weight at most equal to ˆb s capacity, and to a region in R(b) with a weight at least equal to b s excess load. In our example of Figure 2, broer b is overloaded by an amount of 5 because the total demand of s1, s2 and s3 is equal to 14. Suppose that the new broer ˆb has capacity 11. Then because of the first condition that says that the new load of ˆb may not exceed its capacity, we see from Figure 2 (a) that ˆb should be placed inside one of the regions I, II, III, IV or VI. In order to satisfy the second condition that says that enough load should be moved away from b, we see from Figure 2 (b) that ˆb should be placed in one of the regions VI, XIII, or XIV. So

6 P. Garbaci, D.H.J. Epema, M. van Steen 5 the proper locations for ˆb are all points in regions IV and VI. 3 The broer-placement algorithm In this section we present our broer-placement algorithm. First we give a short outline of the algorithm. 3.1 Solution outline The general idea of the algorithm is simple: when b is the overloaded broer, consider the regions r in R one by one, find for each such r the region r b in R(b) that contains it, and chec whether the weights of r and r b satisfy the two conditions for a solution. When the first such pair (r, r b ) is found, the algorithm can be terminated, and the new broer ˆb can be located at any point in r. The main problem that needs to be solved is how to efficiently compute the sets R and R(b) of regions defined by the server balls. A naive approach is to consider all possible subsets of servers and chec whether their balls intersect to create a region. As with n servers there are 2 n such subsets, this approach has exponential complexity, rendering it infeasible. In our solution we use a set of characteristic points (see Section 2.2) that are located on the intersections of at most d server spheres. We will show that it does not matter which points of these intersections we actually pic. This set of characteristic points has two properties: it is small, and finding all the regions containing a particular characteristic point is easy. As there is at least one characteristic point in every region, iterating over all characteristic points and considering all regions that it belongs to, amounts to considering all regions. Because we only have to consider the intersections of at most d server spheres, the complexity of our algorithm is O(n d+1 ). 3.2 The algorithm We introduce the following notation: n the number of servers in the system, B the set of server balls, U the set of server spheres, which are the boundaries of the balls in B, S the set of intersections of any number of spheres from U. According to Theorem 1 in the appendix, S is a set of spheres. The pseudocode of the broer-placement algorithm is presented below. Input: excess load / the excess load that needs to be removed from broer b / capacity / the capacity of the new broer ˆb / d / the dimension of the space / U / the set of server spheres / Output: p / a location for broer ˆb, or NULL if no suitable location exists /

7 6 The Problem of Broer Placement in a Peer-to-Peer Overlay Networs 1: while (p == NULL) do 2: Û := a set of at most d spheres from U which was not considered in any of the previous iterations 3: if (all possibilities for Û are exhausted) then 4: return NULL / solution does not exist / 5: end if 6: ŝ := the intersection of the spheres in Û 7: if ŝ is a 0-sphere or ŝ is empty then 8: ˆP := the set of all points in ŝ 9: else 10: ˆP := the one-element set containing any point of ŝ 11: end if 12: for all (r in the set of regions in R with a point in ˆP ) do 13: w := the weight of region r 14: w b := the weight of the region in R(b) which contains r 15: if (w capacity and w b excess load) then 16: p := one of the points in the interior of region r 17: end if 18: end for 19: end while The algorithm is iterative. Each iteration starts with selecting not more than d spheres from the set U. According to Theorem 2 in the appendix, a set Û consisting of more than d spheres is a 0-sphere or contains at least one redundant sphere which can be removed from Û without any influence on the intersection of the elements in Û. If all possible sets Û have already been examined, the algorithm terminates without providing a solution (line 4). In line 6 we compute the intersection ŝ of all spheres in Û. According to Theorem 1 in the appendix, ŝ is either empty or is a sphere. Because every 0-sphere contains at most two points, the cardinality of the set ˆP defined in lines 7 to 11 is never higher than two. The point selected in line 10 may or may not lead to a solution in current iteration. As we show below, the choice of point in ŝ in line 10 does not however affect the correctness of our algorithm. Let s denote by ˆp a point in ˆP and by ˆR the set of all regions in R to which point ˆp belongs. All we need to now about the regions in ˆR are their weights, which can be computed in the following way. Let r ˆR, let ˆB 1 be the set of server balls that contain ˆp in their interior, and let ˆB 2 be the set of server balls that contain ˆp on their boundary. Because a ball in ˆB 1 contains an environment of ˆp, such a ball contains also all regions in ˆR. Now the weight of r can be expressed as x 1 + x 2, with x 1 the sum of the weights of the balls in ˆB 1 and x 2 the sum of the weights of the balls in ˆB 2 that contain r. To illustrate the meaning of x 1 and x 2, let s consider the system presented in Figure 2 (a). Let point ˆp be the point of the intersection of the server spheres of s3 and s4 that is located inside B(s2). In this case, ˆR contains the regions III, VII, VIII and X, ˆB 1 contains only B(s2), and ˆB 2 contains the balls B(s3) and B(s4). Therefore, the value of x 1 for all regions in ˆR is equal to the weight of ball B(s2), which is 4. As both balls in ˆB 2 contain region X, the value of x 2 for region X is 24, which is the sum of the weights of balls B(s3) and B(s4). The values of x 2 for regions III, VII, and VIII can be found in a similar way to be 0, 8, and 16, respectively. In general, the elements of ˆB 1 and the value x 1, which is independent of the particular region in ˆR, can of course be determined in a simple way. The method of computing x 2 for the regions in ˆR is based on the observation that if ˆB 2 contains balls, these balls divide a local

8 P. Garbaci, D.H.J. Epema, M. van Steen 7 environment of point ˆp into 2 areas, each belonging to one region in ˆR. Considering the sets of spheres in ˆB 2 to which each of those areas belong we can easily determine x 2 of the regions that contains this area. The weight w b defined in line 14 can be computed in a similar way as w. Different executions of our broer-placement algorithm with the same input data may result in different solutions. The nondeterminism is caused by the freedom of choice of the point in line 10. Also the random selection of Û in line 2 does not affect the correctness of our algorithm, but may have influence on the overall performance. The correctness of our algorithm the point p selected in line 16 is a suitable location for broer ˆb is a simple consequence of the conditions in line 15. However, the completeness of our algorithm, which means that it does find a solution if one exists, requires explanation. In order to prove completeness, it is enough to show that if the algorithm terminates without finding a solution, still every region in R has been considered in lines 12 to 18. In turn, it is sufficient to show that the points in sets ˆP selected in all iterations in lines 7 to 11 form a set of characteristic points. According to Theorem 3 in the appendix, we can select a subset Ŝ S with the following property: for every region in R there is a sphere in Ŝ which is either entirely contained in this region or is a 0-sphere that has a point in this region. As explained earlier in this section by taing intersections of at most d spheres in U we consider all spheres in S, so also all spheres in Ŝ. If ŝ computed in line 6 is one of the 0-spheres in Ŝ then our algorithm considers all regions that have non-empty intersection with ŝ. Every region r in R that has an empty intersection with all 0-spheres in Ŝ contains all points of one of the other spheres in Ŝ. If this sphere is computed in line 6 of the algorithm then regardless of the point selected in line 10, the region r is considered in lines 12 to Complexity analysis In this section we present the worst-case and the expected complexity of our broer-placement algorithm as it depends on the number n of servers. We assume the dimension d of the space to be constant. Let s start with the worst-case analysis. We estimate the execution cost of each line of the pseudocode separately. Line 2 requires constant time. We do not specify precisely how to select set Û, but it is not difficult to provide an implementation that executes in constant time and memory (e.g., by defining an order on U s subsets and considering those subsets in increasing order). Line 6 requires computing the intersection of the spheres in Û, which can be done in constant time as the size of set Û is limited by d. Operations performed in lines 7 to 11 require also a constant time. The method of finding the weights w and w b described in the previous section can be implemented in linear time, and so the complexity of lines 12 to 18 is O(n). We have shown that one iteration of the while loop (line 1) costs O(n). iterations is limited by the number of possible sets Û, which is O(nd ). complexity of the broer-placement algorithm is then O(n d+1 ). The number of The worst-case According to [1], page 73, n hyperspheres divide R d into O(n d ) regions. Therefore, every algorithm that considers all regions is not more than O(n) times faster than our solution. Now we turn to the average complexity. Let s define: m the number of possible sets Û, m = d+1 ) i=1, q the probability that the first iteration of the main loop leads to a solution, ( n i

9 8 The Problem of Broer Placement in a Peer-to-Peer Overlay Networs X - the random variable that denotes the number of iterations of the while loop. From a formal point of view the value of q equals the probability of selecting a white card from a set containing white and m blac cards, where = q m. If is not an integer then we round it to the closest integer value. This slight modification of value does not have a big influence on the probability of selecting a white card. Intuitively we should interpret a white card as set Û which, if selected in line 2, results in finding a solution, and a blac card as a set Û which does not lead to a valid location for a new broer. So X equals the number of trials until first success (selecting a white card) in random sampling without replacement. As the probability that a blac card is selected in trial i is (m i + 1)/(m i + 1) and the probability that in the i th trial we draw a white card is /(m i + 1), we have m i + 1 = P (X = i) = m m m i + 2 m i + 2 (m )! (m i)! = = (m i + 1)! m! ( ) ( m i ) m i (m )!! 1 = = ( 1 m! m ). The expected value of the random variable X can be expressed as E[X] = (m )! (m i + 1)! (m i + 1)! m! m i + 1 (m i)! (m )! ( 1)! (m i + 1)! ( 1)! m! T ( m ), where According to [3] (pages ), T = m +1 i=0 i T = m ( ) m i. 1 ( m + 1 ), and so E[X] = T ( m ) = ( m+1 ( m ) ) m = m + 1 m + 1 m + 1 = m q. 5 Discussion and Conclusions The broer-placement algorithm as described in this paper does not tae into account the locality of broer placement. It should be intuitively clear that the algorithm does not have to consider all servers in the system. In order to tae over the responsibility for at least one server of an overloaded broer b, a new broer ˆb has to be placed within a distance 2h from b, where h is the distance between b and the farthest of b s servers. This means that all the servers which are located closer to their actual broer than to any of the points of the ball with radius 2h centered at b s location can be simply omitted, because it is not possible that they are assigned to ˆb. The broer-placement algorithm proposed by us has the worst-case complexity O(n d+1 ), where n is the number of servers. This means that our method is only O(n) times slower than the optimum. The expected complexity analysis shows that the performance of our algorithm

10 P. Garbaci, D.H.J. Epema, M. van Steen 9 increases with the number of distinct solutions to be linear under certain assumptions, such as a low value of the excess load that has to be removed from the overloaded broer and a high capacity of a new broer. Those assumptions are quite realistic in a real life situation. The load balancing algorithm is usually started just after the broer s load reached the broer s capacity, so the value of the excess load is fairly low. At the same time the capacity of the new broer in most of the cases is comparable with the capacity of an overloaded broer. References [1] L. Comtet. Advanced Combinatorics: The Art of Finite and Infinite Expansions. Reidel, Dortrecht, [2] M. Deza and M. Laurent. Geometry of Cuts and Metrics. Springer Verlag, January [3] R. L. Graham, D. E. Knuth, and O. Patashni. Concrete Mathematics. Addison-Wesley, October [4] E. Ng and H. Zhang. Predicting Internet Networ Distance with Coordinates-Based Approaches. In 21st INFOCOM Conf., Los Alamitos, CA, June IEEE, IEEE Computer Society Press. [5] A. Oabe, B. Boots, K. Sugihara, and S. No Chi. Spatial Tessellations: Concepts and Applications of Voronoi Diagrams. Wiley, Chichester, second edition, July [6] M. Pias, J. Crowcroft, S. Wilbur, T. Harris, and S. Bhatti. Lighthouses for Scalable Distributed Location. In Second Int l Worshop on Peer-to-Peer Systems, Berlin, February Springer-Verlag. [7] G. Pierre and M. van Steen. Design and Implementation of a User-Centered Content Delivery Networ. In Third Worshop on Internet Applications, Los Alamitos, CA, June IEEE, IEEE Computer Society Press. [8] G. Pierre, M. van Steen, and A.S. Tanenbaum. Dynamically Selecting Optimal Distribution Strategies for Web Documents. IEEE Trans. Comp., 51(6): , June [9] S. Sivasubramanian, G. Pierre, and M. van Steen. A Case for Dynamic Selection of Replication and Caching Strategies. In Eighth Web Caching Worshop, September [10] M. Szymania, G. Pierre, and M. van Steen. Latency-driven Replica Placement. Technical report, Vrije Universiteit, Department of Computer Science, [11] M. Szymania, G. Pierre, and M. van Steen. Scalable Cooperative Latency Estimation. In Tenth Int l Conf. Parallel and Distributed Systems, Los Alamitos, CA, July IEEE, IEEE Computer Society Press. [12] B. Yang and H. Garcia-Molina. Designing a Super-Peer Networ. In 19th Int l Conf. Data Engineering, Los Alamitos, CA, March IEEE, IEEE Computer Society Press.

11 10 The Problem of Broer Placement in a Peer-to-Peer Overlay Networs Appendix In this section we present proofs of some facts that we refer to in the text. Theorem 1 When the intersection of at least two different spheres in R d (d > 1) is not empty, it is a -sphere with < d 1. Proof. It is enough to show that the theorem holds for n equal to 2, so let S 1 and S 2 be two spheres in R d. We can assume that S 1 and S 2 are hyperspheres, for otherwise we can consider the intersection of S 1 and S 2 in the lowest dimensional subspace of R d that fully contains the sphere of lowest dimension. Also, we can assume that the center of S 1 has all coordinates equal to zero and the center of S 2 has all but one coordinate equal to zero. Then the intersection of S 1 and S 2 is given by the following two equations: x x x 2 d = r2 1 (1) (x 1 a) 2 + x x 2 d = r2 2 (2) with a R, a 0. Substracting Eq. (2) from Eq. (1) we obtain: 2ax 1 a 2 = r 2 1 r 2 2. (3) Now we can substitute the value of x 1 derived from Eq. (3) into the Eq. (2) to obtain a formula describing a (d 2)-dimensional sphere. Theorem 2 If S is the non-empty intersection of d + 1 d-spheres, then S is a 0-sphere or we can select d 1 out of those spheres such that their intersection is still S. Proof. Let S 1,..., S n be a set of d-spheres. According to Theorem 1, the intersection of S 1 and S 2 is either S 1 (if S 1 and S 2 are the same) or a sphere S 1 of dimension lower than d. In the first case we can omit sphere S 1 and the theorem holds. In the second case, consider the intersection of S 1 and S 3. Reasoning similarly we can either omit S 3 or we end up with a sphere S 2 of dimension lower than the dimension of S 1. If we did not omit any sphere in the first d steps then the dimension of S d = S is 0. Theorem 3 If R is the set of regions defined by the set of server balls, then for every region in R, there exists an intersection of server spheres that is contained in this region, or that is 0-sphere that has a point in this region. Proof. Let s denote the set of server balls by B, the set of spheres of balls in B by U, and the set of intersections of any number of spheres in U by S. If for any two points p 1, p 2 R d exists a region r R that contains p 1 but does not contain p 2 then we can find a ball in B which contains only one of those points (by definition of a region). Let s denote the sphere of this ball by s(p 1, p 2 ). Furthermore, if p 1 and p 2 are points of a sphere s S of dimension higher than 0, then s(p 1, p 2 ) has a non-empty intersection with s. According to Theorem 1, this intersection is a sphere of dimension lower than the dimension of s. In addition, we can find a point p in the intersection of s with s(p 1, p 2 ) such that the distance on the sphere s (geodesic) between p and p 1 is shorter than the distance on the sphere s between p 1 and p 2. Let for any region r R, s r be a sphere in S of smallest dimension that has a point in common with r. We claim that if dimension of s r is higher than 0 than s r r. If this

12 P. Garbaci, D.H.J. Epema, M. van Steen 11 claim is not true, we can find a region r R such that s r r. Let s select a point p 1 s r such that p 1 r. We consider the set S r (p 1 ) of intersections of s r with all possible spheres S(p 1, p 2 ) where p 2 s r and p 2 r. Set S r (p 1 ) is not empty because we assumed that s r r. Set S r (p 1 ) is a subset of S because both s r and S(p 1, p 2 ) belong to S. From all points in all spheres in S r (p 1 ) we select a point p which is closest (in terms of distance on the sphere s r ) to p 1. Point p belongs to region r because otherwise the intersection s r with S(p 1, p) contains a point which is closer (in terms of distance on the sphere s r ) to p 1 than p, which is in contradiction with the way p is selected. Let s select the sphere in S r (p 1 ) that contains point p. This sphere has a common point with r and is of dimension lower than s r, which is in contradiction with the way sphere s r is selected.