Special Topics: CSci 8980 Edge History

Transcription

1 Special Topics: CSci 8980 Edge History Jon B. Weissman Department of Computer Science University of Minnesota

2 P2P: What is it? No always-on server Nodes are at the network edge; come and go Role of clients and server are blurred Challenges?

3 Killer App: P2P file-sharing Why? Anonymity Bandwidth Storage Scalability Quickly grown in popularity Hundreds of file sharing applications

4 Searching N 1 N 2 N 3 Key= title Value=MP3 data Publisher N 4 Internet N 5 N 6 Client Lookup( title )

5 Searching (cont d) Needles vs. Haystacks Searching for top 40, or an obscure music track from 1981 that nobody s heard of? Search expressiveness File names? Attributes? Context-based search?

6 Framework Common Primitives for P2P: Join: how do I begin participating? Publish: how do I advertise my file? Search: how to I find a file? Fetch: how to I retrieve a file? What implicit primitive is missing

7 Highly unstructured Gnutella

8 Gnutella: Overview Query Flooding: Join: on startup, client contacts a few other nodes; these become its neighbors Publish: no need O(1) Search: ask neighbors, who ask their neighbors, and so on... when/if found, reply to sender. TTL limits propagation Fetch: get the file directly from peer once located; then you can serve it also

9 Gnutella: Search I have file A. I have file A. Reply Query Where is file A?

10 Gnutella: Discussion Pros: Cons:

11 KaZaA: some structure Add some structure Super peer indexes content of its contained peers

12 Distributed Hash Tables: add Goals even more structure Guaranteed lookup success Bounds on search time Scalable (no query flooding) Makes some things harder Fuzzy queries / full-text search / etc.

13 DHT: Overview Abstraction: a distributed hash-table (DHT) data structure: put(id, item); item = get(id); Implementation: nodes in system form a distributed data structure Can be Ring, Tree, Hypercube, Skip List, Butterfly Network,...

14 DHT: Overview (cont d) Structured Overlay Routing: Join: On startup, contact a bootstrap node and integrate yourself into the distributed data structure; get a node id Publish: Route publication for file id toward a close node id along the data structure

15 DHT: Overview (cont d) Search: Route a query for file id toward a close node id. Data structure guarantees that query will meet the publication Fetch: Retrieve directly from where query stops

16 DHT Example: Chord (MIT) Hash each node (IP) and file - pick from the range [0...2 m ] - m large enough to prevent collisions - placed in a ring - circular ID space: wraps - consistent hashing A key is stored at its successor: node with next higher ID not exceeding ID

17 DHT: Chord Finger/Routing/Successor Table Entry i in the finger table of node n is the first node that succeeds or equals n + 2 i The i th finger points 1/2 n-i way around the ring

18 DHT: Chord Properties: Routing table size is O(log N), where N is the total number of nodes Guarantees that a file is found in O(log N) hops Many DHTs: Pastry, CAN, Most are O(log N)

19 DHT: Chord Basic Lookup N105 N400 N120 N10 Where is key 80? N90 has K80 N32 K80 N90 N60 Suppose K80 was inserted before N90 joined?

20 DHT: Chord Join Assume an identifier space [0..7], m=3 Node n1 joins Succ. Table i id+2 i succ

21 DHT: Chord Join Node n2 joins Succ. Table i id+2 i succ Succ. Table i id+2 i succ

22 DHT: Chord Join Succ. Table Nodes n0, n6 join i id+2 i succ Succ. Table Succ. Table i id+2 i succ i id+2 i succ Succ. Table i id+2 i succ

23 DHT: Chord Join Nodes: n1, n2, n0, n6 Succ. Table i id+2 i succ Items 7 Items: f => 7 0 Succ. Table 7 1 i id+2 i succ Succ. Table 6 2 i id+2 i succ Succ. Table i id+2 i succ

24 DHT: Chord Routing Upon receiving a query for item id, a node: Checks whether stores the item locally If not, forwards the query to the largest node (id+2 i ) in its successor table that does not exceed id Succ. Table i id+2 i succ Succ. Table 7 1 i id+2 i succ query(7) Succ. Table i id+2 i succ Items 7 Succ. Table i id+2 i succ

25 DHT: Discussion Pros: Cons:

26 Grid Computing Harnessing resources all across the Internet for high performance or high throughput applications edge not only stores data but computes

27 Grids BOINC Legion: everything is an object Globus: services-based Condor: job or task based

28 What is Condor? Condor converts a collection of unrelated machines into a high-throughput computing facility machine owners opt-in Condor uses matchmaking to ensure that jobs and resources match up environment is highly heterogeneous in both jobs and resources

29 What is High-Throughput Computing? High-performance: CPU cycles/second under ideal circumstances How fast can I run simulation X on this machine? High-throughput: CPU cycles/day (week, month, year?) under non-ideal circumstances How many times can I run simulation X in the next week using all available machines? How much science can I get done in the next hour?

30 What is High-Throughput Computing? Condor will execute your job under numerous machine failures crash!, disconnection, disk space exhausted relies on checkpointing and migration Nodes can be removed or added from the Condor machine pool nodes are shared

31 Condor Architecture Matchmaker Central Manager User Problem Solver (DAGMAN) Agent schedd Resource Startd. (Master-Worker) Shadow Shadow Sandbox Starter Job Condor Pool: s of Resources

32 What is Matchmaking? Condor uses Matchmaking to make sure that work gets done within the constraints of both users and owners Users (jobs) have constraints: I need an Sun Sparc with 256 MB RAM Owners (machines) have constraints: Only run jobs when I am away from my desk and never run jobs owned by Bob.

33 Remote Execution Job needs to execute on a remote machine What are the issues?

34 Machine States Most machines will be: Owner: The machine s owner is busy at the console, so no Condor jobs may run Condor will seamlessly migrate jobs via checkpointing (rerun on identical machine) Claimed: Condor has selected the machine to run jobs for other users

35 Machine States Only a few should be: Unclaimed: The owner is gone, but Condor has not yet selected the machine Matched: Between claimed and unclaimed Preempting: Condor is busy removing/migrating a job

36 Checkpointing When machine owner returns, job can be checkpointed and restarted on another machine Periodic checkpoint feature can periodically checkpoint the job so that work is not lost should the job be migrated Condor jobs vs. vanilla jobs Condor job executables must be relinked and can be checkpointed, migrated and restarted Vanilla jobs are not relinked and cannot be checkpointed and migrated

37 Matchmaking: ClassAds ClassAds are a simple language for describing both the properties and the requirements of jobs and machines

38 ClassAd for a Machine MyType = "Machine" TargetType = "Job" Name = caesar.cs.umn.edu" START = TRUE VirtualMemory = Disk = Memory = 160 Cpus = 1 Arch = Sparc" OpSys = Solaris

39 ClassAd for a Job MyType = "Job" TargetType = "Machine" Owner = jon" Cmd = "/tmp_mnt/usr/users/jon/test/fib" Out = fib.out.49 Args = 49 ImageSize = 2544 DiskUsage = 2544 Requirements = (Arch == Sparc") && (OpSys == Solaris") && (Disk >= DiskUsage) && (VirtualMemory >= ImageSize)

40 Condor Flocking

41 Next week Start mobile outsourcing to the edge Need paper volunteers! Have a great weekend!