Outline. P2P and Content Distribution. P2P Definitions. More Definitions. P2P is not new. P2P Definitions. P2P Overview P2P systems P2P and DRM

Transcription

1 P2P and Content Distribution ISA 767, Secure Electronic Commerce Xinwen Zhang, George Mason University Outline P2P Overview P2P systems P2P and DRM Some slides of this talk are provided by Dr. Songqing Chen from GMU 2 P2P Definitions A class of systems and applications that employ distributed resources to perform a function in a decentralized manner. The resources encompass computing power, data (storage and content), network bandwidth, and presence (computers, human, and other resources) - by Milojicic et al. Any network that does not have fixed clients and servers, but a number of peer nodes that function as both clients and servers to the other nodes on the network - wikipedia 3 More Definitions the sharing of computer resources and services by direct exchange between systems. - Intel P2P working group SETI@home: the computers at the edge provide power and those in the middle of the network are there only to coordinate them - David Anderson Clay Shirky of O Reilly and Associate A class of applications that takes advantage of resources available at the edges of the Internet. Because accessing these decentralized resources means operating in an environment of unstable connectivity and unpredictable IP addresses, P2P nodes must operate outside the DNS system and have significant or total autonomy from central servers 4 P2P Definitions Sharing Contribution Incentive of sharing and contribution Collaboration As reported in June 2004, P2P trafficmakes up 80% traffic on the Internet, in which the share of BitTorrent is 53%. Others are edonkey, Gnutella, FastTrack, etc. 5 P2P is not new ARPANET, late 60 s Usenet Since 1979 UUCP (Unix to Unix copy protocol), NNTP (network news transport protocol) Copies files between computers without central control At the beginning, only UNC and Duke Later, using news server and ISP s server DNS Mix of p2p and hierarchical model Early Internet: Hosts.txt includes a set of names and IP addresses and copied around the Internet periodically In 1983, DNS was developed for better scalability Hierarchical DNS names Windows workgroups Network File system 6

2 Back to P2P in 21 st century (?) Napster music file sharing Powerful home computing resources High Bandwidth networking Negative views on P2P Illegal copy on copyrighted materials Too much bandwidth consumption of existing networks Too much uncontrolled/inaccurate/junk contents Distributed Resources in P2P Replicable Content files (e.g., MP3, avi) No loss of original contents Replenishable Bandwidth, CPU cycle Cannot be saved for future use Cannot be used-up for future use 7 8 P2P Applications and Functionality Distributed computing SETI@home (compute-intensive), 1996 SETI: Search for Extraterrestrial Intelligence Using computing power of home PCs to search for radio signals from extraterrestrial civilizations Data/content sharing Napster, Gnutella, FreeNet, etc. Communication and collaboration Instant Messaging (ICQ, AIM), discussion board (Groove), games (Dooms), VOIP (skype) Computing System Taxonomy Centralized systems (Mainframes) Flat Computer systems Distributed systems Client-Server Hierarchical -to- Pure Hybrid 9 10 P2P vs. Client-server Server 1 st generation of P2P networks Napster Gnutella Super DHT s Clients 11 12

3 Napster Example Hybrid P2P Bob Directory server Alice When a peer (Bob) connects to Napster, it informs center server with its IP address and song titles 2. Alice queries for a song, then central server returns bob s address 3. Alice request the song to bob and download a copy directly 13 What is Gnutella? Many hybrid application of decentralized and centralized systems Usenet (backbone or heavy-duty peers) DNS (built in hierarchy) ICQ (direct client-to client communication with backup of a server) Napster (decentralized file sharing with centralized file directory) Dramatically reduced latency Better search engine Not for illegal objects distribution 14 Overlay Networks and Gnutella Gnutella is a fully decentralized peer-to-peer protocol for locating resources Standard, not a program. There are many implementations of Gnutella (BearShare, LimeWire, Morpheus) Each member of the network may act as a client, looking for data, or a server, sharing that data Server Serv A network on top of a network Used only for the search aspect of Gnutella Actual transfer of files is not performed with the overlay network Client ent 15 Connecting to the Network Horizon example When you start the application, need to first connect to a node - bootstrapping No set way to do this. Protocol is silent on the issue 16 TTL = Node Cache contact a well known node that maintains a list of nodes Node Web Page lists locations of nodes to bootstrap and get started

4 Ping and Pong After knowing where to go to join a network, a servent uses Ping and Pong to locate other nodes to connect to Originally, Ping message sent to the node connected to, which forwarded to all neighbors All those servents willing to connect to the sender respond with Pong Pongs convey IP, Port, Files shared, and distance (hops) PING PING PING PING PING PING 23 24

5 PONG PONG PONG PONG PONG

6 Queries in Gnutella Queries work very much like the original Ping/Pong scheme Send a Query to neighbors, with TTL=7 and Hops=0 This defines the default horizon for reaching servents. As TTL increases, so does the horizon, and the number of hosts that can be reached, improving odds of a hit 31 Queries in Gnutella, Continued Send Query to neighbors, who send to their neighbors Decrement TTL, increase Hops when you receive Continue forwarding while TTL!= 0 If you have the requested file, return a Query Hit message Generates lots of messages. Called flood search 32 Query Example Transferring Data File I is need requested a out of band. guitar song Query Message Query Hit Message 33 When a file is located using the Gnutella protocol, the Query Hit message returns a host/port to contact to obtain the file The connection to get data is made using HTTP Usually (but not always) same host/port as used for Gnutella traffic Default is 6364 Problems occur when servent is behind firewall The Gnutella network is only used to locate the nodes sharing those resources. 34 Problems with Flood Query Traditional Gnutella flood query has a number of problems Very large number of packets generated to fulfill queries most searches on Gnutella can be satisfied with a search that visits fewer nodes Essentially, just a Breadth First Search (BFS) Some proposals attempt to address this Alternatives to Flood Query Iterative Deepening Directed BFS Local Indices Random Walkers with alternate schemes for searching 35 36

7 Issues Several alternatives (Local Indices, Iterative Deepening) require a global policy to be understood by all nodes Sharing information about file index (Local Indices) or even statistics (Directed BFS) leads to possible security risks Most, require at least some modification Outline Gnutella Super DHT to the servents What are Super s? High-end nodes, with better relative network bandwidth and cpu power than other nodes, that act as a centralized server to a subset of clients (leaf nodes); act as an equal in a network of superpeers; maintain an index of its clients data, answering queries on their behalf. Also called supernodes, ultrapeers. Supers Network characteristics Superpeers are equals in search. All peers are equals in download. Any query results found provides 1 response to client with all the results included. Degenerate superpeer network: clustersize=1 (pure p2p, no superpeers) A well-known example is KaZaA (Morpheus) Super Node based P2P: KaZaA (Morpheus) Super Node based P2P: KaZaA (Morpheus) super peer file get query answer leaf node queries its superpeer; superpeer answers that another leaf has the result; the peer gets its file from the local peer in the cluster; the query never traverses the network 41 super peer flooding query If superpeer doesn t have the result within the cluster, supernode sends flooding query to its supernode neighbors; leaf nodes never see or 42 process the query

8 Super Infrastructure Two-layer hierarchy Super s Clients/Leaf Nodes Cluster a superpeer and its clients Super Selection Can do at bootstrap, and during network operation Make sure it s fair, random Hosts determine whether they re eligible Ultrapeer Capable Uptime Not firewalled Suitable OS Linux, 2000/NT/XP, OS/X good - sockets Not 95/98/ME, Mac Classic Bandwidth 15KB/s downstream; 10KB/sec upstream Redundancy No Redundancy simple example Improves availability and reliability for clients. Reduces load on any single superpeer. Leaf nodes connect to at least 2 ultrapeers; default is 3; NEVER more than 10. Superpeer partners in a cluster form a virtual superpeer. Partners connect to every client, keeps a full index of each, and keeps its partners data Redundancy Super Summary Superpeers provide a balance between centralized search s inherent efficiency, and distributed search s autonomy, load balancing, and robustness. Superpeers take advantage of nodes heterogeneity of capabilities, to place the load where it is best handled. In the case where better query results may be found outside a leaf s cluster even though the query can be satisfied within the cluster, an option to extend the query beyond the cluster s superpeer may be beneficial. Would a three-tiered approach work even better? 47 48

9 Outline Gnutella Super DHT Distributed Hash Table DHT is like having a file cabinet distributed over numerous servers, explained Frans Kaashoek, a professor of Computer Science and Engineering at MIT. Interpretation: 1. It s a hash table: an efficient data structure for key lookup 2. It s distributed : each node stores part of info DHT -- An Interface From Hash Table to DHT Distributed application put(key, data) get (key) Distributed hash table data node node. node k v? K V (k1,v1) K V K V K V DHT provides the information look up service for large-scale distributed applications. Scalable, robust, self-organizing, load-balancing, secure K V Retrive(k1) The shared distributed infrastructure: relieve burdens of individual apps Insert(k1, v1) 51 a. Hash table b. Distributed hash table 52 From Hash Table to DHT (cont) Core questions when introducing distributed : How to divide a whole hash table to multiple distributed hash tables? How to reach the hash table who has the key I want, if I cannot find it from the local hash table? Requirements: Data should be identified using unique numeric keys using hash function such as SHA-1 Nodes should be willing to store keys for each other. Applications of DHT Distributed file systems Multicast overlay networks Event notification systems Distributed query processing Structured P2P systems Other distributed applications 53 54

10 Case Study: CAN CAN: Content-Addressable Network Basic Data Structure: d- dimensional Cartesian coordinate space Every key (k) is mapped to a point (x,y) in the coordinate space x = h1(k), y = h2(k); The coordinate space = key space K1 K2 K3 K4 Zone: Answer to Q1 This coordinate space is partitioned into distinct zones. Every node holds a distinct zone A node should store all keys that fall into the zone it owns K1 1 3 K2 K3 2 4 K Routing: Answer to Q2 Every node only maintains the states of its neighbors Forward lookup request to a neighbor closer to the key in the coordinate space Greedy forwarding Node A wants to lookup k3 A K1 K2 B K3 K4 57 Insertion & Retrieval in CAN 1. Node A inserts (k3, v3) 2. x3=h1(k3), y3=h2(k3) 3. Route Insertion request to (x3,y3) y3 4. (x3, y3) is in the zone of node B, so node B should store (k3,v3) in its hash table 5. Node C retrieves k3 6. Computes x3, y3 like A does 7. Route lookup request to (x3,y3) 8. Node B receives lookup request, and retrieves (k3, v3) from its hash table C A K1 K2 B K3 x3 K4 58 Join Departure Bootstrap The new node find a node already in the CAN Finding a zone Find a node randomly whose zone will be split JOIN request message Splitting Hand over part of (key, value) pairs 1 s coordinate neighbor set = {2, 3, 4, 7} 1 s coordinate neighbor set {2, 3, 4, 5} The neighbors of the 7 s coordinate neighbor set = {1, 2, 4, 5} split zone is notified so that routing can include the new node 59 Joining the routing 1.P 7 K1 2 K2 8 6 K3 K Node 5 is leaving 60

11 Departure Departure Node 7 is leaving K1 2 K2 K3 4 1 K1 2 K2 K K K Multiple Hash Functions Use k different hash functions to map a single key onto k points in the coordinate space, then k replicas Choose to retrieve an entry closest in the coordinate space, then reducing path latency 2 nd Generation of P2P Network BitTorrent Motivation How to alleviate the load & bandwidth costs of central server? Flash crowds Scalability Reliability Napster, Gnutella and DHT solved part of the problem Enabled searching over peers Weakly concern for downloading speed up 65 Motivation (2) Problems of traditional P2P network: Limited bandwidth for most peers s download queue is always full, which causes long start time for other to downloading Also causes low downloading speed Solution by BitTorrent: Using parallel downloading! Problem: how can we download from other peers having incomplete files? Key: split file into many pieces! Data integrity is important here 66

12 Motivation (3) There re large portion of free riders exist: The downloading speed is limited by the peers who are willing to upload How to encourage every peer to contribute to the content distribution? Tit-for-tat: choking those who re not or slowly uploading to you Background BitTorrent invented by Bram Cohen The first usable version of BitTorrent appeared in October 2002 BitTorrent traffic: Was negligible in May 2003 Accounted for over 9% of the sampled traffic by October 2003 [Stanislav Shalunov, Internet2 netflow weekly reports, 2003] BitTorrent s Basic Idea To best utilize parallel downloading, file is treated as many pieces: Using SHA-1 to ensure every piece s data integrity Using rarest first strategy for piece selection to speed up distribution Using a tit-for-tat algorithm (choking for the lowest speed uploader) to encourage uploading 69 Basic Framework Typically there re Web servers to serve.torrent metainfo file Tracker: maintain all peers IP/port information, so each peer can find others Downloaders (leecher/seed) as peers connected to each other 70 Illustration One seed in the central, others are leechers: Content Distribution Content Publisher Original File(s) Computing using CompleteDir.torrent file (containing a valid tracker address) Distribute.torrent file: Post.torrent file to any web server for people to download Running BitTorrent Downloader with the.torrent file Serving as a seed as long as possible 71 72

13 Overall Architecture Web Server Tracker Overall Architecture Web Server Tracker Web page with link to.torrent Web page with link to.torrent.torrent Get-announce A B C [Seed] A B C [Seed] Downloader Downloader US 73 US 74 Overall Architecture Web Server Tracker Overall Architecture Web Server Tracker Web page with link to.torrent Web page with link to.torrent A Response-peer list B C [Seed] A Shake-hand Shake-hand B C [Seed] Downloader Downloader US 75 US 76 Overall Architecture Web Server Tracker Overall Architecture Web Server Tracker Web page with link to.torrent Web page with link to.torrent A pieces B pieces C [Seed] A pieces pieces B pieces C [Seed] Downloader Downloader US 77 US 78

14 Overall Architecture Web Server Tracker Tracker s Functions Web page with link to.torrent A pieces pieces Get-announce Response-peer list B pieces C [Seed] When peers come, get the info_hash as the ID for each.torrent file For each.torrent file, tracker will maintain List of all peers who involving this file IP, port, peer_id Returns random list of peers State information Completed Downloading Downloader US Bencoding Metainfo file (.torrent) and tracker responses are in bencoding: Strings: 4:spam corresponds to 'spam' Integers: i3e corresponds to 3 Lists: l4:spam4:eggse corresponds to ['spam', 'eggs'] Dictionaries: d3:cow3:moo4:spam4:eggse corresponds to {'cow': 'moo', 'spam': 'eggs'} d4:spaml1:a1:bee corresponds to {'spam': ['a', 'b']} Metainfo files (.torrent) announce: The URL of the tracker info (a dictionary that describes the file(s) of the torrent): name: the suggested name to save the file (or directory) piece length: most commonly 2 18 = 256 K pieces: a string whose length is a multiple of 20. Every 20 bytes is the SHA1 hash of the piece at the corresponding index 83 Metainfo files (.torrent) (2) (for single file) length: the length of the file in bytes (for multi-file) files (list of dictionary): length: the same path: a list of strings corresponding to subdirectory names, the last of which is the actual file name 84

15 Tracker send: HTTP GET info_hash: 20-byte SHA1 hash of the value of the info key from the Metainfo file peer_id: 20-byte string used as a unique ID for the client (will be used in peer to peer handshake for security reason) port: The port number that the client is listening on, typically uploaded, downloaded: The total amount uploaded / downloaded so far, encoded in base ten ASCII left: The number of bytes this client still has to download event: If specified: started, completed, or stopped ip: Optional. The true IP address of the client machine. numwant: Optional. Number of peers to receive from the tracker. If omitted, typically defaults to 50 peers. 85 Tracker (2) Tracker response ("text/plain"): failure reason: A human-readable error message as to why the request failed OR -- interval: the client should wait between sending regular requests to the tracker peers: a list of dictionaries, each with the following keys: peer_id: peer's self-selected ID ip: peer's IP address (either IPv6 or IPv4) or DNS name (string) port: peer's port number (integer) 86 Tracker s Merits Light load The bandwidth overhead of the tracker is about a thousandth the total amount of bandwidth used Some minor protocol extensions will probably get it down to a ten thousandth. Can ensure the peer list contains hosts which can accept incoming connections (not firewalled): Reverse connect to newcomer And the misclassified rate is low 87 Tracker s Limits Bottleneck(?) The scalability of BitTorrent largely depends on the bandwidth of the tracker Single point of failure: When a tracker fails: no longer possible for new peers to join or for existing peers to discover each other 88 Tracker Improvement Protocol extension: announce-list: list many tracker addresses in one.torrent file So if one tracker fails, peers still can find each other Problem increase the traffic between trackers and peers Each tracker still have many clients to serve Further Improvement: Using Gnutella/DHT instead of tracker? 89 Merits of BitTorrent Speed up the content distribution Less load on the original seed (content publisher) Scalable to thousands of peers (If tracker doesn t fail) 90

16 Limits of BitTorrent (2) Random write will cause many disk fragments & even damage to disk? Random read will cause no locality for disk accessing? Attacks to BitTorrent Seed only attack If a peer connects only to seeds, it need not contribute anything to the file-sharing network. Minimal upload attack It is possible for a peer to estimate other connected peers download rate by observing the frequency of the have messages So a peer can upload minimal enough to stay within the preferred list of as many peers as possible Attacks to BitTorrent (2) Anti-snubbing attack Since the snub time is large (60 seconds by default), a peer can schedule to satisfy just one request, by uploading a chunk of a piece requested (usually 16kB), every 60 seconds and avoid getting snubbed. Malicious upload attack a peer does not download a complete piece from a single remote peer, and the hash verification is performed only after the complete piece Hence a malicious peer can upload a wrong content when requested for a chunk of the piece. But the attacked peer doesn t have any means of identifying the malicious peer. 93 P2P and DRM P2P P2P networks are good for distribution of unprotected files. Uniformed copies obtained by different peers Difficult to define rights More difficult to enforce >90% of files in P2P are unauthorized. Leveraging P2P for legal content distribution is a new problem. Current Situations P2P networks are popular. Mainly for unauthorized copies of IP Fighting between content owners/ manufactures/providers and P2P users/developers Recent Supreme Court has ruled that developers can be held responsible for the copyright infringement that their P2P file sharing products allow

17 Early DRM with Internet Content syndications Affiliate programs DRM Basic requirements of DRM: Package, publish, and protect object content Expression of digital rights Authentication of user/devices and Authorization to access/use the content Enforcement mechanisms Client side Payment integration DRM Common Architecture P2P vs. DRM Content Repository Product Info Encryption Keys Content Server DRM Packager Financial Transaction License Server DRM Rights License Generator Identities Content Package Encryption Content Metadata DRM Controller Encryption Keys Rights License Client Rendering Application Identity 99 To IP owner: P2P offers open invitations to copyright infringement and IP theft. DRM is the way to protect their revenue on Internet. To end users/consumers: P2P is the open functionality of the Internet, and provides freedom of information era. DRM restricts user behaviors. 100 P2P vs. DRM P2P and DRM As a technical issue, P2P supports many new business models with DRM P2P streaming Paid access with controlled sharing itunes Integration of DRM into P2P network Embracing the functionality of P2P Maintaining control over IP Multiple vendors in value chain: P2P publisher, IP owner, license agent, payment agent, etc

18 DRM Technology Features for P2P Networks (Rosenblatt) Reasonable usage support for users: Use on any devices Space shifting Interoperability of identity schemes for both users and devices Format conversions or transcoding DRM Technology Features for P2P Networks (Rosenblatt) Lightweight superdistribution User-defined business models and easy to implement DRM Technology Features for P2P Networks (Rosenblatt) Standards Support Rights Expression Languages Define rights for peers Network Identification Universal or interoperable identity schemes for users and devices.net Passport Liberty Alliance (Federated id) Web Services Services for DRM schemes authentication, payment, license, etc. Minimize cost and complexity for peers in P2P DRM Technology Features for P2P Networks (Rosenblatt) User Experience: Installation of the DRM has to be seamless Use cross-platform technologies (Java, XML, etc) Payment process should be integrated with ISP and other service providers. E.g., cell phone SP Content usage track should respect user privacy Gaps in Existing DRM Cost-related functionality limitations Device Tethering A content object only can be supported on a specific device. Lack of superdistribution support Complexity of integration P2P Architecture for DRM Hybrid P2P Decentralized content distribution Centralized management: License, authentication, payment, etc

19 What are Digital Containers? DigitalContainers Hybrid P2P File Trading All transactions tracked No bottlenecks DRM E-commerce Great use of resources DigitalContainers wrap files in a secure multimedia digital shell that can only be opened with a key. Simple as a password Unique as an individual s fingerprint (Biometric) Created and delivered in a patented process in which the container talks to remote authorization authorities Any combination of the above: Multi-Factor Authentication The containers are tracked perpetually as their content is passed securely from person to person, with only authorized individuals being able to access the protected content. Who, Why, Where, When, How Much? Credit/Debit/Phone card payments From Secure File Delivery System for Consumer and Enterprise -to- Networks, DigitalContainer Inc Basic Container Structure Basic P2P Business Model Basic P2P Business Model Sample P2P Integration Scheme Packager widely available and easy to use Content owners package digital goods in Digital Container They register content, set price, and agree to payment terms P2P network provider gets percentage Content owner gets percentage People get paid all along the value chain

20 External Control-based DRM Separated distribution of content and meta information Control set (or rules, policies) are distributed separated from content Service-oriented architecture for authentication, control, payment, etc DRM and P2P Pure P2P networks: trust management in P2P Web of trust Datta et al, Beyond web of trust : Enabling P2p E-commerce, IEEE International Conf. on E-Commerce,2003. Reputation management in P2P ebay