Scalable overlay Networks

Transcription

1 overlay Networks Dr. Samu Varjonen 1

2 Contents Course overview Lectures Assignments/Exercises 2

3 Course Overview Overlay networks and peer-to-peer technologies have become key components for building large scale distributed systems. This course will introduce and peer-topeer systems, discuss their general properties, and applications. The course will cover the following topics: Currently deployed peer-to-peer systems and how they work Distributed Hash Tables as a base for structured peer-topeer systems Peer-to-peer storage systems Performance issues, legal aspects, and privacy issues Overlay content distribution algorithms 3

4 General Info Advanced course, 5 credits Requirements: basics of networking Assignments/exercises done as individual work. 4

5 Lectures Lectures Monday 12:15-14:00 in C Thursday 12:15-14:00 in DK (The last one if needed) Assignments/exercises tu Exactum, C323 Course based on book S. Tarkoma. Overlay Networks: Toward Information Networking. 260 pages. CRC Press / Auerbach, February

6 The course book Some of the newer material discussed on the course are not in the book but I will provide links to papers discussing the topics. 6

7 Exam material: additional articles Article: G. DeCandia et al. Dynamo: Amazon s Highly Available Key-value Store. SOSP Article: Theory and Practice of Bloom Filters for Distributed Systems. IEEE Surveys and Tutorials Slideset: Modeling and Analysis of Anonymous-Communication Systems. J. Feigenbaum. Presentation at WITS General idea of MIX and Onion Routing are part of the exam material. Article: R. Cohen et al. Resilience of the Internet to Random Breakdowns. Phys. Rev. Lett. 85, (2000). Background info and derivation of the resiliency formula (power law lectures). Additional details not part of exam material. Article: D. Kreutz, F. M. V. Ramos, P. Verissimo, C. E. Rothenberg, S. Azodolmolky, S. Uhlig. Software-defined Networking: A Comprehensive Survey. Sections I,II, V and VI are part of the exam material. Proceedings of the IEEE, Vol. 103, Issue 1, Jan

8 Lectures MO C122 Introduction. Exercises. Motivation. TH DK117 Unstructured networks I MO C122 Unstructured networks II TH DK117 Bittorrent and evaluation MO C122 Privacy (Freenet etc.) and intro to power-law networks. TH DK117 Consistent hashing. Distributed Hash Tables (DHTs) MO C122 DHTs continued TH DK117 Power-law networks MO C122 Power-law networks and applications. TH DK117 Applications I MO C122 Applications I TH DK117 Advanced topics MO C122 Conclusions and summary TH DK117 Reserved 8

9 Grading Course grading will be based on the final exam and the assignments/exercises. Course exam :00 B123 Separate exam on :00 in B123 You can use the exercise score with the first separate exam 9

10 Assignments/Exercises Assignments are optional and will grant you extra points towards the exam. Assignments are marked and presented in the answers sessions every two weeks. You have to attend the session to get the points. Max exam points 18, max bonus points 4 Tuesdays 12:15-14:00 C323 last session Reception on questions I Answers due to questions I 6.2. Reception on questions I Answers due questions II Reception questions II Answers due to questions II 10

11 Main theme Prerequisites Approaches learning goals Meets learning goals Deepens learning goals Overlay and peerto-peer networks: definitions and systems Basics of data communications and distributed systems (Introduction to Data Communications, Distributed Systems) Knowledge of how to define the concepts of overlay and peer-topeer networks, and state their central features Ability to describe at least one system in detail Ability of being able to compare different overlay and p2p networks in a qualitative manner Ability to assess the suitability of different systems to different use cases Ability to give one s own definition of the central concepts and discuss the key design and deployment issues Distributed hash tables Basics of data communications and distributed systems (Introduction to Data Communications, Distributed Systems) Big-O-notation and basics of algorithmic complexity Knowledge of the concepts of structured and unstructured networks and the ability to classify solutions into these two categories Knowledge of the basics of distributed hash tables Ability to describe at least one distributed hash table algorithm in detail Ability of being able to compare different distributed hash table algorithms Ability of designing distributed hash table-based applications Knowledge of key performance issues of distributed hash table systems and the ability to analyze these systems The knowledge of choosing a suitable distributed hash table design for a problem Familiarity with the state of the art Reliability and performance modelling Basics of probability theory Basics of reliability in distributed systems Ability to model and assess the reliability of overlay and peer-topeer networks by using probability theory Knowledge of the most important factors pertaining to reliability Ability of analytically analyzing the reliability and performance of overlay and peer-to-peer networks Understanding of the design issues that are pertinent for reliable systems Familiarity with the state of the art Content distribution Introduction to Data Communications Knowledge of the basic content distribution solutions Ability to describe at least one overlay and p2p network based content distribution solution Knowledge of different content distribution systes and the ability to compare them in detail Knowledge of several content distribution techniques Familiarity with the state of the art Security Basics of computer security Knowledge of the basic security issues with overlay and p2p networks Knowledge of the sybil attack concept Ability to discuss how security problems and limitations can be solved Knowledge of how to prevent sybil attacks Knowledge of how to prevent sybil attacks Familiarity with the state of the art 11

12 Contact information Lecturer PhD Samu Varjonen Assignments: Juhani Toivonen Course homepage can be found: 12

13 Introduction and unstructured networks 13

14 Introduction and unstructured networks Overlay networks and intro to networking Unstructured networks 14

15 Overlay networks An overlay network is a network that is built on top of an existing network. The overlay therefore relies on the so called underlay network for basic networking functions, namely routing and forwarding. Today, most are built in the application layer on top of the TCP/IP networking suite. 15

16 Consistent hashing alleviates network problems and eventual consistency can be achieved through replication and synchronization Examples: Dynamo, Cassandra Replication, Gossip, etc. Selective flooding Consistent hash (O(1) DHT) Good for arbitrary data and search functions, can aggregate routing info, structure improves scalability Examples: Gnutella and Freenet Limited flooding / depth first / Bloom filters Example: BitTorrent Tracker Good for name/value data, note flat address space, one node is responsible, churn is a concern Examples: Lookup: Chord, CAN, Kademlia Storage: PAST Rendezvous: Scribe (for multicast), i3 DHT Search Storage Rendezvous Search Storage Rendezvous Search Storage Rendezvous Cluster Wide-area (unstructured) Wide-area (structured) Internet (TCP/IP)

17 Overlay networks Overlay technologies can be used to overcome some of the limitations of the underlay, and at the same time offering new routing and forwarding features without changing the routers. The nodes in an overlay network are connected via logical links that can span many physical links. A link between two overlay nodes may take several hops in the underlying network. 17

18 Requirements for overlay networks 1) Support the execution of one or more distributed applications by providing infrastructure for them. 2) Participate and support high-level routing and forwarding tasks. The overlay is expected to provide data forwarding capabilities that are different from those that are part of the basic Internet. 3) Deployed across the Internet in such a way that third parties can participate in the organization and operation of the overlay network. 18

19 Protocol stack Layers are part of a network architecture Provide services for layers above Hiding the complexity of the current layer Multiple layers are needed in order to reduce complexity Separation of network functions Distribution of complexity OSI, TCP/IP Protocols are building blocks of a network design Can exist independently of layering 19

20 Background What is network architecture? Layered architecture Network structure Network evolution Motivation for 20

21 Network architecture A set of principles and basic mechanisms that guide network engineering Physical links Communication protocols Format of messages The way in messages are exchanged Elements of the protocol stack Where is the state? 21

22 Naming, addressing, and routing Naming How to identify and name a node? Even if its address changes. Addressing Where is the node located? Routing How to route information to the node s address? unicast: to a specific node broadcast: to all nodes multicast: to a subset of nodes anycast: to any one in some subset (IPv6) 22

23 TCP/IP network stack Application layer (FTP, SMTP, etc.) Transport layer (TCP (TCP// UDP) UDP) Networking layer (IP) (IP) Underlying network (link (link layer, layer, physical layer) layer) 23

24 The hourglass Diverse applications Divergence Transport layer (TCP/IP) Convergence Diverse physical layers 24

25 Structure of the Internet End-to-End Principle: application logic and state at the edge, core network is simple Network maintains routing tables and forwards packets based on IP prefixes Various protocols for intra-domain routing (OSPF,...) Hierarchical structure: Autonomous Systems with tiers and peering links, economics of inter-domain routing Border Gateway Protocol (BGP) for routing between domains New requirements for IP and network: mobility, multihoming, security, multipath operation,... 25

26 Internet evolution IPv6 Caching CDNs Overlays Cloud computing and datacenter networking Software-defined Networking (SDN) SDN allows to introduce new concepts in the network Push down overlay technology into the network layer 5G Radio and Core Network 26

27 Questions Q: Where is the routing state in? A: Must be at the edge of the network. Easy to bootstrap, because network is not modified. Q: What is the benefit of this from the network point of view? A: Can overcome limitations of the current network and introduce new services. Q: What is limitation from the network point of view? A: Must take network structure into account (economics). Traversal solutions can be complex. 27