Advanced Distributed Systems

Transcription

1 Advanced Distributed Systems Jürgen Schönwälder December 2, / 321

2 Computer Networks and Distributed Systems Course / Lab / Seminar Semester ECTS Advanced Networking Fall 5 Advanced Networking Lab Fall 5 Advanced Distributed Systems Fall 5 Advanced Distributed Systems Lab Fall 5 Networks and Distributed Systems Spring 5 Graduate Project Spring 10 Master Thesis Proposal Fall 20 Master Thesis Spring 30 2 / 321

3 Course Content 1 Wireless Sensor Networks IEEE / 6LoWPAN Operating Systems (TinyOS / Contiki) Synchronization Algorithms Databases and Data Fusion (TinyDB) 2 Peer-to-Peer Systems Unstructured P2P Systems (Gnutella,... ) Structured P2P Systems (Chord, Pastry,... ) Application of P2P Systems Security and Anonymity in P2P Systems 3 Cloud Computing Virtual Machines and Migration (Xen,... ) Infrastructure-as-a-Service (OpenStack, Nova,... ) Platform-as-a-Service (Google AppEngine,... ) Software-as-a-Service 3 / 321

4 Grading Scheme Homeworks (30%) Individual submission of solutions Learning by solving assignments Quizzes (30%) Control your continued learning success Final examination (40%) Oral exam (maybe written if too many students) Covers the whole course 4 / 321

5 Part: Wireless Sensor Networks I 1 Introduction to Wireless Sensor Networks 2 IEEE Radio Characteristics and Topologies Frame Formats, Media Access Control, Security 3 IPv6 over IEEE (6LoWPAN) Header Compression Fragmentation and Reassembly 4 IPv6 Routing Protocol for LLNs (RPL) Instances, DODAGs, Versions, Ranks DODAG Construction and RPL ICMPv6 Messages 5 Constrained Application Protocol (CoAP) Transactions and Methods Message Formats 6 Operating Systems NesC and TinyOS Contiki 5 / 321

6 Introduction to Wireless Sensor Networks 1 Introduction to Wireless Sensor Networks 2 IEEE Radio Characteristics and Topologies Frame Formats, Media Access Control, Security 3 IPv6 over IEEE (6LoWPAN) Header Compression Fragmentation and Reassembly 4 IPv6 Routing Protocol for LLNs (RPL) Instances, DODAGs, Versions, Ranks DODAG Construction and RPL ICMPv6 Messages 5 Constrained Application Protocol (CoAP) Transactions and Methods Message Formats 6 Operating Systems NesC and TinyOS Contiki 6 / 321

7 Wireless Sensor Networks Definition A wireless sensor network is a wireless network consisting of spatially distributed autonomous devices using sensors to cooperatively monitor physical or environmental conditions, such as temperature, pressure, gases or motion. 7 / 321

8 Applications of Wireless Sensor Networks Applications Environmental monitoring Seismic detection Disaster situation monitoring and recovery Health and medical monitoring Inventory tracking and logistics Space monitoring (smart dust on mars) Smart spaces (home/office scenarios) Military surveillance 8 / 321

9 Wireless Sensor Network Motes Definition Motes are small energy efficient computers with a wireless interface and several sensors, able to operate on battery power for months or years. 9 / 321

10 Microcontroller Atmel / TI / Intel Atmel AVR ATmega bit RISC at 16 MHz, 32 registers 4kB RAM, 128kB Flash, 4kB EEPROM TI MSP bit RISC at 8 MHz, 16 registers 10kB RAM, 48kB Flash, 16kB EEPROM Intel PXA271 XScale 32 bit RISC at MHz, 16 registers 256kB SRAM, 32MB SDRAM, 32MB Flash 10 / 321

11 Microcontroller Atmel / TI / Intel Atmel AVR ATmega bit RISC at 16 MHz, 32 registers 4kB RAM, 128kB Flash, 4kB EEPROM TI MSP bit RISC at 8 MHz, 16 registers 10kB RAM, 48kB Flash, 16kB EEPROM Intel PXA271 XScale 32 bit RISC at MHz, 16 registers 256kB SRAM, 32MB SDRAM, 32MB Flash 11 / 321

12 Microcontroller Atmel / TI / Intel Atmel AVR ATmega bit RISC at 16 MHz, 32 registers 4kB RAM, 128kB Flash, 4kB EEPROM TI MSP bit RISC at 8 MHz, 16 registers 10kB RAM, 48kB Flash, 16kB EEPROM Intel PXA271 XScale 32 bit RISC at MHz, 16 registers 256kB SRAM, 32MB SDRAM, 32MB Flash 12 / 321

13 Microcontroller Power Consumption MSP430 / ATmega 128 / PXA271 XScale MSP430 AVR 128 PXA271 design TI AVR Intel mote Telos-B Mica-Z Imote2 voltage 1.8V min 2.5V min 1.3V min active 1.8mA 8mA 44 66mA sleep 5.1µA < 15µA 390µA 13 / 321

14 Radio IEEE (2003) IEEE kbps (2.4 GHz ISM band) Direct Sequence Spread Spectrum (DSSS) CSMA-CA medium access control Link encryption (AES) (no key management) Full / reduced function devices ChipCon CC2420 Popular air interface 128byte TX/RX buffer 14 / 321

15 Radio IEEE a (2007) IEEE a 100 kbps up to 1 mbps (0.5, 2-3, 6-10 GHz) Impulse Radio Ultra Wide Band and Chirp ALOHA and CSMA-CA medium access control Link encryption (AES) (no key management) Full / reduced function devices Ranging support (!) Nanotron NanoLoc Chirp technology in the 2.4 GHz ISM band Enables real time location systems 15 / 321

16 Radio Power Consumption ChipCon CC 2420 / Nanotron Nanoloc CC 2420 NanoLoc standard a data rate 250 kbps 125 kbps / 1 Mbps voltage V V receive 19.7mA 33mA transmit 17.4mA 25 30mA idle?? sleep 20µA 2µA 16 / 321

17 Research Topics Embedded systems and ad-hoc networking Energy-aware resource management Cross-layer design and optimization (Ad-hoc) mesh routing protocols Interworking and distributed algorithms Middleware for wireless sensor networks Localization, time synchronization,... Data fusion, control, actuation,... Security and novel applications 17 / 321

18 References K. Römer and F. Mattern. The Design Space of Wireless Sensor Networks. IEEE Wireless Communications, 11(6):54 61, December A. Wheeler. Commercial Applications of Wireless Sensor Networks using ZigBee. IEEE Communications Magazine, 45(4):70 77, April / 321

19 IEEE Introduction to Wireless Sensor Networks 2 IEEE Radio Characteristics and Topologies Frame Formats, Media Access Control, Security 3 IPv6 over IEEE (6LoWPAN) Header Compression Fragmentation and Reassembly 4 IPv6 Routing Protocol for LLNs (RPL) Instances, DODAGs, Versions, Ranks DODAG Construction and RPL ICMPv6 Messages 5 Constrained Application Protocol (CoAP) Transactions and Methods Message Formats 6 Operating Systems NesC and TinyOS Contiki 19 / 321

20 IEEE IEEE The IEEE standard offers physical and media access control layers for low-cost, low-speed, low-power wireless personal area networks (WPANs) Application Scenarios Home Networking Automotive Networks Industrial Networks Interactive Toys Remote Metering / 321

21 IEEE Standard Versions Original version using Direct Sequence Spread Spectrum (DSSS) with data transfer rates of 20 and 40 kbit/s Revised version using Direct Sequence Spread Spectrum (DSSS) with higher data rates and adding Parallel Sequence Spread Spectrum (PSSS) a-2007 Adding Direct Sequence Ultra-wideband (UWB) and Chirp Spread Spectrum (CSS) physical layers to the 2006 version of the standard (ranging support) 21 / 321

22 Radio Characteristics ( ) Frequencies and Data Rates 0 Channel MHz 902 MHz 2 MHz 928 MHz GHz 5 MHz GHz Frequency Channels Region Data Rate Baud Rate MHz 0 Europe 20 kbit/s 20 kbaud MHz 1-10 USA 40 kbit/s 40 kbaud MHz global 250 kbit/s 62.5 kbaud 22 / 321

23 IEEE Device Classes Full Function Device (FFD) Any topology PAN coordinator capable Talks to any other device Implements complete protocol set Reduced Function Device (RFD) Reduced protocol set Very simple implementation Cannot become a PAN coordinator Limited to leafs in more complex topologies 23 / 321

24 IEEE Definitions Network Device An RFD or FFD implementation containing an IEEE medium access control and physical interface to the wireless medium. Coordinator An FFD with network device functionality that provides coordination and other services to the network. PAN Coordinator A coordinator that is the principal controller of the PAN. A network has exactly one PAN coordinator. 24 / 321

25 IEEE Star Topology Star Topology All nodes communicate via the central PAN coordinator Leafs may be any combination of FFD and RFD devices PAN coordinator is usually having a reliable power source 25 / 321

26 IEEE Peer-to-Peer Topology Peer-to-Peer Topology Nodes can communicate via the centeral PAN coordinator and via additional point-to-point links Extension of the pure star topology 26 / 321

27 IEEE Cluster Tree Topology Cluster Tree Topology Leafs connect to a network of coordinators (FFDs) One of the coordinators serves as the PAN coordinator Clustered star topologies are an important case (e.g., each hotel room forms a star in a HVAC system) 27 / 321

28 IEEE Frame Formats General Frame Format octets: 2 1 0/2 0/2/8 Frame Sequence Destination Destination control number PAN address identifier 0/2 0/2/8 variable 2 Source PAN identifier Source address Frame payload Frame sequence check bits: 0 2 Frame type Security Frame Ack. Intra Reserved enabled pending requested PAN Dst addr Reserved mode Src addr mode IEEE 64-bit extended addresses (globally unique) 16-bit short addresses (unique within a PAN) Optional 16-bit source / destination PAN identifiers max. frame size 127 octets; max. frame header 25 octets 28 / 321

29 IEEE Frame Formats Beacon Frames Broadcasted by the coordinator to organize the network Command Frames Used for association, disassociation, data and beacon requests, conflict notification,... Data Frames Carrying user data this is what we are interested in Acknowledgement Frames Acknowledges successful data transmission (if requested) 29 / 321

30 IEEE Media Access Control Carrier Sense Multiple Access / Collision Avoidance Basic idea of the CSMA/CA algorithm: First wait until the channel is idle. Once the channel is free, start sending the data frame after some random backoff interval. Receiver acknowledges the correct reception of a data frame. If the sender does not receive an acknowledgement, retry the data transmission. 30 / 321

31 IEEE Unslotted Mode Node PAN, Node Node The sender uses CSMA/CA and the receiver sends an ACK if requested by the sender. Receiver needs to listen continuously and can t sleep. PAN Node The receiver polls the PAN whether data is available. The PAN sends an ACK followed by a data frame. Receiving node sends an ACK if requested by the sender. Coordinator needs to listen continuously and can t sleep. 31 / 321

32 IEEE Slotted Mode Superframes CSMA/CA GTS1 GTS2 GTS3 SLEEP B CAP CFP INACTIVE B A superframe consists of three periods: 1 During the Contention-Access-Period (CAP), the channel can be accessed using normal CSMA/CA. 2 The Contention-Free-Period (CFP) has Guaranteed Time Slots (GTS) assigned by the PAN to each node. 3 During the Inactive-Period (IP), the channel is not used and all nodes including the coordinator can sleep. The PAN delimits superframes using beacons. 32 / 321

33 IEEE Security Security Services Security Suite Description Null No security (default) AES-CTR Encryption only, CTR Mode AES-CBC-MAC bit MAC AES-CBC-MAC bit MAC AES-CBC-MAC bit MAC AES-CCM-128 Encryption and 128 bit MAC AES-CCM-64 Encryption and 64 bit MAC AES-CCM-32 Encryption and 32 bit MAC Key management must be provided by higher layers Implementations must support AES-CCM-64 and Null 33 / 321

34 Reading Material I IEEE. IEEE Std Technical Report , IEEE, October IEEE. IEEE Std Technical Report , IEEE, September IEEE. IEEE Std a Technical Report a-2007, IEEE, August Y. Xiao, H.-H. Chen, B. Sun, R. Wang, and S. Sethi. MAC Security and Security Overhead Analysis in the IEEE Wireless Sensor Networks. Journal on Wireless Communications and Networking, 2006:1 12, E. Callaway, P. Gorday, L. Hester, J. A. Gutierrez, M. Naeve, B. Heile, and V. Bahl. Home Networking with IEEE : A Developing Standard for Low-Rate Wireless Personal Area Networks. IEEE Communications Magazine, 40(8):70 77, August L. D. Nardis and M.-G. Di Benedetto. Overview of the IEEE /4a standards for low data rate Wireless Personal Data Networks. In Proc. of the 4th IEEE Workshop on Positioning, Navigation and Communication 2007 (WPNC 07), Hannover, March IEEE. S. Labella M. Petrova, J. Riihijarvi, P. Mahonen. Performance Study of IEEE Using Measurements and Simulations. In Proc. IEEE Wireless Communications and Networking Conference (WCNC 2006), pages , / 321

35 Reading Material II Z. Sahinoglu and S. Gezici. Ranging in the IEEE a Standard. In Proc. IEEE Wireless and Microwave Technology Conference (WAMICON 2006), December / 321

36 IPv6 over IEEE (6LoWPAN) 1 Introduction to Wireless Sensor Networks 2 IEEE Radio Characteristics and Topologies Frame Formats, Media Access Control, Security 3 IPv6 over IEEE (6LoWPAN) Header Compression Fragmentation and Reassembly 4 IPv6 Routing Protocol for LLNs (RPL) Instances, DODAGs, Versions, Ranks DODAG Construction and RPL ICMPv6 Messages 5 Constrained Application Protocol (CoAP) Transactions and Methods Message Formats 6 Operating Systems NesC and TinyOS Contiki 36 / 321

37 6LowPAN Motivation Benefits of IP over (RFC 4919) 1 The pervasive nature of IP networks allows use of existing infrastructure. 2 IP-based technologies already exist, are well-known, and proven to be working. 3 Open and freely available specifications vs. closed proprietary solutions. 4 Tools for diagnostics, management, and commissioning of IP networks already exist. 5 IP-based devices can be connected readily to other IP-based networks, without the need for intermediate entities like translation gateways or proxies. 37 / 321

38 6LowPAN Challenge Header Size Calculation... IPv6 header is 40 octets, UDP header is 8 octets MAC header can be up to 25 octets (null security) or 25+21=46 octets (AES-CCM-128) With the frame size of 127 octets, we have = 54 octets (null security) = 33 octets (AES-CCM-128) of space left for application data! IPv6 MTU Requirements IPv6 requires that links support an MTU of 1280 octets Link-layer fragmentation / reassembly is needed 38 / 321

39 6LowPAN Overview (RFC 4944) Overview The 6LowPAN protocol is an adaptation layer allowing to transport IPv6 packets over links Uses in unslotted CSMA/CA mode (strongly suggests beacons for link-layer device discovery) Based on IEEE standard Fragmentation / reassembly of IPv6 packets Compression of IPv6 and UDP/ICMP headers Mesh routing support (mesh under) Low processing / storage costs 39 / 321

40 6LowPAN Dispatch Codes All LoWPAN encapsulated datagrams are prefixed by an encapsulation header stack. Each header in the stack starts with a header type field followed by zero or more header fields. Bit Pattern Short Code Description 00 xxxxxx NALP Not A LoWPAN Packet IPv6 uncompressed IPv6 addresses LOWPAN HC1 HC1 Compressed IPv6 header LOWPAN BC0 BC0 Broadcast header ESC Additional Dispatch octet follows 10 xxxxxx MESH Mesh routing header xxx FRAG1 Fragmentation header (first) xxx FRAGN Fragmentation header (subsequent) 40 / 321

41 6LowPAN Frame Formats Uncompressed IPv6/UDP (worst case scenario) max. 127 octets preamble MAC header DSP uncompressed IPv6 header UDP payload FCS max. 23 / up to 54 / 33 2 Dispatch code ( ) indicates no compression Up to 54 / 33 octets left for payload with a max. size MAC header with null / AES-CCM-128 security The relationship of header information to application payload is obviously really bad 41 / 321

42 6LowPAN Frame Formats Compressed Link-local IPv6/UDP (best case scenario) max. 127 octets preamble MAC header DSP HC1 IPv6 UDP payload FCS max. 23 / up to 92 / 71 2 max. 127 octets preamble MAC header DSP HC1 HC2 IPv6 UDP payload FCS max. 23 / up to 97 / 76 2 Dispatch code ( ) indicates HC1 compression HC1 compression may indicate HC2 compression follows This shows the maximum compression achievable for link-local addresses (does not work for global addresses) Any non-compressable header fields are carried after the HC1 or HC1/HC2 tags (partial compression) 42 / 321

43 Header Compression Compression Principles (RFC 4944) Omit any header fields that can be calculated from the context, send the remaining fields unmodified Nodes do not have to maintain compression state (stateless compression) Support (almost) arbitrary combinations of compressed / uncompressed header fields RFC 6282 Update of RFC 4944 Stateful compression (IPHC, NHC) 43 / 321

44 Fragmentation and Reassembly Fragmentation Principles (RFC 4944) IPv6 packets to large to fit into a single frame are fragmented. A first fragment carries a header that includes the datagram size (11 bits) and a datagram tag (16 bits). Subsequent fragments carry a header that includes the datagram size, the datagram tag, and the offset (8 bits). Time limit for reassembly is 60 seconds. Ongoing Work Recovery protocol for lost fragments (RFC 4944 requires to resend the whole set of fragments) 44 / 321

45 Fragmentation and Reassembly Fragmentation Example (compressed link-local IPv6/UDP) max. 127 octets preamble MAC header FRAG1 DSP HC1 HC2 IPv6 UDP payload FCS max. 23 / max. 127 octets preamble MAC header FRAGN payload FCS max. 23 / Homework Question (consult RFC 4944 first) How many fragments are created for an 1280 octet IPv6 packet with no / maximum compression and none / AES-CCM-128 link-layer security? How many fragmented datagrams can be in transit concurrently for a source / destination pair? 45 / 321

46 Interoperability Evaluation (2009) 6LowPAN Implementations Name OS / License Hardware Maintained Jacobs TinyOS / 3BSD Telos B,... no Berkley IP TinyOS / 3BSD Telos B,... active Arch Rock TinyOS / EULA Raven,... active SICSlowpan Contiki / 3BSD Raven,... active Sensinode Own / EULA Sensinode active Hitachi Own / EULA Renesas unknown Unfortunately... The Jacobs implementation uses the TinyOS Active Message framing format and thus does not interoperate 46 / 321

47 Interoperability Evaluation (2009) Feature Comparison Feature Jacobs Berkley Contiki Arch Rock Dispatch Header Dispatch Type Mesh Header Mesh Routing - + Multicasting Header Multicasting Fragmentation HC HC2 for UDP HC1g - - o o ICMPv6 Echo = supported and tested, o = supported but not tested, - = not supported, = see [?] for details 47 / 321

48 Implementation via USB Serial Interfaces kernel space user space UDP nc6 socket tuntap serial... TCP IPv6 stack 6lowpan rewrite Serial Tun/Tap USB bridging USB 48 / 321

49 Implementation via USB Network Interfaces kernel space user space UDP nc6 socket... TCP IPv6 stack wireshark libpcap pcap BPF USB/Net USB lowpan rewrite USB RNDIS 49 / 321

50 Reading Material I K. D. Korte, I. Tumar, and J. Schönwälder. Evaluation of 6lowpan Implementations. In Proc. 4th IEEE International Workshop on Practical Issues in Building Sensor Network Applications (SenseApp 2009), pages IEEE, October N. Kushalnagar, G. Montenegro, and C. Schumacher. IPv6 over Low-Power Wireless Personal Area Networks (6LoWPANs): Overview, Assumptions, Problem Statement, and Goals. RFC 4919, Intel Corp, Microsoft Corporation, Danfoss A/S, August G. Montenegro, N. Kushalnagar, J. Hui, and D. Culler. Transmission of IPv6 Packets over IEEE Networks. RFC 4944, Microsoft Corporation, Intel Corp, Arch Rock Corp, September J. Hui and P. Thubert. Compression Format for IPv6 Datagrams over IEEE Based Networks. RFC 6282, Arch Rock Corporation, Cisco, September M. Harvan and J. Schönwälder. TinyOS Motes on the Internet: IPv6 over (6lowpan). Praxis der Informationsverarbeitung und Kommunikation, 31(4): , December / 321

51 IPv6 Routing Protocol for LLNs (RPL) 1 Introduction to Wireless Sensor Networks 2 IEEE Radio Characteristics and Topologies Frame Formats, Media Access Control, Security 3 IPv6 over IEEE (6LoWPAN) Header Compression Fragmentation and Reassembly 4 IPv6 Routing Protocol for LLNs (RPL) Instances, DODAGs, Versions, Ranks DODAG Construction and RPL ICMPv6 Messages 5 Constrained Application Protocol (CoAP) Transactions and Methods Message Formats 6 Operating Systems NesC and TinyOS Contiki 51 / 321

52 Motivation and Requirements Routing Requirements Urban LLNs [RFC5548] Industrial LLNs [RFC5673] Home Automation LLNs [RFC5826] Building Automation LLNs [RFC5867] Common Characteristics Low power and Lossy Networks (LLNs) consisting largely of constrained nodes. Lossy and unstable links, typically supporting low data rates, relatively low packet delivery rates. Traffic patterns are not simply point-to-point, but in many cases point-to-multipoint or multipoint-to-point. Potentially comprising up to thousands of nodes. 52 / 321

53 RPL Instance and DODAGs up down DODAG RPL Instance Definition An RPL Instance consists of multiple Destination Oriented Directed Acyclic Graphs (DODAGs). Traffic moves either up towards the DODAG root or down towards the DODAG leafs. 53 / 321

54 DODAG and RPL Instance Properties DODAG Properties Many-to-one communication: upwards One-to-many communication: downwards Point-to-point communication: upwards-downwards RPL Instance Properties DODAGS are disjoint (no shared nodes) Link properties: (reliability, latency,... ) Node properties: (powered or not,... ) RPL Instance has an optimization objective Multiple RPL Instances with different optimization objectives can coexist at the same time 54 / 321

55 Version Numbers and Ranks R=0 R=1 R=1 R=3 R=2 R=3 Version n+1 Version n Definition A node s Rank defines the node s individual position relative to other nodes with respect to a DODAG root. The scope of a node s Rank is a DODAG Version. 55 / 321

56 Route Construction and Forwarding Rules Route Construction Up routes towards nodes of decreasing rank (parents) Down routes towards nodes of increasing rank Nodes inform parents of their presence and reachability to descendants Source route for nodes that cannot maintain down routes Forwarding Rules All routes go upwards and/or downwards along a DODAG When going up, always forward to lower rank when possible, may forward to sibling if no lower rank exists When going down, forward based on down routes 56 / 321

57 RPL Control Messages DAG Information Object (DIO) A DIO carries information that allows a node to discover an RPL Instance, learn its configuration parameters and select DODAG parents DAG Information Solicitation (DIS) A DIS solicits a DODAG Information Object from an RPL node Destination Advertisement Object (DAO) A DAO propagates destination information upwards along the DODAG 57 / 321

58 DODAG Construction Construction Nodes periodically send link-local multicast DIO messages Stability or detection of routing inconsistencies influence the rate of DIO messages Nodes listen for DIOs and use their information to join a new DODAG, or to maintain an existing DODAG Nodes may use a DIS message to solicit a DIO Based on information in the DIOs the node chooses parents that minimize path cost to the DODAG root Comment Essentially a distance vector routing protocol with ranks to prevent count-to-infinity problems. 58 / 321

59 Reading Material I T. Winter, P. Thubert, A. Brandt, T. Clausen, J. Hui, R. Kelsey, P. Levis, K. Pister, R. Struik, and JP. Vasseur. RPL: IPv6 Routing Protocol for Low power and Lossy Networks. Internet-Draft (work in progress) <draft-ietf-roll-rpl-19>, Cisco Systems, Sigma Designs, LIX, Arch Rock Corporation, Ember Corporation, Stanford University, Dust Networks, March M. Dohler, T. Watteyne, T. Winter, and D. Barthel. Routing Requirements for Urban Low-Power and Lossy Networks. RFC 5548, CTTC, UC Berkeley, Eka Systems, France Telecom R&D, May K. Pister, P. Thubert, S. Dwars, and T. Phinney. Industrial Routing Requirements in Low-Power and Lossy Networks. RFC 5673, Dust Networks, Cisco Systems, Shell, October A. Brandt, J. Buron, and G. Porcu. Home Automation Routing Requirements in Low-Power and Lossy Networks. RFC 5826, Sigma Designs, Telecom Italia, April J. Martocci, P. De Mil, N. Riou, and W. Vermeylen. Building Automation Routing Requirements in Low-Power and Lossy Networks. RFC 5867, Johnson Controls Inc, Ghent University, Schneider Electric, Arts Centre Vooruit, June P. Levis, T. Clausen, J. Hui, O. Gnawali, and J. Ko. The Trickle Algorithm. RFC 6206, Stanford University, LIX, Arch Rock Corporation, Johns Hopkins University, March / 321

60 Reading Material II P. Levis, E. Brewer, D. Culler, D. Gay, S. Madden, N. Patel, J. Polastre, S. Shenker, R. Szewczyk, and A. Woo. The Emergence of a Networking Primitive in Wireless Sensor Networks. Communications of the ACM, 51(7):99 106, July K. Korte, A. Sehgal, and J. Schönwälder. Definition of Managed Objects for the IPv6 Routing Protocol for Low power and Lossy Networks (RPL). Internet Draft <draft-sehgal-roll-rpl-mib-01>, Jacobs University, March K. Korte. Evaluation of ContikiRPL Local Repairs. Master s thesis, Jacobs University Bremen, August / 321

61 Constrained Application Protocol (CoAP) 1 Introduction to Wireless Sensor Networks 2 IEEE Radio Characteristics and Topologies Frame Formats, Media Access Control, Security 3 IPv6 over IEEE (6LoWPAN) Header Compression Fragmentation and Reassembly 4 IPv6 Routing Protocol for LLNs (RPL) Instances, DODAGs, Versions, Ranks DODAG Construction and RPL ICMPv6 Messages 5 Constrained Application Protocol (CoAP) Transactions and Methods Message Formats 6 Operating Systems NesC and TinyOS Contiki 61 / 321

62 CoAP Overview Characteristics Constrained machine-to-machine web protocol Representational State Transfer (REST) architecture Simple proxy and caching capabilities Asynchronous transaction support Low header overhead and parsing complexity URI and content-type support UDP binding (may use IPsec or DTLS) Reliable unicast and best-effort multicast support Built-in resource discovery 62 / 321

63 Larger Picture CoAP Layers in the Protocol Stack CoAP transactions provide reliable UDP messaging CoAP methods resemble HTTP method requests and responses CoAP method calls may involve multiple CoAP transactions Roles at the transaction layer may change during a method request / response execution Application CoAP Methods CoAP Transactions UDP IPv6 / RPL 6LoWPAN / 321

64 CoAP Transactions Messages Message CON NON ACK RST Description Confirmable requests that the receiving peer sends an acknowledgement or a reset Non-confirmable messages do not request any message being sent by the receiving peer Acknowledges that a CON has been received, may carry payload Indicates that a CON has been received but some context is missing to process it Transactions are invoked peer to peer (not client/server) Transactions are identified by a Message ID (MID) 64 / 321

65 CoAP Methods Methods Method Description GET Retrieves information of an identified resource POST Creates a new resource under the requested URI PUT Updates the resource identified by an URI DELETE Deletes the resource identified by an URI Resources are identified by URIs Methods are very similar to HTTP methods Response codes are a subset of HTTP response codes Options carry additional information (similar to HTTP header lines, but using a more compact encoding) 65 / 321

66 CoAP Message Exchanges Examples Client Server Client Server Client Server CON tid=47 code=get /foo CON tid=48 code=get /foo CON tid=50 code=get /foo ACK tid=47 code=200 "..." ACK tid=48 code=000 ACK tid=50 code=000 CON tid=53 code=get /bar ACK tid=53 code=404 "..." CON tid=49 code=200 /foo "..." ACK tid=49 code=000 CON tid=51 code=200 /foo "..." RST tid=51 code=000 Synchronous transaction (left) Asynchronous transaction (middle) Orphaned transaction (right) 66 / 321

67 CoAP Message Format CoAP Header Ver T OC Code Message ID Options (if any) Payload (if any) The Ver field contains the version number, the T field the message type, and the OC field the number of options. The Code field carries the method code / response code (methods are numbers not strings). The unique Message ID is changed for every new request but not during retransmissions. 67 / 321

68 CoAP Message Format CoAP Option Format option delta length for for : option delta length The option delta identifies the option type, encoded as the delta (difference) to the previous option code. The option code implies the type of the encoded data. URI parameters are carried in options. 68 / 321

69 Reading Material I Z. Shelby, K. Hartke, C. Bormann, and B. Frank. Constrained Application Protocol (CoAP). Internet-Draft (work in progress) <draft-ietf-core-coap-07>, Sensinode, Universitaet Bremen TZI, SkyFoundry, July / 321

70 Operating Systems 1 Introduction to Wireless Sensor Networks 2 IEEE Radio Characteristics and Topologies Frame Formats, Media Access Control, Security 3 IPv6 over IEEE (6LoWPAN) Header Compression Fragmentation and Reassembly 4 IPv6 Routing Protocol for LLNs (RPL) Instances, DODAGs, Versions, Ranks DODAG Construction and RPL ICMPv6 Messages 5 Constrained Application Protocol (CoAP) Transactions and Methods Message Formats 6 Operating Systems NesC and TinyOS Contiki 70 / 321

71 NesC: Programming Language for Embedded Systems The NesC programming language is a dialect/extension of C uses static memory allocation only (no malloc/free) does whole-program analysis, efficient optimization supports race condition detection The NesC implementation essentially compiling NesC code into C program uses gcc to cross-compile the C program for the target platform statically links functions and other symbols NesC was created to implement TinyOS Developed in 2002 by a consortium led by UC Berkeley 71 / 321

72 NesC: Interfaces commands can be called by other modules think functions events signalled by other modules have to be handled by this module Interface Example interface Send { command error_t send(message_t* msg, uint8_t len); event void senddone(message_t* msg, error_t error);... } 72 / 321

73 NesC: Components Figure: NesC Interface a NesC application consists of components components provide and use interfaces components can be accessed only via interfaces (cannot call an arbitrary C-function from another module) 73 / 321

74 NesC: Modules and Configurations Figure: NesC Configurations and components modules implement interfaces configurations connect modules together via their interfaces (wiring) 74 / 321

75 NesC: Tasks Define a Task task void task_name() {... } Post a Task post task_name(); posting a task: the task is placed on an internal task queue which is processed in FIFO order a task runs to completion before the next task is run, i.e., tasks do not preempt each other however, tasks can be preempted by hardware events 75 / 321

76 NesC: Tasks when a task is executed, it runs to completion before the next task is run a task can re-post itself a post fails if and only if the task is already pending to run (it has been posted successfully and has not been invoked yet) tasks should be kept short two types of functions synchronous can only run in a task asynchronous can run in a task or in an interrupt used for race condition detection 76 / 321

77 NesC: Tasks Task Example (oops) task void computetask() { uint32_t i; for (i = 0; i < ; i++) {} } event void Timer0.fired() { call Leds.led0Toggle(); post computetask(); call Leds.led0Toggle(); } 77 / 321

78 TinyOS written in nesc event-driven architecture no kernel/user space differentiation single shared stack no process or memory management no virtual memory multi-layer abstractions components statically linked together 78 / 321

79 TinyOS: Functionality hardware abstraction access to sensors access to actuators scheduler (tasks, hardware interrupts), timer radio interfaces 6LoWPAN/IPv6 and RPL mesh-under networking algorithms storage (using flash memory on the motes) dynamic code loading TOSSIM - simulator / 321

80 TinyOS: Flashing an LED BlinkC.nc module BlinkC { uses interface Timer<TMilli> as Timer0; uses interface Leds; uses interface Boot; } implementation { event void Boot.booted() { call Timer0.startPeriodic(250); } event void Timer0.fired() { call Leds.led0Toggle(); } } 80 / 321

81 TinyOS: Flashing an LED BlinkAppC.nc configuration BlinkAppC { } implementation { components MainC, BlinkC, LedsC; components new TimerMilliC() as Timer0; } BlinkC -> MainC.Boot; BlinkC.Timer0 -> Timer0; BlinkC.Leds -> LedsC; 81 / 321

82 Contiki Overview Overview Operating system for networked embedded systems Small memory footprint, portability, no documentation Fast increasing research and industrial use Created by the Swedish Institute of Computer Science Features Protothreads Dynamic loading TCP/IP and 6LoWPAN/IPv6/RPL support Power profiling Flash file system Simulation tools... See the lab slides for details! 82 / 321

83 References I R. Sugihara and R. K. Gupta. Programming Models for Sensor Networks: A Survey. ACM Transactions on Sensor Networks, 4:8:1 8:29, April W. Dong, C. Chen, X. Liu, and J. Bu. Providing OS Support for Wireless Sensor Networks: Challenges and Approaches. IEEE Communications Surveys and Tutorials, 12(4): , D. Gay, P. Levis, R. von Behren, M. Welsh, E. Brewer, and D. Culler. The nesc Language: A Holistic Approach to Networked Embedded Systems. In Proc. of the ACM Conference on Programming Language Design and Implementation (PLDI03). ACM, June D. Gay, P. Levis, and D. Culler. Software Design Patterns for TinyOS. ACM Transactions on Embedded Computing Systems, 6, September A. Dunkels, B. Gronvall, and T. Voigt. Contiki A Lightweight and Flexible Operating System for Tiny Networked Sensors. In Proc. 29th IEEE International Conference on Local Computer Networks (LCN 04), A. Dunkels, O. Schmidt, T. Voigt, and M. Ali. Protothreads: Simplifying Event-Driven Programming of Memory-Constrained Embedded Systems. In Proc. of the ACM Conference on Embedded Networked Sensor Systems (SenSys 06), November A. Dunkels, N. Finne, J. Eriksson, and T. Voigt. Run-Time Dynamic Linking for Reprogramming Wireless Sensor Networks. In Proc. of the ACM Conference on Embedded Networked Sensor Systems (SenSys 06), November / 321

84 References II A. Dunkels, T. Voigt, and J. Alonso. Making TCP/IP Viable for Wireless Sensor Networks. In Proc. of the 1st European Workshop on Wireless Sensor Networks (EWSN 2004), January M. Durvy, J. Abeille, P. Wetterwald, C O Flynn, B. Leverett, E. Gnoske, M. Vidales, G. Mulligan, N. Tsiftes, N. Finne, and Adam Dunkels. Poster Abstract: Making Sensor Networks IPv6 Ready. In ACM SenSys 2008, N. Tsiftes, A. Dunkels, Z. He, and T. Voigt. Enabling Large-Scale Storage in Sensor Networks with the Coffee File System. In 8th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN 2009), April / 321

85 Part: Distributed Algorithms I 7 Model for Distributed Algorithms Transition System Local and Distributed Algorithms Induced Transition Systems 8 Events, Causality, Logical Clocks Events and Causal Order Computations and Executions Logical Clocks and Vector Clocks 9 Snapshots Chandy-Lamport Algorithm 10 Mutual Exclusion Centralized Algorithm Ricard Agrawala Algorithm Suzuki Kasami Algorithm 85 / 321

86 Model for Distributed Algorithms 7 Model for Distributed Algorithms Transition System Local and Distributed Algorithms Induced Transition Systems 8 Events, Causality, Logical Clocks Events and Causal Order Computations and Executions Logical Clocks and Vector Clocks 9 Snapshots Chandy-Lamport Algorithm 10 Mutual Exclusion Centralized Algorithm Ricard Agrawala Algorithm Suzuki Kasami Algorithm 86 / 321

87 Transition System Definition (transition system) A transition system is a triple S = (C,, I) where C is a set of configurations, is a binary transition relation on C, and I is a subset of C of initial configurations. Definition (execution) Let S = (C,, I) be a transition system. An execution of S is a maximal sequence E = (γ 0, γ 1,...), where γ 0 I and γ i γ i+1 for all i 0. Notes A transition relation is a subset of C C. The notation γ δ is used for (γ, δ). 87 / 321

88 Transition System Definition (reachability) Configuration δ is reachable from γ, notation γ δ, if there exists a sequence γ = γ 0, γ 1,..., γ k = δ with γ i γ i+1 for all 0 i < k. Notes A terminal configuration is a configuration γ for which there is no δ such that γ δ A sequence E(γ 0, γ 1,...) with γ i γ i+1 for all i is maximal if it is either infinite or ends in a terminal configuration Configuration δ is said to be reachable if it is reachable from an initial configuration. 88 / 321

89 Local Algorithm Definition (local algorithm) The local algorithm of a process is a quintuple (Z, I, i, s, r ), where Z is a set of states, I is a subset of Z of initial states, i is a relation on Z Z, and s and r are relations on Z M Z. The binary relation on Z is defined by c d (c, d) i m M((c, m, d) ( s r )). Notes Let M be a set of possible messages. We denote the collection of multisets with elements from M with M(M). The relations i, s, and r correspond to state transitions related with internal, send, and receive events. 89 / 321

90 Distributed Algorithm Definition (distributed algorithm) A distributed algorithm for a collection P = {p 1,..., p N } of processes is a collection of local algorithms, one for each process in P. Notes A configuration of a transition system consists of the state of each process and the collection of messages in transit The transitions are the events of the processes, which do not only affect the state of the process, but can also affect (and be affected by) the collection of messages The initial configurations are the configurations where each process is in an initial state and the message collection is empty 90 / 321

91 Induced Async. Transition System Definition (Induced Async. Transition System) The transition system S = (C,, I) is induced under asynchronous communication by a distributed algorithm for processes p 1,..., p N, where the local algorithm for process p i is (Z pi, I pi, i p i, s p i, r p i ), is given by (1) C = {(c p1,..., c pn, M) : ( p P : c p Z p ) M M(M)} (2) (see next slide) (3) I = {(c p1,..., c pn, M) : ( p P : c p I p ) M = } 91 / 321

92 Induced Async. Transition System Definition (Induced Async. Transition System (cont.)) (2) = ( p P p), where the p are the transitions corresponding to the state changes of process p; pi the set of pairs is (c p1,..., c pi,..., c pn, M 1 ), (c p1,..., c p i,..., c pn, M 2 ) for which one of the following three conditions holds: (c pi, c p i ) i p i and M 1 = M 2 for some m M, (c pi, m, c p i ) s p i for some m M, (c pi, m, c p i ) r p i and M 2 = M 1 {m} and M 1 = M 2 {m} 92 / 321

93 Induced Sync. Transition System Definition (Induced Sync. Transition System) The transition system S = (C,, I) is induced under synchronous communication by a distributed algorithm for processes p 1,..., p N, where the local algorithm for process p i is (Z pi, I pi, i p i, s p i, r p i ), is given by (1) C = {(c p1,..., c pn ) : ( p P : c p Z p )} (2) (see next slide) (3) I = {(c p1,..., c pn ) : ( p P : c p I p )} 93 / 321

94 Induced Sync. Transition System Definition (Induced Sync. Transition System (cont.)) (2) = ( p P p) ( p,q P:p q pq), where pi is the set of pairs (c p1,..., c pi,..., c pn ), (c p1,..., c p i,..., c pn ) for which (c pi, c p i ) i p i pi p j is the set of pairs (..., c pi,..., c pj,...)(..., c p i,..., c p j,...) for which there is a message m M such that (c pi, m, c p i ) s p i and (c pj, m, c p j ) r p j 94 / 321

95 Events, Causality, Logical Clocks 7 Model for Distributed Algorithms Transition System Local and Distributed Algorithms Induced Transition Systems 8 Events, Causality, Logical Clocks Events and Causal Order Computations and Executions Logical Clocks and Vector Clocks 9 Snapshots Chandy-Lamport Algorithm 10 Mutual Exclusion Centralized Algorithm Ricard Agrawala Algorithm Suzuki Kasami Algorithm 95 / 321

96 Events and Causal Order A transition a is said to occur earlier than transition b if a occures in the sequence of transitions before b An execution E = (γ 0, γ 1,...) can be associated with a sequence of events Ē = (e 0, e 1,...), where e i is the event by which the configuration changes from γ i to γ i+1 Events of a distributed execution can sometimes be interchanged without affecting the later configurations of the execution The notion of time as a total order on the events is not suitable and instead the notion of causal dependence is introduced 96 / 321

97 Dependence of Events Theorem Let γ be a configuration of a distributed system (with asynchronous message passing) and let e p and e q be events of different processes p and q, both applicable in γ. Then e p is applicable in e q (γ), e q is applicable in e p (γ), and e p (e q (γ)) = e q (e p (γ)). Let e p and e q be two events that occur consecutively in an execution. The premise of the theorem applies to these events except in the following two cases: a) p = q or b) e p is a send event, and e q is the corresponding receive event The fact that a particular pair of events cannot be exchanged is expressed by saying that there is a causal relation between these two events 97 / 321

98 Causal Order Definition (causal order) Let E be an execution. The relation, called the causal order, on the events of the execution is the smallest relation that satisfies the following requirements: (1) If e and f are different events of the same process and e occurs before f, then e f. (2) If s is a send event and r the corresponding receive event, then s r. (3) is transitive. Write a b to denote (a b a = b) The relation is a partial order and there may be events a and b for which neither a b nor b a holds Such events are said to be concurrent, notation a b 98 / 321

99 Computations The events of an execution can be reordered in any order consistent with the causal order, without affecting the result of the execution Such a reordering of the events gives rise to a different sequence of configurations, but this execution will be regarded as equivalent to the original execution Let E = (γ 0, γ 1,...) be an execution with an associated sequence of events Ē = (e 0, e 1,...), and assume f is a permutation of Ē The permutation (f 0, f 1,...) of the events of E is consistent with the causal order if f i f j implies i j, i.e., if no event is preceded in the sequence by an event it causally precedes 99 / 321

100 Equivalent Executions Theorem Let f = (f 0, f 1,...) be a permutation of the events of E that is consistent with the causal order of E. Then f defines a unique execution F starting in the initial configuration of E. F has as many events as E, and if E is finite, the last configuration of F is the same as the last configuration of E. If the conditions of this theorem apply, we say that E and F are equivalent executions, denoted as E F A global observer, who has access to the actual sequence of events, may distinguish between two equivalent executions The processes, however, cannot distinguish between two equivalent executions 100 / 321

101 Computation Definition (computation) A computation of a distributed algorithm is an equivalence class under of executions of the algorithm. It makes no sense to speak about the configurations of a computation, because different executions of the computation may not have the same configurations It does make sense to speak about the collection of events of a computation, because all executions of the computation consist of the same set of events The causal order of the events is defined for a computation 101 / 321

102 Logical Clocks Definition (clock) A clock is a function Θ from the set of events Ē to an ordered set (X, <) such that for a, b Ē Definition (lamport clock) a b Θ(a) < Θ(b). A Lamport clock is a clock function Θ L which assigns to every event a the length k of the longest sequence (e 1,..., e k ) of events satisfying e 1 e 2... e k = a. Note A clock function Θ expresses causal order, but does not necessarily express concurrency 102 / 321

103 Lamport Clocks The value of Θ L can be computed as follows: Θ L (a) is 0 if a is the first event in a process If a is an internal event or send event, and a the previous event in the same process, then Θ L (a) = Θ L (a ) + 1 If a is a receive event, a the previous event in the same process, and b the send event corresponding to a, then Θ L (a) = max(θ L (a ), Θ L (b)) + 1 The per process clock value may be combined with a process identifier to obtain a globally unique value 103 / 321

104 Lamport Clock Example P1: e1,1 e1,2 e1,3 e1,4 e1,5 P2: 1 2 e2,1 e2,2 e2,3 7 P3: e3,1 e3,2 e3,3 e3,4 e3,5 e3,6 104 / 321

105 Vector Clocks Definition (vector clocks) A vector clock for a set of N processes is a clock function Θ V which is defined by Θ V (a) = (a 1,..., a N ), where a i is the number of events e in process p i for which e a. Vectors are naturally ordered by the vector order: (a 1,... a n ) V (b 1,... b n ) i (1 i b) : a i b i Vector clocks can express concurrency since concurrent events are labelled with incomparable clock values: a b Θ V (a) < Θ V (b) Vector clocks require more space in the messages, but element compression can reduce message overhead 105 / 321

106 Vector Clock Example P1: (1,0,0) (2,0,0) (3,2,3) (4,2,3) (5,2,3) e1,1 e1,2 e1,3 e1,4 e1,5 P2: (0,1,0) (0,2,0) e2,1 e2,2 e2,3 (4,3,3) P3: (0,0,1) (0,2,2) (0,2,3) (0,2,4) (0,2,5) (0,2,6) e3,1 e3,2 e3,3 e3,4 e3,5 e3,6 106 / 321

107 Snapshots 7 Model for Distributed Algorithms Transition System Local and Distributed Algorithms Induced Transition Systems 8 Events, Causality, Logical Clocks Events and Causal Order Computations and Executions Logical Clocks and Vector Clocks 9 Snapshots Chandy-Lamport Algorithm 10 Mutual Exclusion Centralized Algorithm Ricard Agrawala Algorithm Suzuki Kasami Algorithm 107 / 321

108 Properties of Computations It is often required to analyze certain properties of a computation. An important class of properties are so called stable properties. A property P of configurations is stable if P(γ) γ δ P(δ). If a computation ever reaches a configuration γ for which P holds true, P remains true in every configuration δ from then on. Examples of stable properties: termination, deadlock, loss of tokens, non-reachable objects in dynamic memory structures,... Stable properties can be analyzed off-line by taking a snapshot of a computation. 108 / 321

109 Preliminaries Let C be a computation of a distributed system consisting of a set of P processes. The set of events of the computation C is denoted Ev. The local computation of process p consists of a sequence c p (0), c p (1),... of process states, where c p (0) is an initial state of process p. The transition from state c p (i 1) to c p (i) is marked by the occurrence of an event e (i) p. It follows that Ev = p P {e(1) p, e p (2),...}. 109 / 321

110 Preliminaries (cont.) Goal: construct a system configuration composed from local states (snapshot states). The local state c p of a process p is called its local snapshot state. If the snapshot state is c (i) p between e (i) p and e (i+1) p, i.e., p takes its snapshot, the events e p (j) with j i are called preshot events of p and the event with j > i are called postshot events of p A (global) snapshot consists of a snapshot state c p for each process p; we write S = (c p 1,..., c p N ) In time diagrams, local snapshots are depicted by open circles. 110 / 321

111 Simplification If a channel from p to q exists, then the state c p (i) of p includes a list sent pq (i) of messages that p has sent to q in the events e p (1) through e p (i). The state c q (i) of q includes a list rcvd pq (i) of messages that q has received from p in the events e p (1) through e p (i). The state of channel pq is defined to be the set of messages sent by p (according to cp) but not received by q (according to cq); that is sentpq \ rcvdpq. The simplification ensures that the channel state is recorded in the local snapshots. Note that this assumption can be lifted later on to avoid the storage of all messages. 111 / 321

112 Anomalies P1: e1,1 e1,2 e1,3 e1,4 e1,5 P2: 1 2 e2,1 e2,2 e2,3 7 P3: e3,1 e3,2 e3,3 e3,4 e3,5 e3,6 Anomalies exist if rcvd pq is not a subset of sent pq Anomalies occur if a post-shot message in the snapshot of one process is a pre-shot message in the snapshot of another process. 112 / 321

113 Feasibility, Cuts and Meaningfulness Definition Snapshot S is feasible if for each two (neighbor) processes p and q, rcvd pq sent pq. Definition A cut of Ev is a set L Ev such that e L e p e e L. Cut L 2 is said to be later than L 1 if L 1 L / 321

114 Feasibility, Cuts and Meaningfulness Definition A consistent cut of Ev is a set L Ev such that e L e e e L. Definition Snapshot S is meaningful in computation C if there exists an execution E C such that S is a configuration of E. 114 / 321

115 Feasible vs. Meaningful Snapshots Theorem Let S be a snapshot and L the cut implied by S. The following statements are equivalent. (1) S is feasible. (2) L is a consistent cut. (3) S is meaningful. The proof shows that (1) implies (2), (2) implies (3), and (3) implies (1). See Gerard Tel [?] for the details. Note that feasibility is a local property between neighbors, while meaningfulness is a global property of the snapshot. 115 / 321

116 Chandy-Lamport Algorithm var taken_p : boolean init false; To initiate the algorithm: { record the local state; taken_p := true; forall q \in Neigh_p do send <mkr> to q } If a marker has arrived: { receive <mkr>; if not taken_p { record local state; taken_p := true; forall q \in Neigh_p do send <mkr> to q } } 116 / 321

117 Chandy-Lamport Properties Processes inform each other about snapshot construction be sending special marker messages. The algorithm must be initiated by at least one process, but it works correctly if initiated by any arbitrary non-empty set of processes. The algorithm of Chandy-Lamport computes a meaningful snapshot withing finite time after its initialization by at least one process. The channels are typically assumed to be FIFO, i.e., they are not allowed to reorder messages. 117 / 321

118 Construction of the Channel State Lemma In a feasible snapshot, send pq \ rcvd pq equals the set of messages sent by p in a preshot event and received by q in a postshot event if the channels have FIFO property. Chandy-Lamport Algorithm: All preshot messages from p to q are received before the < mkr > message sent from p to q. Moreover, only preshot messages are received before the marker. The state of the channel pq is the collection of messages received by q afer recording its state but before the receipt of p s < mkr > message. 118 / 321

119 Chandy-Lamport Snapshot Example (see ft-2001) 119 / 321

120 Mutual Exclusion 7 Model for Distributed Algorithms Transition System Local and Distributed Algorithms Induced Transition Systems 8 Events, Causality, Logical Clocks Events and Causal Order Computations and Executions Logical Clocks and Vector Clocks 9 Snapshots Chandy-Lamport Algorithm 10 Mutual Exclusion Centralized Algorithm Ricard Agrawala Algorithm Suzuki Kasami Algorithm 120 / 321

121 Review: Semaphores A semaphore is a protected integer variable which can only be manipulated by the atomic operations up() and down() High-level definition of the behavior of semaphors: down(s) { s = s - 1; if (s < 0) queue_this_process_and_block(); } up(s) { s = s + 1; if (s <= 0) dequeue_and_wakeup_process(); } Dijkstra called the operations P() and V(), other popular names are wait() and signal() 121 / 321

122 Review: Critical Sections with Semaphores Mutual exclusion using semaphores semaphore mutex = 1; uncritical_section(); down(&mutex); critical_section(); up(&mutex); uncritical_section(); uncritical_section(); down(&mutex); critical_section(); up(&mutex); uncritical_section(); Rule of thumb: Every access to a shared data object must be protected by a mutex semaphore for the shared data object as shown above However, some synchronization problems require more creative usage of semaphores for proper coordination 122 / 321

123 Mutual Exclusion in Distributed Systems Semaphores rely on atomic operations on shared memory Building distributed shared memory with atomic operations requires to solve a synchronization problem as well A different approach is needed for distributed systems Message complexity should be low Failures due to lost messages or server discontinuities are important to consider 123 / 321

124 Solution #1: Centralized Coordinator Clients who want to enter a critical section send a request message to a central coordination server and wait for a reply message Clients who want to leave a critical section send a release message to the central coordination server The coordination server maintains a FIFO queue of the request messages, which establishes a global order on the synchronization requests A reply message is send to the next client waiting in the queue once a process sends a release message 124 / 321

125 Solution #1: Centralized Coordinator Properties: Conceptually very simple Three messages needed per critical secion entry Queue size bounded by the number of processes Problems: Server failure brings the whole system down Lost messages can deadlock the whole system Centralized server can become a bottleneck Can we have a fully distributed solution? 125 / 321

126 Solution #2: Ricard Agrawala Algorithm When process p i wants to enter its critical section, it generates a new timestamp, t, and sends the message request < p i, t i > to all other processes in the system. When process p j receives a request message, it may reply immediately or it may defer sending a reply back. When process p i receives a reply message from all other processes in the system, it can enter its critical section. After exiting its critical section, the process p i sends reply messages to all its deferred requests. 126 / 321

127 Solution #2: Ricard Agrawala Algorithm The decision whether process p j replies immediately to a request < p i, t i > message or defers its reply is based on three factors: 1 If p j is in its critical section, then it defers its reply to p i. 2 If p j did not request itself to enter its critical section, then it sends a reply immediately to p i. 3 If p j wants to enter its critical section but has not yet entered it, then it compares its own request timestamp t j with the timestamp t i : If its own request timestamp t j is greater than t i, then it sends a reply immediately to p i (p i asked first). Otherwise, the reply is deferred until p j leaves the requested critical section. 127 / 321

128 Ricard Agrawala Example P1 P2 P3 P4 TS=2.2 TS=2.2 TS=2.2 TS=4.1 TS=3.3 TS=5.3 TS=3.3 TS=3.4 TS=5.4 request reply TS=6.1 critical section TS=7.2 X.Y logical clock. pid 128 / 321

129 Ricard Agrawala Evaluation Properties: 2 (n 1) messages needed per critical section entry Problems: Identities of all processes in the system must be known by all processes Dynamic addition and removal of processes complicated A single failing process can break the whole scheme For more details, see [?] 129 / 321

130 Solution #3: Suzuki Kasami Algorithm When process p i wants to enter its critical section, it broadcasts a request containing its own identifier as well as a sequence number n. When process p j receives a request, it compares the last seen sequence number for p i and updates it if the received sequence number is larger. A node which holds the token and has left its critical section sends the token to a process still waiting for the token. 130 / 321

131 Solution #3: Suzuki Kasami Algorithm In order to determine which process is still waiting, the token contains vector (a 1,..., a n ). The element a i indicates the sequence number of the last time process p i was in its critical region. If a token received by p j indicates that a i is larger or equal to the sequence number known at p j for p i, then the known request for p i is outdated and ignored. 131 / 321

132 Suzuki Kasami Example P1 P2 P3 P4 (P1, 1) Q={} Q={} Q={(P1,1)} Q={(P1,1)} T=(0,0,0,0) broadcast (P3, 1) token transfer Q={(P3,1)} Q={(P3,1)} Q={(P1,1)} Q={(P1,1),(P3,1)} critical section Q={} T=(1,0,0,0) (P1, 2) Q={(P1,2),(P3,1)} Q={} Q={(P1,2)} Q={} X.Y Q={(P1,2),(P3,1)} pid. seq T=(1,0,1,0) 132 / 321

133 Suzuki Kasami Evaluation Properties: n messages needed per critical section entry Problems: Assumes an infinite sequence number space Number of processes in the system must be known by all processes Failing processes can deadlock the system Messages should be reliably transmitted For more details, see [?] 133 / 321

134 References G. Tel. Introduction to Distributed Algorithms. Cambridge University Press, 2 edition, G. Ricard and A. K. Agrawala. An Optimal Algorithm for Mutual Exclusion in Computer Networks. Communications of the ACM, 21(2):9 17, I. Suzuki and T. Kasami. A Distributed Mutual Exclusion Algorithm. ACM Transactions on Computer Systems, 3(4): , November L. Lamport. Time, Clocks, and the Ordering of Events in a Distributed System. Communications of the ACM, 21(7), July M. Raynal. About logical clocks in distributed systems. Operating Systems Review, 26(1):41 48, K. M. Chandy and L. Lamport. Distributed snapshots: Determining global states of distributed systems. ACM Transactions on Computing Systems, 3(1):63 75, E. W. Dijkstra. Cooperating Sequential Processes. Programming Languages, / 321

135 Part: Distributed Algorithms II 11 Wave Algorithms and their Properties 12 Equivalence with Related Problems Propagation of Information with Feedback Synchronization Infimum Functions 13 Wave Algorithms Waves on Ring Networks Waves on Tree Networks Waves on Arbitrary Networks Echo Algorithm 14 Traversal Algorithms Tarry s Algorithm Classical Depth-First Algorithm 135 / 321

136 Wave Algorithms and their Properties 11 Wave Algorithms and their Properties 12 Equivalence with Related Problems Propagation of Information with Feedback Synchronization Infimum Functions 13 Wave Algorithms Waves on Ring Networks Waves on Tree Networks Waves on Arbitrary Networks Echo Algorithm 14 Traversal Algorithms Tarry s Algorithm Classical Depth-First Algorithm 136 / 321

137 Motivation Message passing schemes which involve all processes are common subroutines in more specific algorithms. So called wave algorithms exchange a finite number of messages and then they make a decision, which depends causally on some event in each process. Considering wave algorithms first and in isolation will make it easier to understand more detailed algorithms. Certain problems in distributed computations can be solved by generic constructions that yield a specific algorithm. 137 / 321

138 Preliminaries We consider distributed systems where the network topology is fixed, the channels are bidirectional, the topology is free of partitions, and the system is asynchronous and there is no notion of global time. The set of all processes is denoted as P, and the set of channels by E. The number of events of computation C is denoted C and the subset of events that occur in process p is denoted C p. There is a special type of internal event called decide event (represented by the statement decide). 138 / 321

139 Wave Algorithm Definition A wave algorithm is a distributed algorithm that satisfies the following three requirements. (1) Each computation is finite: C : C < (2) Each computation contains at least one decide event: C : e C : e is a decide event (3) In each computation each decide event is causally preceded by an event in each process: C : e C : (e is decide event q P : f C q : f e) 139 / 321

140 Terminology The computation of a wave algorithm is called a wave. A process is called an initiator if it starts the execution of its local algorithm spontaneously. A non-initiator becomes involved only when a message of the algorithm arrives and triggers the execution of the local algorithm. As a consequence, the first event of an initiator is an internal or send event while the first event of a non-initiator is a receive event. An algorithm is called centralized if there must be exactly one initiator in each computation, and decentralized if the algorithm can be started spontaneously by an arbitrary subset of the processes. 140 / 321

141 Wave Properties Lemma For each event e C there exists an initiator p and an event f in C p such that f e. Lemma Let C be a wave with one initiator p and, for each non-initiator q, let father q be the neighbor of q from which q received a message in its first event. Then the graph T = (P, E T ), with E T = {qr : q p r = father q } is a spanning tree directed towards p. Lemma Let C be a wave and d p C a decide event in process p. Then q p : f C q : (f d p f is a send event) 141 / 321

142 Wave Properties (cont.) Theorem Let C be a wave with one initiator p, such that a decide event d p occurs in p. Then at least N messages are exchanged in C. Theorem Let A be a wave algorithm for arbitrary networks without initial knowledge of the neighbors identities. Then A exchanges at least E messages in each computation. 142 / 321

143 Equivalence with Related Problems 11 Wave Algorithms and their Properties 12 Equivalence with Related Problems Propagation of Information with Feedback Synchronization Infimum Functions 13 Wave Algorithms Waves on Ring Networks Waves on Tree Networks Waves on Arbitrary Networks Echo Algorithm 14 Traversal Algorithms Tarry s Algorithm Classical Depth-First Algorithm 143 / 321

144 Propagation of Information with Feedback Propagation of Information with Feedback (PIF) A subset of processes is formed by those that have a message M (all of these have the same message), which must be broadcasted, i.e., all processes must receive and accept M. Certain processes must be notified of termination of the broadcast; that is, they must execute a special notify event, with the requirement that a notify event may be executed only when all processes have already received the message M. Notification by the PIF algorithm is considered as a decide event. 144 / 321

145 PIF and Wave Algorithms Theorem Every PIF algorithm is a wave algorithm. Theorem Every wave algorithm can be employed as a PIF algorithm. The constructed PIF algorithm has the same message complexity as A but has a different bit complexity. 145 / 321

146 Synchronization Synchronization Problem In each process q an event a q must be executed, and in some processes an event b p must be executed, such that the execution of all a q events must have taken place temporally before any of the b p events is executed. In a synchronization algorithm (SYN) the b p events will be considered as decide events. 146 / 321

147 Synchronization and Wave Algorithms Theorem Every SYN algorithm is a wave algorithm. Theorem Every wave algorithm can be employed as a SYN algorithm. 147 / 321

148 Infimum Functions Definition If (X, <) is a partial order, then c is called the infimum of a and b if c a, c b, and d : (d a d b d c). Let a b denote the infimum of a and b. The binary operator is commutative and associative and can be generalized to finite sets: inf {j 1,..., j k } = j 1... j k Infimum Problem Each process q holds an input j q, which is an element of a partially ordered set X. It is required that certain processes compute the value of inf {j q : q P} and that these processes know when the computation is terminated. 148 / 321

149 Infimum Functions and Wave Algorithms Theorem Every INF algorithm is a wave algorithm. Theorem Every wave algorithm can be employed as a INF algorithm. 149 / 321

150 Infimum Theorem Theorem If is a binary operator on a set X such that (1) is commutative, i.e., a b = b a, (2) is associative, i.e., (a b) c = a (b c), and (3) is idempotent, i.e., a a = a then there is a partial order on X such that is the infimum function. Corollary,, min, max, gcd, lcm,, and of values local to the processes can be computed in a single wave. 150 / 321

151 Wave Algorithms 11 Wave Algorithms and their Properties 12 Equivalence with Related Problems Propagation of Information with Feedback Synchronization Infimum Functions 13 Wave Algorithms Waves on Ring Networks Waves on Tree Networks Waves on Arbitrary Networks Echo Algorithm 14 Traversal Algorithms Tarry s Algorithm Classical Depth-First Algorithm 151 / 321

152 Ring Algorithm For the initiator: { send <tok> to $Next_p$; receive <tok>; decide } For non-initiators: { receive <tok>; send <tok> to $Next_p$; } Theorem The ring algorithm is a wave algorithm. 152 / 321

153 Tree Algorithm var rec_p[q] foreach q \in Neigh_p : boolean init false; /* rec_p[q] is true if p has received a message from q */ { } while #{q : rec_p[q] is false} > 1 do { receive <tok> from q; rec_p[q] = true } send <tok> to q_0 with rec_p[q_0] is false; recv <tok> from q_0; rec_p[q_0] = true; decide forall q \in Neigh_p, q!= q_0 { // optional send <tok> to q } 153 / 321

154 Tree Algorithm Characteristics The tree algorithm starts from the leaves (no other node can start). Multiple nodes can decide (see example 6.4 in Tel s book) To have all nodes decide, the optional forall loop is needed. Theorem The tree algorithm is a wave algorithm. 154 / 321

155 Echo Algorithm var rec_p : integer init 0; father_p : pid init udef; For the initiator: { forall q \in Neigh_p do send <tok> to q; while rec_p < #Neigh_p { receive <tok>; rec_p++ } decide } For non-initiators: { receive <tok> from neighbor q; father_p = q; rec_p++; forall q \in Neigh_p, q!= father_p { send <tok> to q; } while rec_p < #Neigh_p { receive <tok>; rec_p++; } send <tok> to father_p; } 155 / 321

156 Echo Algorithm Characteristics The echo algorithm builds a spanning tree rooted at the initiator. The wave flowing back from the leaves is similar to the tree algorithm. Theorem The echo algorithm is a wave algorithm for arbitrary topologies. 156 / 321

157 Traversal Algorithms 11 Wave Algorithms and their Properties 12 Equivalence with Related Problems Propagation of Information with Feedback Synchronization Infimum Functions 13 Wave Algorithms Waves on Ring Networks Waves on Tree Networks Waves on Arbitrary Networks Echo Algorithm 14 Traversal Algorithms Tarry s Algorithm Classical Depth-First Algorithm 157 / 321

158 Traversal Algorithm Definition A traversal algorithm is an algorithm with the following properties: (1) In each computation there is one initiator, which starts the algorithm by sending exactly one message. (2) A process, upon receipt of a message, either sends out one message or decides. (3) The algorithm terminates in the initiator and when this happens, each process has sent a message at least once. A traversal algorithm is a special class of wave algorithms in which all events of a wave are totally ordered by the causality relation and in which the last event occurs in the same process as the first event. 158 / 321

159 Ring and Polling Algorithms Theorem The Ring Algorithm is a traversal algorithm. Theorem The sequential polling algorithm is a traversal algorithm. 159 / 321

160 Computing Sums Observation A large number of functions are not computable using general wave algorithms, e.g., the sum over all inputs, because the sum operator is not idempotent. Computing sums is possible in special cases, e.g., when the wave algorithm is a traversal algorithm, when the processes have identities, or when the algorithm induces a spanning tree. 160 / 321

161 Tarry s Traversal Algorithm A traversal algorithm for arbitrary connected networks was given by Tarry in The algorithm is formulated in the following two rules: R1 A process never forwards the token twice through the same channel. R2 A non-initiator forwards the token to its father (the neighbor from which it first received the token) only if there is no other channel possible according to rule R1. Theorem Tarry s algorithm is a traversal algorithm. Note that a node receiving the token has the freedom to choose any neighbor as long as R1 and R2 are not violated. 161 / 321

162 Tarry s Traversal Algorithm var used_p[q] : boolean init false for each q \in Neigh_p; /* Indicates whether p has already sent do q */ father_p : process init udef; For the initiator only, execute once: { father_p = p; choose q \in Neigh_p; used_p[q] = true; send <tok> to q; } For each process, upon receipt of <tok> from q_0: { if father_p = udef { father_p = q_0; } if forall q \in Neigh_p : used_p[q] { decide } else { if exists q \in Neigh_p : (q!= father_p and! used_p[q]) { choose q \in Neigh_p \ father_p with!used_p[q]; used_p[q] = true; send <tok> q; } else { used_p[father_p] = true; send <tok> father_p; } } } 162 / 321

163 Depth-First Search The classical depth-first search algorithm is obtained when the freedom to choose a neighbor to forward the token in Tarry s algorithm is restricted by adopting the following additional rule: R3 When a process receives the token it sends it back through the same channel if this is allowed by rules R1 and R2. The classical depth-first search algorithm uses 2 E messages and 2 E time units. 163 / 321

164 Classical Depth-First Algorithm var used_p[q] : boolean init false for each q \in Neigh_p; /* Indicates whether p has already sent do q */ father_p : process init udef; For the initiator only, execute once: { father_p = p; choose q \in Neigh_p; used_p[q] = true; send <tok> to q; } For each process, upon receipt of <tok> from q_0: { if father_p = udef { father_p = q_0; } if forall q \in Neigh_p : used_p[q] { decide } else { if exists q \in Neigh_p : (q!= father_p and! used_p[q]) { if father_p!= q_0 and! used_p[q_0] then q = q_0 else choose q \in Neigh_p \ father_p with!used_p[q]; used_p[q] = true; send <tok> q; } else { used_p[father_p] = true; send <tok> father_p; } } } 164 / 321

165 Classical Depth-First Improvements Can we trade message complexity against time complexity? Perhaps we can discover frond edges with additional messages in parallel, so that the discovery does not slow down the tree construction. Awerbuch s algorithm introduces vis and ack messages that are exchanged whenever a node is first reached. Neighboring nodes may then later suppress messages to this node. Awerbuch s algorithm uses 4E messages and 4N 2 time units. Cidon s algorithm further improves Awerbuch s algorithm by removing the ack messages. Cidon s algorithm uses 4E messages and 2N 2 time units. 165 / 321

166 References G. Tel. Introduction to Distributed Algorithms. Cambridge University Press, 2 edition, / 321

167 Part: Distributed Algorithms III 15 Election Algorithms Election on Tree Networks Election on Unidirectional Ring Networks LeLann s Algorithm Chang-Roberts Algorithm Election on Arbitrary Networks Echo Algorithm with Extinction 16 Robustness and Fault Tolerance Dependable Systems Fault-Tolerant Broadcasts Fault-Tolerant Broadcast Algorithms 167 / 321

168 Election Algorithms 15 Election Algorithms Election on Tree Networks Election on Unidirectional Ring Networks LeLann s Algorithm Chang-Roberts Algorithm Election on Arbitrary Networks Echo Algorithm with Extinction 16 Robustness and Fault Tolerance Dependable Systems Fault-Tolerant Broadcasts Fault-Tolerant Broadcast Algorithms 168 / 321

169 Election Algorithm Definition An election algorithm is a distributed algorithm that satisfies the following properties. (1) Each process has the same local algorithm. (2) The algorithm is decentralized, i.e., a computation can be initiated by an arbitrary non-empty subset of the processes. (3) The algorithm reaches a terminal configuration in each computation, and in each reachable terminal configuration there is exactly one process in the state leader and all other processes are in the state lost. 169 / 321

170 Election Algorithm for Trees (Idea) Use a wave initiated by the leafs to compute the minimum process id. The root decides and pushes the result down the tree so that each node can change its state to either leader or lost. To make the leafs initiate the wave, any initiating nodes send out <wakeup> messages. The wave algorithm starts when a node has received a <wakeup> message from all neighbours. 170 / 321

171 Election Algorithm for Trees var ws_p : boolean init false; wr_p : integer init 0; rec_p[q] foreach q \in Neigh_p : boolean init false; v_p : pid init p; state_p : (sleep, leader, lost) init sleep; { if p is initiator { ws_p = true; forall q \in Neigh_p { send <wakeup> to q; } } while wr_p < #Neigh_p { receive <wakeup>; wr_p++; if (! ws_p) { ws_p = true; forall q \in Neigh_p { send <wakeup> to q; } } } 171 / 321

172 Election Algorithm for Trees (cont) } while #{q : rec_p[q] is false} > 1 do { receive <tok, r> from q; rec_p[q] = true v_p = min(v_p, r); } send <tok, r> to q_0 with! rec_p[q_0]; recv <tok, r> from q_0; v_p = min(v_p, r); state = (v_p == p)? leader : lost; forall q \in Neigh_p, q!= q_0 { send <tok, v_p> to q } 172 / 321

173 Election Algorithm for Rings (Idea) Only processes initiating the election run as candidates. Collect the identity of all initiators. Once complete, compute the minimum and decide whether you become a leader or lost. 173 / 321

174 LeLann s Algorithm var List_p : set of pid init { p }; state_p : (sleep, cand, leader, lost) init sleep; { if p is initiator { state_p = cand; send <tok, p> to Next_p; receive <tok, q>; while q!= p { List_p = List_p + { q }; send <tok, q> to Next_p; receive <tok, q>; } state_p = (p == min(list_p))? leader : lost; } else { while true { receive <tok, q>; send <tok, q> to Next_p; if state_p == sleep { state_p = lost; } } } } 174 / 321

175 Chang-Roberts Algorithm var state_p : (sleep, cand, leader, lost) init sleep; { if p is initiator { state_p = cand; send <tok, p> to Next_p; while state!= leader { receive <tok, q>; if p == q { state_p = leader; } else if q < p { if state_p = cand { state_p = lost; } send <tok, q> to Next_p; } } } else { while true { receive <tok, q>; send <tok, q> to Next_p; if state_p == sleep { state_p = lost; } } } } 175 / 321

176 Discussion and Properties LeLann s algorithm assumes all channels are FIFO. Non-initiators all enter the lost state, but remain waiting for more messages forever. This can be resolved by having the leader announce that the election is over. A process can still decide when it receives the first token whether it wants to become an initiator or not. LeLann s algorithm uses wave from all initiators. The Chang-Roberts algorithm cancels waves from initiators that can t win the election. The Chang-Roberts algorithm achieves a O(N log N) message complexity in the average case, but not in the worst case. The Peterson/Dolev-Klawe-Rodeh Algorithm achieves O(N log N) message complexity also in the worst case. 176 / 321

177 Election Algorithm for Arbitrary Networks (Idea) Every process initiating an election starts a wave. The wave carries the identity of the initator. Waves carrying an identity that can t win the election are aborted (extinction) Only the wave of the initiator with the smalles identity will run to completion. 177 / 321

178 Echo Algorithm with Extinction var caw_p : pid init udef; rec_p : integer init 0; father_p : pid init udef; lrec_p : integer init 0; win_p : pid init udef; { if p is initiator { caw_p = p; forall q \in Neigh_p do send <tok, p> to q; } while lrec_p < #Neigh_p { receive msg from q; if msg is <ldr, r> { if lrec_p == 0 { forall q \in Neigh_p do send <ldr, r> to q; lrec_p++; win_p = r; } 178 / 321

179 Echo Algorithm with Extinction (cont) } } else { /* a <tok, r> message */ if r < caw_p { caw_p = r; rec_p = 0; father_p = q; forall s \in Neigh_p, s!= q { send <tok, r> to s; } } if r == caw_p { rec_p++; if rec_p == Neigh_p { if caw_p = p { forall s \in Neigh_p { send <ldr, p> to s; } } else { send <tok, r> to father_p; } } } } } state_p = (win_p == p)? leader : lost; 179 / 321

180 References G. Tel. Introduction to Distributed Algorithms. Cambridge University Press, 2 edition, / 321

181 Robustness and Fault Tolerance 15 Election Algorithms Election on Tree Networks Election on Unidirectional Ring Networks LeLann s Algorithm Chang-Roberts Algorithm Election on Arbitrary Networks Echo Algorithm with Extinction 16 Robustness and Fault Tolerance Dependable Systems Fault-Tolerant Broadcasts Fault-Tolerant Broadcast Algorithms 181 / 321

182 Dependable Systems Dependability of a computing system is the ability to deliver service that can justifiably be trusted. The service delivered by a system is its behavior as it is perceived by its user(s). A user is another system (physical or human) that interacts with the former at the service interface. The function of a system is what the system is intended for, and is described by the system specification. 182 / 321

183 Attributes of a Dependable System Availability: readiness for correct service Reliability: continuity of correct service Safety: absence of catastrophic consequences on user(s) and the environment Confidentiality: absence of unauthorized disclosure of information Integrity: absence of improper system state alterations Maintainability: ability to undergo repairs and modifications 183 / 321

184 Threats: Faults, Errors, and Failures A system failure is an event that occurs when the delivered service deviates from correct service. A failure can be seen as a transition from correct service to incorrect service. The transition from incorrect service to correct service is service restoration. An error is that part of the system state that may cause a subsequent failure. A fault is the adjudged or hypothesized cause of an error. Note that a fault may lead to an error which may lead to a failure. 184 / 321

185 Fault Models Initially dead: The fault that causes a component to not participate during the lifetime of the system. Crash fault: The fault causes the component to halt or to lose its internal state. Omission fault: A fault that causes a component to not respond to some input. Timing fault: A fault that causes a component to respond either too early or too late. Byzantine fault: An arbitrary fault which causes the component to behave in a totally arbitrary manner during failure. 185 / 321

186 Hierarchy of Fault Models initially dead crash faults omission faults byzantine faults = Incorrect computation faults are a subset of Byzantine faults where a component does not have any timing fault, but simply produces an incorrect output in response to the given input. 186 / 321

187 Benign vs. Malign Failures Initially dead processes and crashes are called benign failure types. Byzantine failures which are not bening failures are called malign failure types. For several distributed problems, it turns out that a collection of N processes can tolerate < N benign failures. 2 For several distributed problems in an asynchronous system, it turns out that a collection of N processes can tolerate < N malign failures. 3 For several distributed problems in a synchronous system, a higher level of robustness can be achieved, especially if messages can be signed. 187 / 321

188 Means to Deal with Faults Fault Prevention: Methods and procedures that prevent faults from coming into existance (e.g., rigorous programming and maintenance rules) Fault Tolerance: Mechanisms that allow a system to deliver correct service in the presence of active faults. Fault Removal: Techniques and methods aiming to remove faults during the development phase and during the operational life of a system. Fault Forecasting: Fault forecasting is conducted by performing an evaluation of the system behavior with respect to fault occurence or activation. 188 / 321

189 Approaches to Fault-Tolerance Robust Algorithms Correct processes should continue behaving correctly in spite of failures Tolerate failures by using replication and voting Never wait for all processes (Tree algorithm, Echo algorithm) because processes could fail Stabilizing Algorithms (sometimes Self-stabilizing) Correct processes might be affected by failures, but will eventually become correct The system can start in any state (possibly faulty), but should eventually resume correct behavior 189 / 321

190 Robust Decision Algorithms Robust algorithms typically try to solve some decision problem where each correct process irreversibly decides. Three requirements on decision problems: Termination: All correct processes eventually decide Consistency: Constraint on different processes decisions: Consensus problem: every decide should be equal Election: Every decide except one should be the same Non-triviality: Fixed trivial outputs (e.g., always decide yes ) are excluded; processes should need to communicate to be able to solve the problem Application: All processes in a distributed databases must agree whether to commit or abort a transaction. 190 / 321

191 Reliable Broadcasts Reliable Broadcast: All correct processes deliver the same set of messages The set only contains messages from correct processes Atomic Broadcast (reliable) A reliable broadcast where it is guaranteed that every process receives its messages in the same order as all the other processes Given a reliable atomic broadcast, we can implement a consensus algorithm Let every node broadcast either 0 or 1 Decide on the first number that is received Since every correct process will receive the messages in the same order, they will all decide on the same value Solving Reliable Atomic Broadcast is equivalent to solving consensus 191 / 321

192 Broadcast System Model P1 Pi Pn deliver() broadcast() deliver() multicast layer receive() send() receive() transport layer communication network Important distinction between send() / receive() and broadcast() / deliver() primitives 192 / 321

193 Reliable Broadcast Definition A reliable broadcast is a broadcast which satisfies the following three properties: 1 Validity: If a correct process broadcasts a message m, then all correct processes eventually deliver m. 2 Agreement: If a correct process delivers a message m, then all correct processes eventually deliver m. 3 Integrity: For any message m, every correct process delivers m at most once and only if m was previously broadcast by the sender of m. 193 / 321

194 FIFO Broadcast Example P1 x=3 x=6 x=7 P2 P3 x=3 x=3 x=6 x=7 x := 2x x := x+1 x=4 x=8 Definition A broadcast is called a FIFO broadcast if the following condition holds: If a process broadcasts a message m before it broadcasts a message m, then no correct process delivers m unless it has previously delivered m. 194 / 321

195 Causal Broadcast Example x=3 x=4 P1 x := x+1 x=8 P2 x=3 x=4 x=8 x := 2x P3 x=3 x=6 x=7 Definition A broadcast is called a causal broadcast if the following condition holds: If a broadcast of a message m causally precedes the broadcast of a message m, then no correct process delivers m unless it has previously delivered m. 195 / 321

196 Atomic Broadcast Example x=3 x=6 x=7 P1 x := 2x x=3 x=4 x=8 P2 P3 x=3 x=6 x=7 P4 x=3 x := x+1 x=4 x=8 Definition A broadcast is called an atomic or totally ordered broadcast if the following condition holds: If correct processes p and q both deliver message m and m, then p delivers m before m if and only if q delivers m before m. 196 / 321

197 Broadcast Variants Reliable Broadcast Total Order Atomic Broadcast FIFO Order FIFO Order FIFO Broadcast Total Order FIFO Atomic Broadcast Causal Order Causal Order Causal Broadcast Total Order Causal Atomic Broadcast 197 / 321

198 -Timeliness and Timed Broadcasts Definition A broadcast algorithm satisfies the -timeliness property if there is a known constant such that a message m which is broadcasted at time t is delivered by all correct processes in the interval [t, t + ]. Definition A broadcast algorithm which satisfies the -timeliness property is called a timed broadcast algorithm. A timed broadcast where the time t is measured in real-time is called a real-time timed broadcast. A timed broadcast where the time t is measured in the time of the sender is called a local-time timed broadcast. 198 / 321

199 -Timeliness and Timed Broadcasts Theorem In asynchronous systems, no reliable broadcast algorithm can satisfy either Real-Time or Local-Time -Timeliness 199 / 321

200 Reliable Broadcast broadcast(r, m): tag m with sender(m) and seq#(m); send(m) to all neighbors including p; deliver(r, m): upon receive(m) do if (p has not previously executed deliver(r, m) then if sender(m)!= p then send(m) to all neighbors; deliver(r, m); fi done Consider a system where every two correct processes are connected via apath of processes and links that never fail, then the algorithm provides a reliable broadcast. 200 / 321

201 FIFO Broadcast initialize: msgbag := {}; next[q] := 1 for all processes q; broadcast(f, m): broadcast(r, m); deliver(f, m): upon deliver(r, m) do q := sender(m); msgbag := msgbag + {m}; while (exists n in msgbag: sender(n)=q and seq#(n)=next[q]) do deliver(f, n); next[q] := next[q] + 1; msgbag := msgbag - {n}; done done 201 / 321

202 Causal Broadcast initialize: prevdlvrs := <>; broadcast(c, m): broadcast(f, <prevdlvrs m>); prevdlvrs := <>; deliver(c, m): upon receive(f, <m1,m2,...,ml>) for some l do for i := 1..l do if p has not previously executed deliver(c, mi) then deliver(c, mi); prevdlvrs := prevdlvrs mi; fi done done Probably not the most efficient way of doing things / 321

203 Timed Atomic Broadcast broadcast(a, m): broadcast(r, m); deliver(a, m): upon deliver(r, m) do schedule deliver(a, m) at time ts(m) + delta; done 203 / 321

204 FIFO Atomic Broadcast initialize: msgbag := {}; next[q] := 1 for all processes q; broadcast(fa, m): broadcast(r, m); deliver(fa, m): upon deliver(r, m) do q := sender(m); msgbag := msgbag + {m}; while (exists n in msgbag: sender(n)=q and seq#(n)=next[q]) do schedule deliver(fa, m) at time ts(m) + delta; next[q] := next[q] + 1; msgbag := msgbag - {n}; done done 204 / 321

205 Causal Atomic Broadcast initialize: prevdlvrs := {}; suspects := {}; broadcast(ca, m): broadcast(fa, <m,prevdlvrs>); prevdlvrs := {}; deliver(ca,m): upon deliver(fa, <m,d>) do if sender(m) not in suspects and p has previously executed deliver(ca, n) for all n in D deliver(ca, m); prevdlvrs := prevdlvrs + {m}; else discard(m); suspects := suspects + {sender(m)} fi done 205 / 321

206 Vector Clock Causal Atomic Broadcast initialize: msgbag := ; next[q] := 1 for all processes q; broadcast(c, m): attach(m, Vi); broadcast(r, m); update(vi); deliver(c, m): upon receive(r, m) do msgbag := msgbag + {m}; while (exists n in msgbag: seq#(n) = next[sender(n)] and k, k i, k sender(n): Vi[k] V(n)[k]) do deliver(c, m); update(vi); next[sender(n)] := next[sender(n)]+1; msgbag := msgbag - {n}; done done 206 / 321

207 References A. Avizienis, J.-C. Laprie, and B. Randell. Fundamental Concepts of Dependability. In Proc. 3rd Information Survivability Workshop (ISW-2000), October / 321

208 Part: Distributed Algorithms IV 17 Distributed Consensus Paxos Algorithm 208 / 321

209 Distributed Consensus 17 Distributed Consensus Paxos Algorithm 209 / 321

210 Problem Definition Assume a collection of processes that can propose values. A consensus algorithm ensures that a single one among the proposed values is chosen. If a value has been chosen, then processes should be able to learn the chosen value. Safety requirements: Only a value that has been proposed may be chosen. Only a single value is chosen. A process never learns that a value has been chosen unless it actually has been. Lifeness requirement / 321

211 Preliminaries We consider distributed systems where processors operate at arbitrary speed, processors may experience failures, processors with stable storage may re-join the protocol after failures, processors do not lie or otherwise divert the protocol (no Byzantine failures) processors can send messages to any other processor, messages are sent asynchronously with arbitrary delivery delays, messages may be lost, reordered, or duplicated, messages are delivered without corruption. We assume that less than half of the processors fail simultaneously. 211 / 321

212 Terminology Processes may take one or multiple roles during the execution of the algorithm: Proposer: advocates a certain value and attempts to convince acceptors to accept the proposed value. Acceptor: receives proposed values and may accept them if certain conditions are met Learner: learns about chosen values and further distributes the information Acceptors can accept proposed values. A proposed value is chosen if a large enough set of acceptors (e.g., a majority of the acceptors) have accepted it. Each value proposed by a proposer is uniquely numbered. 212 / 321

213 Invariants P1 An acceptor must accept the first proposal that it receives. P2 If a proposal with value v is chosen, then every higher-numbered proposal that is chosen has value v. P2 For any v and n, if a proposal with value v and number n is issued, then there is a set S consisting of a majority of acceptors such that either (a) no acceptor in S has accepted any proposal numbered less than n, or (b) v is the value of the higher-numbered proposal among all proposals numbered less than n accepted by the acceptors in S. 213 / 321

214 Algorithm for Issuing Proposals 1 A proposer chooses a new proposal number n and sends a request to each member of some set of acceptors, asking it to respond with: (a) A promise never again to accept a proposal numbered less than n. (b) The proposal with the higher number less than n that it has accepted, if any. Such a request is called a prepare request with number n. 2 If the proposer receives the requested responses from a majority of the acceptors, then it can issue a proposal with number n and value v, where v is the value of the higher-numbered proposal among the responses, or is any value selected by the proposer if the responders reported no proposals. 214 / 321

215 Basic Paxos Phase 1 (1a) A proposer selects a proposal number n and sends a prepare request with number n to a majority of acceptors. (1b) If an acceptor receives a prepare request with number n greater than that of any prepare request to which it has already responded, then it responds to the request with a promise not to accept any more proposals numbered less than n and with the higher-numbered proposal (if any) that it has accepted. 215 / 321

216 Basic Paxos Phase 2 (2a) If the proposer receives a response to its prepare requests (numbered n) from a majority of acceptors, then it sends an accept request to each of those acceptors for a proposal numbered n with a value v, where v is the value of the higher-numbered proposal among the responses, or is any value if the responses reported no proposals. (2b) If an acceptor receives an accept request for a proposal numbered n, it accepts the proposal unless it has already responded to a prepare request having a number greater than n. If accepted, a accepted message is returned to the proposer and any learner. 216 / 321

217 Deadlocks Two proposers can deadlock: Proposer p completes phase 1 for proposal number n 1. Proposer q then completes phase 1 for proposal number n 2 > n 1. Proposer p s accept requests are ignored by the acceptors and hence p starts another phase 1 for proposal number n 3 > n 2. Proposer q s accept requests are ignored by the acceptors and hence q starts another phase 1 for proposal number n 4 > n Solution: Elect a distinguished proposer as the only one trying to issue proposals. 217 / 321

218 Paxos Failures Multiple proposers sending conflicting prepare messages proposer starts a new round. Proposer does not receive a quorum proposer starts a new round. Proposer fails during the execution elect a new proposer. Proposer fails, new leader is elected, failed proposer returns (e.g., after a network partitioning)? 218 / 321

219 Variations Multi-Paxos: For repeated execution with the same distinguished proposer, skip phase 1 after the first execution of Paxos by adding an instance number to the accept request. Distinguished Learners: The accepted messages are sent to a single distinguished learner who informs other learners. Cheap Paxos: Allow to tolerate more processor failures by dynamically reconfiguring after each failure. Fast Paxos: Variation that reduces the message delays by collapsing prepare and accept requests. Generalized Paxos: Extension for distributed state machines allowing commutative operations that are stable or can become stable. Byzantine Paxos: Variation that handles Byzantine failures through the addition of verify messages. 219 / 321

220 References L. Lamport. The Part-Time Parliament. ACM Transactions on Computer Systems, 16(2): , May L. Lamport. Paxos Made Simple. ACM SIGACT News (Distributed Computing Column), 32(4):51 58, / 321

221 Part: Wireless Sensor Networks II 221 / 321

222 Part: Clock Synchronization 18 Motivation and Introduction 19 Lundelius and Lynch Algorithm 20 Christian s Algorithm 21 Marzullo s Algorithm 22 Intersection Algorithm 23 Time-Stamp Synchronization (TSS) 24 Reachback Firefly Algorithm (RFA) 25 Reference Broadcast Synchronization (RBS) 26 Flooding Time Synchronization Protocol (FTSP) 222 / 321

223 Motivation and Introduction 18 Motivation and Introduction 19 Lundelius and Lynch Algorithm 20 Christian s Algorithm 21 Marzullo s Algorithm 22 Intersection Algorithm 23 Time-Stamp Synchronization (TSS) 24 Reachback Firefly Algorithm (RFA) 25 Reference Broadcast Synchronization (RBS) 26 Flooding Time Synchronization Protocol (FTSP) 223 / 321

224 Why Clock Synchronization in WSNs? Motivation Time correlated sensor readings or time series data aggregation in the network Coordination of low duty cycles (power up hardware only when necessary) to increase overall network lifetime Transmission scheduling algorithms such as TDMA 224 / 321

225 Clock Synchronization in WSNs Approach #1 (external synchronization) All clocks are synchronized to a real time standard such as Coordinated Universal Time (UTC). Approach #2 (internal synchronization) Clocks are relatively synchronized to each other to provide ordering of events. 225 / 321

226 Special Constraints in WSNs Energy consumption: According to Hill et al., each transmitted bit consumes as much power as executing instructions. One goal is therefore to work with a minimum number of messages. Processing power: Sensor nodes usually use microcontrollers with limited computational power. Memory capacity: Sensor notes have limited amount of data and program memory, making it difficult to fit lengthy algorithms or to maintain larger data sets. Transmission range: The transmission range is a function of transmission power and since energy is a scarce resource, the transmission range of sensor nodes is typically short. Depending on the application scenario, there might also be issues with noise. 226 / 321

227 Sources of Synchronization Errors Send time: Time spend at the sending node from creating a message until it reaches the network interface. Access time: Time spend getting access to the channel for transmission. Depending on the MAC, the access time can vary widely. Propagation time: Travel time of a message after leaving the sender node until it is received by a receiving node. Typically very small (in the order of nano seconds in LANs). Receive time: Time needed to process an incoming message and to notify the receiving application. 227 / 321

228 Synchronization Metrics 1 Energy consumption 2 Synchronization precision 3 Scalability to support large networks with many nodes 4 Robustness against failures (self-organization) 5 Synchronization scope (local synchronization for some nodes versus global synchronization of all nodes) 6 Computational complexity 7 Memory requirements (both RAM and ROM) 8 Piggy backing of synchronization message 228 / 321

229 Criteria and Classification 1 Master/slave vs. peer-to-peer synchronization 2 Clock correction vs. untethered clocks 3 Internal vs. external synchronization 4 Probabilistic vs. deterministic synchronization bounds 5 Sender to receiver vs. receiver to receiver synchronization 6 Single-hop vs. multi-hop networks 7 Stationary vs. dynamic network topologies 229 / 321

230 Lundelius and Lynch Algorithm 18 Motivation and Introduction 19 Lundelius and Lynch Algorithm 20 Christian s Algorithm 21 Marzullo s Algorithm 22 Intersection Algorithm 23 Time-Stamp Synchronization (TSS) 24 Reachback Firefly Algorithm (RFA) 25 Reference Broadcast Synchronization (RBS) 26 Flooding Time Synchronization Protocol (FTSP) 230 / 321

231 Lundelius and Lynch Algorithm Assumptions The clock C i (t) is composed out of the hardware clock H i (t) plus a correction R i (t): C i (t) = H i (t) + R i (t) The drift of the hardware clock is limited: d(h i (t)/dt) 1 < ρ The number n of clocks in the system is known and the number of faulty clocks f is limited by 3f < n The clocks are initially approximately synchronized: T i (t 0 ) T j (t 0 ) < β The message delay is bounded by δ and ɛ such that a message arrives in the interval [δ ɛ, δ + ɛ] 231 / 321

232 Lundelius and Lynch Algorithm round = 0; T = clock(); forever { if (round > 0) { avg = average(reduce(stamps)); R_i = R_i + (T + delta - avg); } set_timer(t + DELTA); do { if (receive(message, sender)) stamps[sender] = clock; } until (timer_expired()); T = clock(); broadcast(t); set_timer(t + (1 + rho) (beta + delta + epsilon)); do { if (receive(message, sender)) stamps[sender] = clock; } until (timer_expired()); round++; } 232 / 321

233 Christian s Algorithm 18 Motivation and Introduction 19 Lundelius and Lynch Algorithm 20 Christian s Algorithm 21 Marzullo s Algorithm 22 Intersection Algorithm 23 Time-Stamp Synchronization (TSS) 24 Reachback Firefly Algorithm (RFA) 25 Reference Broadcast Synchronization (RBS) 26 Flooding Time Synchronization Protocol (FTSP) 233 / 321

234 Marzullo s Algorithm 18 Motivation and Introduction 19 Lundelius and Lynch Algorithm 20 Christian s Algorithm 21 Marzullo s Algorithm 22 Intersection Algorithm 23 Time-Stamp Synchronization (TSS) 24 Reachback Firefly Algorithm (RFA) 25 Reference Broadcast Synchronization (RBS) 26 Flooding Time Synchronization Protocol (FTSP) 234 / 321

235 Intersection Algorithm 18 Motivation and Introduction 19 Lundelius and Lynch Algorithm 20 Christian s Algorithm 21 Marzullo s Algorithm 22 Intersection Algorithm 23 Time-Stamp Synchronization (TSS) 24 Reachback Firefly Algorithm (RFA) 25 Reference Broadcast Synchronization (RBS) 26 Flooding Time Synchronization Protocol (FTSP) 235 / 321

236 Time-Stamp Synchronization (TSS) 18 Motivation and Introduction 19 Lundelius and Lynch Algorithm 20 Christian s Algorithm 21 Marzullo s Algorithm 22 Intersection Algorithm 23 Time-Stamp Synchronization (TSS) 24 Reachback Firefly Algorithm (RFA) 25 Reference Broadcast Synchronization (RBS) 26 Flooding Time Synchronization Protocol (FTSP) 236 / 321

237 Time-Stamp Synchronization (TSS) Focus on temporal relationships between events (x happened before y) Assumption that not all nodes can directly talk to each other / 321

238 Time Transformation Time Transformation The real-time time difference t can be transformed into computer clock time difference c as follows: (1 ρ) c t (1 + ρ) (1) The parameter ρ is the bound of the computer clock drift. An equivalent notation is the following: (1 ρ) t c (1 + ρ) t (2) c (1 + ρ) t c (1 ρ) (3) 238 / 321

239 Time Transformation Example In order to transform a time difference c from the local time of one node with upper bound ρ 1 to the local time of another node with upper bound ρ 2, t is first estimated by the real time interval [ ] c (1 + ρ 1 ), c (4) (1 ρ 1 ) which in turn is estimated by the computer time interval [ c (1 ρ 2) (1 + ρ 1 ), c (1 + ρ ] 2) (1 ρ 1 ) relative to the local time of the second node. (5) 239 / 321

240 Reachback Firefly Algorithm (RFA) 18 Motivation and Introduction 19 Lundelius and Lynch Algorithm 20 Christian s Algorithm 21 Marzullo s Algorithm 22 Intersection Algorithm 23 Time-Stamp Synchronization (TSS) 24 Reachback Firefly Algorithm (RFA) 25 Reference Broadcast Synchronization (RBS) 26 Flooding Time Synchronization Protocol (FTSP) 240 / 321

241 Reference Broadcast Synchronization (RBS) 18 Motivation and Introduction 19 Lundelius and Lynch Algorithm 20 Christian s Algorithm 21 Marzullo s Algorithm 22 Intersection Algorithm 23 Time-Stamp Synchronization (TSS) 24 Reachback Firefly Algorithm (RFA) 25 Reference Broadcast Synchronization (RBS) 26 Flooding Time Synchronization Protocol (FTSP) 241 / 321

242 Reference Broadcast Synchronization NIC NIC Sender Sender Receiver Receiver 1 Time Critical Path Receiver 2 Critical Path Figure 1: A critical path analysis for traditional time synchronization protocols (left) and RBS (right). For traditional protocols working on a LAN, the largest contributions to nondeterministic latency are the Send Time (from the sender s clockexploit read to delivery theofbroadcast packet to its NIC, capability including protocol of wireless processing) sensor and Access Time (the delay in the NIC until the channel becomes free). The Receive Time tends to be much smaller than the Send Time because the clock can be networks read at interrupt time, before protocol processing. In RBS, the critical path length is shortened to include only the time from the injection of the packet into the channel to the last clock read. Number of Trials Idea Reduce the critical path by eliminating send time and access time notification. Instead of synchronizing the sender with the receiver, synchronize a set of receivers with each other The phase offset distribution appears Gaussian; the chisquared test indicates 99.8% confidence in the best fit parameters µ = 0,σ = 11.1µsec. This is useful for several reasons. As we will see in later sections, a well-behaved distribution enables us to significantly improve on the error bound statistically, by sending multiple broadcast packets. In addition, Gaussian receivers 242 / 321

243 Reference Broadcast Synchronization Algorithm Individual nodes broadcast reference beacons to their neighbors (beacons do not include timestamps). Receivers timestamp the arrival of the beacons using their local clocks Receivers exchange their timestamps. The only source of errors are different propagation delays (but usually very small) and receive time errors. 243 / 321

244 Reference Broadcast Synchronization Phase Offsets Errors mainly due to receive time variations since propagation time variation is usually negligible Calculate the offset over a number of received beacons Let m be the number of beacons and let T r,b denote the time of receiver r s clock when it receives broadcast b. Then the offset o[i, j] of the notes i and j is given by: o[i, j] = 1 m m (T j,k T i,k ) (6) k=1 By averaging the time differences, we take care of phase errors. 244 / 321

245 Reference Broadcast Synchronization Clock Skew Clocks typically have a bounded frequency error, i.e., the freqency is not precisely the expected frequency. The frequency of real-world clocks can change but it tends to stay stable over time periods. Short-term instability is primarily due to environmental factors, such as variations in temperature, supply voltage, and shock. Long-term instability results from more subtle effects, such as oscillator aging. Use linear regression to handle clock screw. 245 / 321

246 Reference Broadcast Synchronization Experiment Broacast a reference beacon in a network with 5 motes Every mote has a clock resolution of 2µs to timestamp the reception times of incoming broadcasts The following plot shows points (x, y) with and the best linear fit. (x, y) = (T r1,k, T r2,k T r1,k) (7) The vertical pulses visualize the distance from each point to the best-fit line. 246 / 321

247 Reference Broacast Synchronization Linear fit seems to be a good enough approximation for clock screw. 247 / 321

248 Reference Broacast Synchronization Comparision of RBS and NTP Implemented RBS on a Linux kernel in user space Each regression is based on a window of the 30 most recent pulse reception reports Outliers are automatically rejected based on an adaptive threshold equal to 3 times the median fit error of the set of points not yet rejected Light network load: The network had very little background traffic Heavy network load: Two additional machines generated traffic by sending a series of randomly-sized UDP datagrams, each picked uniformly between 500 and 15,000 bytes (IP fragmentation being required for larger packets). The inter-packet delay was 10msec. 248 / 321

249 RBS Comparison to NTP RBS outperforms NTP and also a modified version of NTP, which allows a fairer comparison to RBS. 249 / 321

250 Flooding Time Synchronization Protocol (FTSP) 18 Motivation and Introduction 19 Lundelius and Lynch Algorithm 20 Christian s Algorithm 21 Marzullo s Algorithm 22 Intersection Algorithm 23 Time-Stamp Synchronization (TSS) 24 Reachback Firefly Algorithm (RFA) 25 Reference Broadcast Synchronization (RBS) 26 Flooding Time Synchronization Protocol (FTSP) 250 / 321

251 Flooding Time Synchronization Protocol Goals Achieve high precision clock synchronization Support sparse multi-hop network topologies Must run with limited resources (Mica2 / TinyOS) Approach Advanced MAC layer time stamping Linear regression of clock skew Flooding protocol with a dynamically elected root Graceful handling of election / join phases 251 / 321

252 Flooding Time Synchronization Protocol Clock Skew Time synchronization error between two motes. After 30 minutes, time synchronization is stopped and the initially small error results in increasing synchronization error over time. 252 / 321

253 Flooding Time Synchronization Protocol Linear Regression Distribution of error of linear regression on Mica2 motes: T = 30s time sync interval results in an average absolute error of 1.48µs and a maximum absolute error of 6.48µs T = 300s time sync interval results in an average absolute error of 2.24µs and a maximum absolute error of 8.64µs 253 / 321

254 Flooding Time Synchronization Protocol Protocol Idea Every synchronized node periodically broadcasts a synchronization message A node is synchronized if it has collected sufficient reference points Only synchronization messages from an elected synchronization root node are taken into account Message Format The synchronization message contains timestamp synchronization timestamp rootid ID of the synchronization root seqnum sequence number (assigned by sync. root) 254 / 321

255 Flooding Time Synchronization Protocol Protocol Details Nodes must ignore messages which are from an invalid synchronization root or have an old sequence number Nodes must collect sufficient reference points before they are allowed to broadcast Protocol must keep stability when a sync. root fails and a new one is elected with a different clock drift Need to handle partitions and (massive) joins gracefully 255 / 321

256 Flooding Time Synchronization Protocol event Radio.receive(TimeSync *msg) { if (msg->rootid < myrootid) myrootid = msg->rootid; else if (msg->rootid > myid msg->seqnum <= highestseqnum) return; highestseqnum = msg->seqnum; if (myrootid < myid) heartbeats = 0; if (numentries >= NUMENTRIES_LIMIT && geterror(msg) > TIME_ERROR_LIMIT) clearregressiontable(); else addentryandestimatedrift(msg); } 256 / 321

257 Flooding Time Synchronization Protocol event Timer.fired() { ++heartbeats; if (myrootid!= myid) && heartbeats >= ROOT_TIMEOUT) myrootid = myid; if (numentries >= NUMENTRIES_LIMIT myrootid == myid) { msg.rootid = myrootid; msg.seqnum = highestseqnum; Radio.send(msg); } } if (myrootid == myid) ++highestseqnum; 257 / 321

258 Flooding Time Synchronization Protocol Experiment Evaluation on a 5 12 grid of Mica / Mica2 nodes Nodes only communicate with their direct neighbors Topology enforced by software 258 / 321

259 Flooding Time Synchronization Protocol Experiment Goal: measure average pairwise error, maximum pairwise error, percentage of synchonized motes Test plan: A: at 00:04 all motes were turned on B: at 01:00 the root ID 1 was switched off and ID 2 becomes root eventually C: between 02:00 and 02:15, randomly selected nodes were reset one by one (30 seconds period) D: at 02:30 all motes with odd node IDs were switched off E: at 03:01 all motes with odd node IDs were switched on F: at 04:02 experiment ended 259 / 321

260 Flooding Time Synchronization Protocol 260 / 321