Advanced Distributed Systems

Transcription

1 November 26, 2007

2 Part: Introduction 1 Definition and Applications 2 Wireless Sensor Network Motes 3 Research Topics 4 Motes on the Internet (6lowpan)

3 Definition and Applications 1 Definition and Applications 2 Wireless Sensor Network Motes 3 Research Topics 4 Motes on the Internet (6lowpan)

4 Wireless Sensor Networks Definition A wireless sensor network is a wireless network consisting of spatially distributed autonomous devices using sensors to cooperatively monitor physical or environmental conditions, such as temperature, pressure, gases or motion.

5 Applications of Wireless Sensor Networks Applications Environmental monitoring Seismic detection Disaster situation monitoring and recovery Health and medical monitoring Inventory tracking and logistics Space monitoring (smart dust on mars) Smart spaces (home/office scenarios) Military surveillance

6 Wireless Sensor Network Motes 1 Definition and Applications 2 Wireless Sensor Network Motes 3 Research Topics 4 Motes on the Internet (6lowpan)

7 Wireless Sensor Network Motes Definition Motes are small energy efficient computers with a wireless interface and several sensors, able to operate on battery power for months or years.

8 Microcontroller Atmel / TI / Intel Atmel AVR ATmega bit RISC at 16 MHz, 32 registers 4kB RAM, 128kB Flash, 4kB EEPROM TI MSP bit RISC at 8 MHz, 16 registers 10kB RAM, 48kB Flash, 16kB EEPROM Intel PXA271 XScale 32 bit RISC at MHz, 16 registers 256kB SRAM, 32MB SDRAM, 32MB Flash

9 Microcontroller Atmel / TI / Intel Atmel AVR ATmega bit RISC at 16 MHz, 32 registers 4kB RAM, 128kB Flash, 4kB EEPROM TI MSP bit RISC at 8 MHz, 16 registers 10kB RAM, 48kB Flash, 16kB EEPROM Intel PXA271 XScale 32 bit RISC at MHz, 16 registers 256kB SRAM, 32MB SDRAM, 32MB Flash

10 Microcontroller Atmel / TI / Intel Atmel AVR ATmega bit RISC at 16 MHz, 32 registers 4kB RAM, 128kB Flash, 4kB EEPROM TI MSP bit RISC at 8 MHz, 16 registers 10kB RAM, 48kB Flash, 16kB EEPROM Intel PXA271 XScale 32 bit RISC at MHz, 16 registers 256kB SRAM, 32MB SDRAM, 32MB Flash

11 Microcontroller Power Consumption MSP430 / ATmega 128 / PXA271 XScale MSP430 AVR 128 PXA271 design TI AVR Intel mote Telos-B Mica-Z Imote2 voltage 1.8V min 2.5V min 1.3V min active 1.8mA 8mA 44 66mA sleep 5.1µA < 15µA 390µA

12 Radio IEEE (2003) IEEE kbps (2.4 GHz ISM band) Direct Sequence Spread Spectrum (DSSS) CSMA-CA medium access control Link encryption (AES) (no key management) Full / reduced function devices ChipCon CC2420 Popular air interface 128byte TX/RX buffer

13 Radio IEEE (2003) IEEE kbps (2.4 GHz ISM band) Direct Sequence Spread Spectrum (DSSS) CSMA-CA medium access control Link encryption (AES) (no key management) Full / reduced function devices ChipCon CC2420 Popular air interface 128byte TX/RX buffer

14 Radio IEEE a (2007) IEEE a 100 kbps up to 1 mbps (0.5, 2-3, 6-10 GHz) Impulse Radio Ultra Wide Band and Chirp ALOHA and CSMA-CA medium access control Link encryption (AES) (no key management) Full / reduced function devices Ranging support (!) Nanotron NanoLoc Chirp technology in the 2.4 GHz ISM band Enables real time location systems

15 Radio IEEE a (2007) IEEE a 100 kbps up to 1 mbps (0.5, 2-3, 6-10 GHz) Impulse Radio Ultra Wide Band and Chirp ALOHA and CSMA-CA medium access control Link encryption (AES) (no key management) Full / reduced function devices Ranging support (!) Nanotron NanoLoc Chirp technology in the 2.4 GHz ISM band Enables real time location systems

16 Radio Power Consumption ChipCon CC 2420 / Nanotron Nanoloc CC 2420 NanoLoc standard a data rate 250 kbps 125 kbps / 1 Mbps voltage V V receive 19.7mA 33mA transmit 17.4mA 25 30mA idle?? sleep 20µA 2µA

17 Research Topics 1 Definition and Applications 2 Wireless Sensor Network Motes 3 Research Topics 4 Motes on the Internet (6lowpan)

18 Research Topics Embedded systems and ad-hoc networking Energy-aware resource management Cross-layer design and optimization (Ad-hoc) mesh routing protocols Interworking and distributed algorithms Middleware for wireless sensor networks Localization, time synchronization,... Data fusion, control, actuation,... Security and novel applications

19 Motes on the Internet (6lowpan) 1 Definition and Applications 2 Wireless Sensor Network Motes 3 Research Topics 4 Motes on the Internet (6lowpan)

20 Motes on the Internet IPv6 over (6lowpan) Talk IPv6 directly to motes Supports header compression Requires link-layer fragmentation / reassembly Should support mesh networking Jacobs 6lowpan Implementation Based on the TinyOS 2.0 embedded OS Buffer management is major design issue Serial tunnel daemon / base station support Tested on Telos-B and Mica-Z motes

21 Motes on the Internet IPv6 over (6lowpan) Talk IPv6 directly to motes Supports header compression Requires link-layer fragmentation / reassembly Should support mesh networking Jacobs 6lowpan Implementation Based on the TinyOS 2.0 embedded OS Buffer management is major design issue Serial tunnel daemon / base station support Tested on Telos-B and Mica-Z motes

22 Experimental Setup serial_tunnel daemon Linux kernel / IP stack TinyOS mote running BaseStationCC2420 ethernet interface tun(4) interface serial interface USB serial interface interface 6lowpan-encapsulated IPv6 packets IPv6 packets 6lowpan-encapsulated IPv6 packets Internet IPv6 packets TinyOS mote with a IPv6/6lowpan stack interface

23 Try it yourself! ping tests $ ping6 mote-14.eecs.jacobs-university.de $ ping6 mote-18.eecs.jacobs-university.de command line interface $ nc6 -u mote-14.eecs.jacobs-university.de 1234 help visual control (video streaming) $ vlc

24 References K. Römer and F. Mattern. The Design Space of Wireless Sensor Networks. IEEE Wireless Communications, 11(6):54 61, December A. Wheeler. Commercial Applications of Wireless Sensor Networks using ZigBee. IEEE Communications Magazine, 45(4):70 77, April L.D. Nardis and M.-G. Di Benedetto. Overview of the IEEE /4a standards for low data rate Wireless Personal Data Networks. In Proc. of the 4th IEEE Workshop on Positioning, Navigation and Communication 2007 (WPNC 07), Hannover, March IEEE. N. Kushalnagar, G. Montenegro, and C. Schumacher. IPv6 over Low-Power Wireless Personal Area Networks (6LoWPANs): Overview, Assumptions, Problem Statement, and Goals. RFC 4919, Intel Corp, Microsoft Corporation, Danfoss A/S, August G. Montenegro, N. Kushalnagar, J. Hui, and D. Culler. Transmission of IPv6 Packets over IEEE Networks. RFC 4944, Microsoft Corporation, Intel Corp, Arch Rock Corp, September M. Harvan. Connecting Wireless Sensor Networks to the Internet a 6lowpan Implementation for TinyOS 2.0. Master s thesis, Jacobs University Bremen, June 2007.

25 Part: NesC and TinyOS 5 History 6 NesC Language Overview 7 TinyOS: Operating System for WSNs 8 Demonstration

26 History 5 History 6 NesC Language Overview 7 TinyOS: Operating System for WSNs 8 Demonstration

27 NesC and TinyOS History developed by a consortium led by UC Berkeley two versions TinyOS 1.1 TinyOS not backwards compatible with 1.1

28 NesC Language Overview 5 History 6 NesC Language Overview 7 TinyOS: Operating System for WSNs 8 Demonstration

29 NesC: Programming Language for Embedded Systems Programming language: a dialect/extension of C static memory allocation only (no malloc/free) whole-program analysis, efficient optimization race condition detection Implementation: pre-processor output is a C-program, that is compiled using gcc for the specific platform statically linking functions For more details, see [?]

30 NesC Interfaces commands can be called by other modules think functions events signalled by other modules have to be handled by this module Interface Example interface Send { command error_t send(message_t* msg, uint8_t len); event void senddone(message_t* msg, error_t error);... }

31 NesC Components a NesC application consists of components components provide and use interfaces components can be accessed only via interfaces (cannot call an arbitrary C-function from another module) Figure: NesC Interface

32 NesC Components modules implement interfaces configurations connect modules together via their interfaces (wiring) Figure: NesC Configuration

33 NesC Concurrency Tasks Define a Task task void task_name() {... } Post a Task post task_name(); posting a task the task is placed on an internal task queue which is processed in FIFO order a task runs to completion before the next task is run, i.e. tasks do not preempt each other tasks can be preempted by hardware events

34 TinyOS: Operating System for WSNs 5 History 6 NesC Language Overview 7 TinyOS: Operating System for WSNs 8 Demonstration

35 TinyOS written in nesc event-driven architecture no kernel/user space differentiation single shared stack no process or memory management no virtual memory multi-layer abstractions components statically linked together

36 TinyOS Functionality hardware abstraction access to sensors access to actuators scheduler (tasks, hardware interrupts) timer radio interface Active Messages (networking) storage (using flash memory on the motes)...

37 Demonstration 5 History 6 NesC Language Overview 7 TinyOS: Operating System for WSNs 8 Demonstration

38 TinyOS Demo Blink no screen on which we could print Hello World let s blink an led instead using a timer to blink an led 2 source files BlinkC.nc BlinkAppC.nc

39 BlinkC.nc module BlinkC { uses interface Timer<TMilli> as Timer0; uses interface Leds; uses interface Boot; } implementation { event void Boot.booted() { call Timer0.startPeriodic(250); } event void Timer0.fired() { call Leds.led0Toggle(); } }

40 BlinkAppC.nc configuration BlinkAppC { } implementation { components MainC, BlinkC, LedsC; components new TimerMilliC() as Timer0; } BlinkC -> MainC.Boot; BlinkC.Timer0 -> Timer0; BlinkC.Leds -> LedsC;

41 TinyOS Demo Blink in reality using 3 timers 250 ms 500 ms 1000 ms each timer toggling one led the result is a 3-bit counter

42 TinyOS Demo Oscilloscope motes periodically get a sensor reading and broadcast over the radio BaseStation mote forward packets between the radio and the serial interface PC - java application reading packets from the serial interface and plotting sensor readings Figure: Oscilloscope Setup

43 TinyOS Demo Oscilloscope node ID sensor 7 light 8 light 18 temperature temperature = data

44 References D. Gay, P. Levis, R. von Behren, M. Welsh, E. Brewer, and D. Culler. The nesc Language: A Holistic Approach to Networked Embedded Systems. In PLDI03. ACM, June K. Römer and F. Mattern. The Design Space of Wireless Sensor Networks. IEEE Wireless Communications, 11(6):54 61, December A. Wheeler. Commercial Applications of Wireless Sensor Networks using ZigBee. IEEE Communications Magazine, 45(4):70 77, April L. D. Nardis and M.-G. Di Benedetto. Overview of the IEEE /4a standards for low data rate Wireless Personal Data Networks. In Proc. of the 4th IEEE Workshop on Positioning, Navigation and Communication 2007 (WPNC 07), Hannover, March IEEE.

45 Part: Distributed Algorithms 9 Transition System 10 Local and Distributed Algorithms 11 Induced Transition Systems 12 Events and Causal Order 13 Computations and Executions 14 Logical Clocks

46 Transition System 9 Transition System 10 Local and Distributed Algorithms 11 Induced Transition Systems 12 Events and Causal Order 13 Computations and Executions 14 Logical Clocks

47 Transition System Definition (transition system) A transition system is a triple S = (C,, I) where C is a set of configurations, is a binary transition relation on C, and I is a subset of C of initial configurations. Definition (execution) Let S = (C,, I) be a transition system. An execution of S is a maximal sequence E = (γ 0, γ 1,...), where γ 0 I and γ i γ i+1 for all i 0. Notes A transition relation is a subset of C C. The notation γ δ is used for (γ, δ),

48 Transition System Definition (reachability) Configuration δ is reachable from γ, notation γ δ, if there exists a sequence γ = γ 0, γ 1,..., γ k = δ with γ i γ i+1 for all 0 i < k. Notes A terminal configuration is a configuration γ for which there is no δ such that γ δ A sequence E(γ 0, γ 1,...) with γ i γ i+1 for all i is maximal if it is either infinite or ends in a terminal configuration Configuration δ is said to be reachable if it is reachable from an initial configuration.

49 Local and Distributed Algorithms 9 Transition System 10 Local and Distributed Algorithms 11 Induced Transition Systems 12 Events and Causal Order 13 Computations and Executions 14 Logical Clocks

50 Local Algorithm Definition (local algorithm) The local algorithm of a process is a quintuple (Z, I, i, s, r ), where Z is a set of states, I is a subset of Z of initial states, i is a relation on Z Z, and s and r are relations on Z M Z. The binary relation on Z is defined by c d (c, d) i m M((c, m, d) ( s r )). Notes Let M be a set of possible messages. We denote the collection of multisets with elements from M with M(M). The relations i, s, and r correspond to state transitions related with internal, send, and receive events.

51 Distributed Algorithm Definition (distributed algorithm) A distributed algorithm for a collection P = {p 1,..., p N } of processes is a collection of local algorithms, one for each process in P. Notes A configuration of a transition system consists of the state of each process and the collection of messages in transit The transitions are the events of the processes, which do not only affect the state of the process, but can also affect (and be affected by) the collection of messages The initial configurations are the configurations where each process is in an initial state and the message collection is empty

52 Induced Transition Systems 9 Transition System 10 Local and Distributed Algorithms 11 Induced Transition Systems 12 Events and Causal Order 13 Computations and Executions 14 Logical Clocks

53 Induced Async. Transition System Definition (Induced Async. Transition System) The transition system S = (C,, I) is induced under asynchronous communication by a distributed algorithm for processes p 1,..., p N, where the local algorithm for process p i is (Z pi, I pi, i p i, s p i, r p i ), is given by (1) C = {(c p1,..., c pn, M) : ( p P : c p Z p ) M M(M)} (2) (see next slide) (3) I = {(c p1,..., c pn, M) : ( p P : c p I p ) M = }

54 Induced Async. Transition System Definition (Induced Async. Transition System (cont.)) (2) = ( p P p), where the p are the transitions corresponding to the state changes of process p; pi the set of pairs is (c p1,..., c pi,..., c pn, M 1 ), (c p1,..., c p i,..., c pn, M 2 ) for which one of the following three conditions holds: (c pi, c p i ) i p i and M 1 = M 2 for some m M, (c pi, m, c p i ) s p i for some m M, (c pi, m, c p i ) r p i and M 2 = M 1 {m} and M 1 = M 2 {m}

55 Induced Sync. Transition System Definition (Induced Sync. Transition System) The transition system S = (C,, I) is induced under synchronous communication by a distributed algorithm for processes p 1,..., p N, where the local algorithm for process p i is (Z pi, I pi, i p i, s p i, r p i ), is given by (1) C = {(c p1,..., c pn ) : ( p P : c p Z p )} (2) (see next slide) (3) I = {(c p1,..., c pn ) : ( p P : c p I p )}

56 Induced Sync. Transition System Definition (Induced Sync. Transition System (cont.)) (2) = ( p P p) ( p,q P:p q pq), where pi is the set of pairs (c p1,..., c pi,..., c pn ), (c p1,..., c p i,..., c pn ) for which (c pi, c p i ) i p i pi p j is the set of pairs (..., c pi,..., c pj,...)(..., c p i,..., c p j,...) for which there is a message m M such that (c pi, m, c p i ) s p i and (c pj, m, c p j ) r p j

57 Events and Causal Order 9 Transition System 10 Local and Distributed Algorithms 11 Induced Transition Systems 12 Events and Causal Order 13 Computations and Executions 14 Logical Clocks

58 Events and Causal Order A transition a is said to occur earlier than transition b if a occures in the sequence of transitions before b An execution E = (γ 0, γ 1,...) can be associated with a sequence of events Ē = (e 0, e 1,...), where e i is the event by which the configuration changes from γ i to γ i+1 Events of a distributed execution can sometimes be interchanged without affecting the later configurations of the execution The notion of time as a total order on the events is not suitable and instead the notion of causal dependence is introduced

59 Dependence of Events Theorem Let γ be a configuration of a distributed system (with asynchronous message passing) and let e p and e q be events of different processes p and q, both applicable in γ. Then e p is applicable in e q (γ), e q is applicable in e p (γ), and e p (e q (γ)) = e q (e p (γ)). Let e p and e q be two events that occur consecutively in an execution. The premise of the theorem applies to these events except in the following two cases: a) p = q or b) e p is a send event, and e q is the corresponding receive event The fact that a particular pair of events cannot be exchanged is expressed by saying that there is a causal relation between these two events

60 Causal Order Definition (causal order) Let E be an execution. The relation, called the causal order, on the events of the execution is the smallest relation that satisfies the following requirements: (1) If e and f are different events of the same process and e occurs before f, then e f. (2) If s is a send event and r the corresponding receive event, then s r. (3) is transitive. Write a b to denote (a b a = b) The relation is a partial order and there may be events a and b for which neither a b nor b a holds Such events are said to be concurrent, notation a b

61 Computations and Executions 9 Transition System 10 Local and Distributed Algorithms 11 Induced Transition Systems 12 Events and Causal Order 13 Computations and Executions 14 Logical Clocks

62 Computations The events of an execution can be reordered in any order consistent with the causal order, without affecting the result of the execution Such a reordering of the events gives rise to a different sequence of configurations, but this execution will be regarded as equivalent to the original execution Let E = (γ 0, γ 1,...) be an execution with an associated sequence of events Ē = (e 0, e 1,...), and assume f is a permutation of Ē The permutation (f 0, f 1,...) of the events of E is consistent with the causal order if f i f j implies i j, i.e., if no event is preceded in the sequence by an event it causally precedes

63 Equivalent Executions Theorem Let f = (f 0, f 1,...) be a permutation of the events of E that is consistent with the causal order of E. Then f defines a unique execution F starting in the initial configuration of E. F has as many events as E, and if E is finite, the last configuration of F is the same as the last configuration of E. If the conditions of this theorem apply, we say that E and F are equivalent executions, denoted as E F A global observer, who has access to the actual sequence of events, may distinguish between two equivalent executions The processes, however, cannot distinguish between two equivalent executions

64 Computation Definition (computation) A computation of a distributed algorithm is an equivalence class under of executions of the algorithm. It makes no sense to speak about the configurations of a computation, because different executions of the computation may not have the same configurations It does make sense to speak about the collection of events of a computation, because all executions if the computation consist of the same set of events The causal order of the events is defined for a computation

65 Logical Clocks 9 Transition System 10 Local and Distributed Algorithms 11 Induced Transition Systems 12 Events and Causal Order 13 Computations and Executions 14 Logical Clocks

66 Logical Clocks Definition (clock) A clock is a function Θ from the set of events Ē to an ordered set (X, <) such that for a, b Ē Definition (lamport clock) a b Θ(a) < Θ(b). A Lamport clock is a clock function Θ L which assigns to every event a the length k of the longest sequence (e 1,..., e k ) of events satisfying e 1 e 2... e k = a. Note A clock function Θ expresses causal order, but does not necessarily express concurrency

67 Lamport Clocks The value of Θ L can be computed as follows: Θ L (a) is 0 if a is the first event in a process If a is an internal event or send event, and a the previous event in the same process, then Θ L (a) = Θ L (a ) + 1 If a is a receive event, a the previous event in the same process, and b the send event corresponding to a, then Θ L (a) = max(θ L (a ), Θ L (b)) + 1 The per process clock value may be combined with a process identifier to obtain a globally unique value

68 Lamport Clock Example P1: e1,1 e1,2 e1,3 e1,4 e1,5 P2: 1 2 e2,1 e2,2 e2,3 7 P3: e3,1 e3,2 e3,3 e3,4 e3,5 e3,6

69 Vector Clocks Definition (vector clocks) A vector clock for a set of N processes is a clock function Θ V which is defined by Θ V (a) = (a 1,..., a N ), where a i is the number of events e in process p i for which e a. Vectors are naturally ordered by the vector order: (a 1,... a n ) V (b 1,... b n ) i (1 i b) : a i b i Vector clocks can express concurrency since concurrent events are labelled with incomparable clock values: a b Θ V (a) < Θ V (b) Vector clocks require more space in the messages, but element compression can reduce message overhead

70 Vector Clock Example P1: (1,0,0) (2,0,0) (3,2,3) (4,2,3) (5,2,3) e1,1 e1,2 e1,3 e1,4 e1,5 P2: (0,1,0) (0,2,0) e2,1 e2,2 e2,3 (4,3,3) P3: (0,0,1) (0,2,2) (0,2,3) (0,2,4) (0,2,5) (0,2,6) e3,1 e3,2 e3,3 e3,4 e3,5 e3,6

71 References L. Lamport. Time, Clocks, and the Ordering of Events in a Distributed System. Communications of the ACM, 21(7), July M. Raynal. About logical clocks in distributed systems. Operating Systems Review, 26(1):41 48, G. Tel. Introduction to Distributed Algorithms. Cambridge University Press, 2 edition, 2000.

72 Part: Clock Synchronization in WSNs 15 Motivation and Introduction 16 Time-Stamp Synchronization (TSS) 17 Reachback Firefly Algorithm (RFA) 18 Reference Broadcast Synchronization (RBS) 19 Flooding Time Synchronization Protocol (FTSP)

73 Motivation and Introduction 15 Motivation and Introduction 16 Time-Stamp Synchronization (TSS) 17 Reachback Firefly Algorithm (RFA) 18 Reference Broadcast Synchronization (RBS) 19 Flooding Time Synchronization Protocol (FTSP)

74 Why Clock Synchronization in WSNs? Motivation Time correlated sensor readings or time series data aggregation in the network Coordination of low duty cycles (power up hardware only when necessary) to increase overall network lifetime Transmission scheduling algorithms such as TDMA

75 Clock Synchronization in WSNs Approach #1 (external synchronization) All clocks are synchronized to a real time standard such as Coordinated Universal Time (UTC). Approach #2 (internal synchronization) Clocks are relatively synchronized to each other to provide ordering of events.

76 Special Constraints in WSNs Energy consumption: According to Hill et al., each transmitted bit consumes as much power as executing instructions. One goal is therefore to work with a minimum number of messages. Processing power: Sensor nodes usually use microcontrollers with limited computational power. Memory capacity: Sensor notes have limited amount of data and program memory, making it difficult to fit lengthy algorithms or to maintain larger data sets. Transmission range: The transmission range is a function of transmission power and since energy is a scarce resource, the transmission range of sensor nodes is typically short. Depending on the application scenario, there might also be issues with noise.

77 Sources of Synchronization Errors Send time: Time spend at the sending node from creating a message until it reaches the network interface. Access time: Time spend getting access to the channel for transmission. Depending on the MAC, the access time can vary widely. Propagation time: Travel time of a message after leaving the sender node until it is received by a receiving node. Typically very small (in the order of nano seconds in LANs). Receive time: Time needed to process an incoming message and to notify the receiving application.

78 Synchronization Metrics 1 Energy consumption 2 Synchronization precision 3 Scalability to support large networks with many nodes 4 Robustness against failures (self-organization) 5 Synchronization scope (local synchronization for some nodes versus global synchronization of all nodes) 6 Computational complexity 7 Memory requirements (both RAM and ROM) 8 Piggy backing of synchronization message

79 Criteria and Classification 1 Master/slave vs. peer-to-peer synchronization 2 Clock correction vs. untethered clocks 3 Internal vs. external synchronization 4 Probabilistic vs. deterministic synchronization bounds 5 Sender to receiver vs. receiver to receiver synchronization 6 Single-hop vs. multi-hop networks 7 Stationary vs. dynamic network topologies

80 Time-Stamp Synchronization (TSS) 15 Motivation and Introduction 16 Time-Stamp Synchronization (TSS) 17 Reachback Firefly Algorithm (RFA) 18 Reference Broadcast Synchronization (RBS) 19 Flooding Time Synchronization Protocol (FTSP)

81 Time-Stamp Synchronization (TSS) Focus on temporal relationships between events (x happened before y) Assumption that not all nodes can directly talk to each other...

82 Time Transformation Time Transformation The real-time time difference t can be transformed into computer clock time difference c as follows: (1 ρ) c t (1 + ρ) (1) The parameter ρ is the bound of the computer clock drift. An equivalent notation is the following: (1 ρ) t c (1 + ρ) t (2) c (1 + ρ) t c (1 ρ) (3)

83 Time Transformation Example In order to transform a time difference c from the local time of one node with upper bound ρ 1 to the local time of another node with upper bound ρ 2, t is first estimated by the real time interval [ ] c (1 + ρ 1 ), c (4) (1 ρ 1 ) which in turn is estimated by the computer time interval [ c (1 ρ 2) (1 + ρ 1 ), c (1 + ρ ] 2) (1 ρ 1 ) relative to the local time of the second node. (5)

84

85 Reachback Firefly Algorithm (RFA) 15 Motivation and Introduction 16 Time-Stamp Synchronization (TSS) 17 Reachback Firefly Algorithm (RFA) 18 Reference Broadcast Synchronization (RBS) 19 Flooding Time Synchronization Protocol (FTSP)

86 Reference Broadcast Synchronization (RBS) 15 Motivation and Introduction 16 Time-Stamp Synchronization (TSS) 17 Reachback Firefly Algorithm (RFA) 18 Reference Broadcast Synchronization (RBS) 19 Flooding Time Synchronization Protocol (FTSP)

87 Reference Broadcast Synchronization NIC NIC Sender Sender Receiver Receiver 1 Time Critical Path Receiver 2 Critical Path Figure 1: A critical path analysis for traditional time synchronization protocols (left) and RBS (right). For traditional protocols working on a LAN, the largest contributions to nondeterministic latency are the Send Time (from the sender s clockexploit read to delivery theofbroadcast packet to its NIC, capability including protocol of wireless processing) sensor and Access Time (the delay in the NIC until the channel becomes free). The Receive Time tends to be much smaller than the Send Time because the clock can be networks read at interrupt time, before protocol processing. In RBS, the critical path length is shortened to include only the time from the injection of the packet into the channel to the last clock read. Number of Trials Idea Reduce the critical path by eliminating send time and access time notification. Instead of synchronizing the sender with the receiver, synchronize a set of receivers with each other The phase offset distribution appears Gaussian; the chisquared test indicates 99.8% confidence in the best fit parameters µ = 0,σ = 11.1µsec. This is useful for several reasons. As we will see in later sections, a well-behaved distribution enables us to significantly improve on the error bound statistically, by sending multiple Advanced broadcast Distributed packets. Systems In addition, Gaussian receivers

88 Reference Broadcast Synchronization Algorithm Individual nodes broadcast reference beacons to their neighbors (beacons do not include timestamps). Receivers timestamp the arrival of the beacons using their local clocks Receivers exchange their timestamps. The only source of errors are different propagation delays (but usually very small) and receive time errors.

89 Reference Broadcast Synchronization Phase Offsets Errors mainly due to receive time variations since propagation time variation is usually negligible Calculate the offset over a number of received beacons Let m be the number of beacons and let T r,b denote the time of receiver r s clock when it receives broadcast b. Then the offset o[i, j] of the notes i and j is given by: o[i, j] = 1 m m (T j,k T i,k ) (6) k=1 By averaging the time differences, we take care of phase errors.

90 Reference Broadcast Synchronization Clock Skew Clocks typically have a bounded frequency error, i.e., the freqency is not precisely the expected frequency. The frequency of real-world clocks can change but it tends to stay in table over time periods. Short-term instability is primarily due to environmental factors, such as variations in temperature, supply voltage, and shock. Long-term instability results from more subtle effects, such as oscillator aging. Use linear regression to handle clock screw.

91 Reference Broadcast Synchronization Experiment Broacast a reference beacon in a network with 5 motes Every mote has a clock resolution of 2µs to timestamp the reception times of incoming broadcasts The following plot shows points (x, y) with and the best linear fit. (x, y) = (T r1,k, T r2,k T r1,k) (7) The vertical pulses visualize the distance from each point to the best-fit line.

92 Reference Broacast Synchronization Linear fit seems to be a good enough approximation for clock screw.

93 Reference Broacast Synchronization Comparision of RBS and NTP Implemented RBS on a Linux kernel in user space Each regression is based on a window of the 30 most recent pulse reception reports Outliers are automatically rejected based on an adaptive threshold equal to 3 times the median fit error of the set of points not yet rejected Light network load: The network had very little background traffic Heavy network load: Two additional machines generated traffic by sending a series of randomly-sized UDP datagrams, each picked uniformly between 500 and 15,000 bytes (IP fragmentation being required for larger packets). The inter-packet delay was 10msec.

94 RBS Comparison to NTP RBS outperforms NTP and also a modified version of NTP, which allows a fairer comparison to RBS.

95 Flooding Time Synchronization Protocol (FTSP) 15 Motivation and Introduction 16 Time-Stamp Synchronization (TSS) 17 Reachback Firefly Algorithm (RFA) 18 Reference Broadcast Synchronization (RBS) 19 Flooding Time Synchronization Protocol (FTSP)

96 Flooding Time Synchronization Protocol Goals Achieve high precision clock synchronization Support sparse multi-hop network topologies Must run with limited resources (Mica2 / TinyOS) Approach Advanced MAC layer time stamping Linear regression of clock skew Flooding protocol with a dynamically elected root Graceful handling of election / join phases

97 Flooding Time Synchronization Protocol Clock Skew Time synchronization error between two motes. After 30 minutes, time synchronization is stopped and the initially small error results in increasing synchronization error over time.

98 Flooding Time Synchronization Protocol Linear Regression Distribution of error of linear regression on Mica2 motes: T = 30s time sync interval results in an average absolute error of 1.48µs and a maximum absolute error of 6.48µs T = 300s time sync interval results in an average absolute error of 2.24µs and a maximum absolute error of 8.64µs

99 Flooding Time Synchronization Protocol Protocol Idea Every synchronized node periodically broadcasts a synchronization message A node is synchronized if it has collected sufficient reference points Only synchronization messages from an elected synchronization root node are taken into account Message Format The synchronization message contains timestamp synchronization timestamp rootid ID of the synchronization root seqnum sequence number (assigned by sync. root)

100 Flooding Time Synchronization Protocol Protocol Details Nodes must ignore messages which are from an invalid synchronization root or have an old sequence number Nodes must collect sufficient reference points before they are allowed to broadcast Protocol must keep stability when a sync. root fails and a new one is elected with a different clock drift Need to handle partitions and (massive) joins gracefully

101 Flooding Time Synchronization Protocol event Radio.receive(TimeSync *msg) { if (msg->rootid < myrootid) myrootid = msg->rootid; else if (msg->rootid > myid msg->seqnum <= highestseqnum) return; highestseqnum = msg->seqnum; if (myrootid < myid) heartbeats = 0; if (numentries >= NUMENTRIES_LIMIT && geterror(msg) > TIME_ERROR_LIMIT) clearregressiontable(); else addentryandestimatedrift(msg); }

102 Flooding Time Synchronization Protocol event Timer.fired() { ++heartbeats; if (myrootid!= myid) && heartbeats >= ROOT_TIMEOUT) myrootid = myid; if (numentries >= NUMENTRIES_LIMIT myrootid == myid) { msg.rootid = myrootid; msg.seqnum = highestseqnum; Radio.send(msg); } } if (myrootid == myid) ++highestseqnum;

103 Flooding Time Synchronization Protocol Experiment Evaluation on a 5 12 grid of Mica / Mica2 nodes Nodes only communicate with their direct neighbors Topology enforced by software

104 Flooding Time Synchronization Protocol Experiment Goal: measure average pairwise error, maximum pairwise error, percentage of synchonized motes Test plan: A: at 00:04 all motes were turned on B: at 01:00 the root ID 1 was switched off and ID 2 becomes root eventually C: between 02:00 and 02:15, randomly selected nodes were reset one by one (30 seconds period) D: at 02:30 all motes with odd node IDs were switched off E: at 03:01 all motes with odd node IDs were switched on F: at 04:02 experiment ended

105 Flooding Time Synchronization Protocol

106 References K. Römer. Time Synchronization in Ad Hoc Networks. In Proc. ACM Symposium on Mobile Ad Hoc Networking and Computing (MobiHoc 2001), pages , October K. Römer. Time Synchronization and Localization in Sensor Networks. PhD thesis, ETH Zurich, J. Elson, L. Girod, and D. Estrin. Fine-Grained Network Time Synchronization using Reference Broadcasts. In Proc. Fifth Symposium on Operating Systems Design and Implementation (OSDI 2002), pages , December J. E. Elson. Time Synchronization in Wireless Sensor Networks. PhD thesis, University of California Los Angeles, M. Maróti, B. Kusy, G. Simon, and A. Lédeczi. The Flooding Time Synchronization Protocol. In Proc. of the ACM Conference on Embedded Networked Sensor Systems (SenSys 04), November G. Werner-Allen, G. Tewari, A. Patel, M. Welsh, and R. Nagpal. Firefly-Inspired Sensor Network Synchronicity with Realistic Radio Effects. In Proc. of the ACM Conference on Embedded Networked Sensor Systems (SenSys 05), November 2005.

107 Part: Wave and Traversal Algorithms 20 Wave Algorithms and their Properties 21 Equivalence with Related Problems Propagation of Information with Feedback Synchronization Infimum Functions 22 Wave Algorithms Ring Algorithm Tree Algorithm Echo Algorithm Polling Algorithm 23 Traversal Algorithms

108 Wave Algorithms and their Properties 20 Wave Algorithms and their Properties 21 Equivalence with Related Problems Propagation of Information with Feedback Synchronization Infimum Functions 22 Wave Algorithms Ring Algorithm Tree Algorithm Echo Algorithm Polling Algorithm 23 Traversal Algorithms

109 Motivation Message passing schemes which involve all processes are common subroutines in more specific algorithms. So called wave algorithms exchange a finite number of messages and then they make a decision, which depends causally on some event in each process. Considering wave algorithms first and in isolation will make it easier to understand more detailed algorithms. Certain problems in distributed computations can be solved by generic constructions that yield a specific algorithm.

110 Preliminaries We consider distributed systems where the network topology is fixed, the channels are bidrectional, the topology is free of partitions, and the system is asynchronous and there is no notion of global time. The set of all processes is denoted as P, and the set of channels by E. The number of events of computation C is denoted C and the subset of events that occur in process p is denoted C p. There is a special type of internal event called decide event (represented by the statement decide).

111 Wave Algorithm Definition A wave algorithm is a distributed algorithm that satisfies the following three requirements. (1) Each computation is finite: C : C < (2) Each computation contains at least one decide event: C : e C : e is a decide event (3) In each computation each decide event is causally preceded by an event in each process: C : e C : (e is decide event q P : f C q : f e)

112 Terminology The computation of a wave algorithm is called a wave. A process is called an initiator if it starts the execution of its local algorithm spontaneously. A non-initiator becomes involved only when a message of the algorithm arrives and triggers the execution of the local algorithm. As a consequence, the first event of an initiator is an internal or send event while the first event of a non-initiator is a receive event. An algorithm is called centralized if there must be exactly one initiator in each computation, and decentralized if the algorithm can be started spontaneously by an arbitrary subset of the processes.

113 Wave Properties Lemma For each event e C there exists an initiator p and an event f in C p such that f e. Lemma Let C be a wave with one initiator p and, foreach non-initiator q, let father q be the neighbor of q from which q received a message in its first event. Then the graph T = (P, E T ), with E T = {qr : q p r = father q } is a spanning tree directed towards p. Lemma Let C be a wave and d p C a decide event in process p. Then q p : f C q : (f d p f is a send event)

114 Wave Properties (cont.) Theorem Let C be a wave with one initiator p, such that a decide event d p occurs in p. Then at least N messages are exchanged in C. Theorem Let A be a wave algorithm for arbitrary networks without initial knowledge of the neighbors identities. Then A exchanges at least E messages in each computation.

115 Equivalence with Related Problems 20 Wave Algorithms and their Properties 21 Equivalence with Related Problems Propagation of Information with Feedback Synchronization Infimum Functions 22 Wave Algorithms Ring Algorithm Tree Algorithm Echo Algorithm Polling Algorithm 23 Traversal Algorithms

116 Propagation of Information with Feedback Propagation of Information with Feedback (PIF) A subset of processes is formed by those that have a message M (all of these have the same message), which must be broadcasted, i.e., all processes must receive and accept M. Certain processes must be notified of termination of the broadcast; that is, they must execute a special notify event, with the requirement that a notify event may be executed only when all processes have already received the message M. Notification by the PIF algorithm is considered as a decide event.

117 PIF and Wave Algorithms Theorem Every PIF algorithm is a wave algorithm. Theorem Every wave algorithm can be employed as a PIF algorithm. The constructed PIF algorithm has the same message complexity as A but has a different bit complexity.

118 Synchronization Synchronization Problem In each process q an event a q must be executed, and in some processes an event b p must be executed, such that the execution of all a q events must have taken place temporally before any of the b p events is executed. In a synchronization algorithm (SYN) the b p events will be considered as decide events.

119 Synchronization and Wave Algorithms Theorem Every SYN algorithm is a wave algorithm. Theorem Every wave algorithm can be employed as a SYN algorithm.

120 Infimum Functions Definition If (X, <) is a partial order, then c is called the infimum of a and b if c a, c b, and d : (d a d b d c). Let a b denote the infimum of a and b. The binary operator is cummutative and associative and can be generalized to finite sets: inf {j 1,..., j k } = j 1... j k Infimum Problem Each process q holds and input j q, which is an element of a partially ordered set X. It is required that certain processes compute the value of inf j q : q P and that these processes know when the computation is terminated.

121 Infimum Functions and Wave Algorithms Theorem Every INF algorithm is a wave algorithm. Theorem Every wave algorithm can be employed as a INF algorithm.

122 Infimum Theorem Theorem If is a binary operator on a set X such that (1) is commutative, i.e., a b = b a, (2) is associative, i.e., (a b) c = a (b c), and (3) is idempotent, i.e., a a = a then there is a partial order on X such that is the infimum function. Corollary,, min, max, gcd, lcm,, and cup of values local to the processes can be computed in a single wave.

123 Wave Algorithms 20 Wave Algorithms and their Properties 21 Equivalence with Related Problems Propagation of Information with Feedback Synchronization Infimum Functions 22 Wave Algorithms Ring Algorithm Tree Algorithm Echo Algorithm Polling Algorithm 23 Traversal Algorithms

124 Ring Algorithm

125 Tree Algorithm

126 Echo Algorithm

127 Polling Algorithm

128 Traversal Algorithms 20 Wave Algorithms and their Properties 21 Equivalence with Related Problems Propagation of Information with Feedback Synchronization Infimum Functions 22 Wave Algorithms Ring Algorithm Tree Algorithm Echo Algorithm Polling Algorithm 23 Traversal Algorithms

129 Traversal Algorithm Definition A traversal algorithm is an algorithm with the following properties: (1) In each computation there is one initiator, which starts the algorithm by sending exactly one message. (2) A process, upon receipt of a message, either sends out one message or decides. (3) The algorithm terminates in the initiator and when this happens, each process has sent a message at least once.

130 Ring and Polling Algorithms Theorem The Ring Algorithm is a traversal algorithm. Theorem The sequential polling algorithm is a traversal algorithm.

131 Computing Sums Observation A large number of functions are not computable using general wave algorithms, e.g., the sum over all inputs, because the sum operator is not idempotent. Computing sums is possible in special cases, e.g., when the wave algorithm is a traversal algorithm, when the processes have identities, or when the algorithm induces a spanning tree.

132 Tarry s Traversal Algorithm A traversal algorithm for arbitrary connected networks was given by Tarry in The algorithm is formulated in the following two rules: R1 A process never forwards the token twice through the same channel. R2 A non-initiator forwards the token to its father (the neighbor from which it first received the token) only if there is no other channel possible according to rule R1. Theorem Tarry s algorithm is a traversal algorithm.

133 References G. Tel. Introduction to Distributed Algorithms. Cambridge University Press, 2 edition, 2000.

134 Part: