Data-driven Link Estimation in Low-power Wireless Networks: An Accuracy Perspective

Transcription

1 Data-driven Link Estimation in Low-power Wireless Networks: An Accuracy Perspective Hongwei Zhang, Lifeng Sang, Anish Arora Technical Report: WSU-CS-DNC-TR-8- Department of Computer Science, Wayne State University, USA, Department of Computer Science and Engineering, The Ohio State University, USA, {sangl, Abstract Link estimation is a basic element of routing in low-power wireless networks, and several approaches using broadcast beacons and/or unicast MAC feedback have been proposed in the past years. Nonetheless, there has been no systematic study on the inherent drawbacks of beacon-based link estimation and the subtleties of data-driven link estimation. Using a testbed of 98 XSM motes (an enhanced version of MICA motes), we characterize the inherent errors in predicting unicast properties via broadcast beacons, and we show that data-driven link estimation and routing achieves higher event reliability (e.g., by up to 8.7%) and energy efficiency (e.g., by up to a factor of.96) than beacon-based approaches. Through mathematical and experimental analysis, we examine the accuracies of different data-driven link estimation methods and their impacts on routing performance. We discover, counterintuitively, that two representative, seemingly similar methods of data-driven link estimation differ significantly in performance: in our testbed, for instance, data delivery reliability differs by.8%, and energy efficiency differs by a factor of 3.7. These findings provide new insight into the subtle design decisions in link estimation that significantly impact routing performance in low-power wireless networks. Keywords Low-power wireless networks, sensor networks, link estimation and routing, data-driven, beacon-based Introduction Wireless communication assumes complex spatial and temporal dynamics [,, 43, 44], thus estimating link properties is a basic element of routing in wireless networks. One commonly used approach of link estimation is letting neighbors exchange broadcast beacon packets, and then estimating link properties of unicast data transmissions via those of broadcast beacons. Nonetheless, there are significant differences between unicast and broadcast link properties [, 9], and it is difficult to precisely estimate unicast link properties via those of broadcast due to temporal correlations in link properties and dynamic, unpredictable network traffic patterns [38, 4]. The research community has proposed mechanisms to ameliorate the impact of the differences between broadcast and unicast link properties [, 9], and MAC feedback carrying information about unicast data transmissions has also been used in link estimation [7,, 3, 6, 7, 39, 4]. Nonetheless, there has been no systematic characterization of the inherent drawbacks in estimating unicast properties via those of broadcast; some protocols [7, 39] use MAC feedback mainly for saving energy in link estimation (e.g., by reducing the frequency of broadcast beacon exchanges), and some protocols [7, 3, 39] use both broadcast-based and MAC-feedbackbased link estimation. One open question here is, from the perspective of estimation accuracy, whether broadcast beacons should be used as the basis of link estimation in low-power wireless networks such as sensor networks. This is an important question because link estimation accuracy significantly affects routing performance as we discuss later in the paper. In the context of data-driven link estimation that uses MAC feedback for unicast data transmissions, MAC feedback of mote networks carries rich information (e.g., both the number of physical transmissions and the time taken for a unicast transmission), and different methods of using these information have been proposed [7,, 7]. But there has been no systematic study on the accuracies of these different datadriven link estimation methods, and thus it is still unclear what the guidelines should be for using the rich information in MAC feedback. Focusing on the accuracy of estimating unicast data transmission properties, our objectives in this paper are to characterize the limitations of beacon-based link estimation and to comparatively study different data-driven link estimation methods. Contributions of the paper. Using a testbed of 98 XSM motes, we characterize the significant, unpredictable errors in estimating unicast properties via broadcast beacons, and we examine the impact of interference patterns on link properties in low-power wireless sensor networks. We also demonstrate the complex, unpredictable nature of temporal correlations in link properties, which, together with uncertainties in interference patterns, motivates the approach of data-driven link estimation and routing. Through experimentation with a realistic event traffic trace, we also show that data-driven link estimation and routing greatly improves the event reliability (e.g., by 8.7%) and energy efficiency (e.g., by a factor of.96) of beacon-based approaches. These results provide solid empirical evidence on the drawbacks of beacon-based estimation and the benefits of data-driven link estimation and routing. Having justified the necessity of data-driven link estimation and routing, we classify existing methods of using unicast MAC feedback into two broad categories represented by two seemingly similar protocols: L-NT where the number of physical transmissions for each unicast is directly used to estimate the expected number of transmissions along a link, and L-ETX where we use the number of physical transmissions for each unicast to calculate the reliability of unicast physical transmissions which is then used to estimate the expected number of transmissions along a link. Through mathematical and experimental analysis, we show that L-ETX achieves higher estimation accuracy than L-NT, and thus L-ETX uses the optimal routes with a higher probability than L-NT does. Using a realistic event traffic trace, we show that L-ETX achieves higher event reliability (e.g., by.8%) and energy efficency (e.g.,

2 by a factor of 3.7) than L-NT, and that, compared with L-NT, L-ETX is able to find and use longer yet more reliable links. We also find out that link estimation is much more stable in L- ETX than in L-NT, which enables stable routing structures in L-ETX. Additionally, we show that L-ETX outperforms other variants of L-NT too. These findings elucidate important subtleties of data-driven link estimation and provide new insight into the question of how to use MAC feedback in designing routing protocols for low-power wireless networks. Organization of the paper. We examine the necessity of data-driven link estimation and routing in low-power wireless networks in Section. We compare different methods of using MAC feedback in Section 3, and we present the impact of link estimation accuracy on routing performance in Section 4. We discuss protocols similar to L-NT and L-ETX, and we present additional performance evaluation results in Section. Finally, we discuss related work in Section 6 and make concluding remarks in Section 7. Why data-driven link estimation In this section, we characterize, via the Kansei [6] sensor network testbed, the errors and inherent difficulties in predicting unicast data transmission properties via broadcast beacons in low-power wireless networks. We first present the experiment design and then the experimental results.. Experiment design In an open warehouse with flat aluminum walls (see Figure (a)), Kansei deploys 98 XSM motes [] in a 4 7 grid (see Figure (b)) where the separation between neighboring grid points is.9 meter (i.e., 3 feet). The grid deployment pat- transmissions have all failed; a broadcast packet is transmitted only once at the MAC layer (without retransmission even if the transmission has failed). For convenience, we call the individual transmissions involved in transmitting a unicast packet the unicast-physical-transmissions. To demonstrate the difficulty in precisely estimating unicast properties via those of broadcast beacons, we let the mote on the left end of the middle row (shown as black dots in Figure (b)) be the sender and the rest 3 motes of the middle row as the receivers, and we measure the unicast and broadcast properties of the links between the sender and individual receivers. (We have observed similar phenomena as what we will present in Section. for other sender-receiver pairs.) The sender transmits, broadcast and, unicast packets to each of the receivers with a 8-millisecond inter-packet interval, and each packet has a data payload of 3 bytes. Based on packet reception status (i.e., success or failure) at the receivers, we measure unicast and broadcast link properties. To examine the impact of traffic-induced interference on link properties and link estimation, we randomly select 4 motes out of the light-colored (of color cyan) 6 rows as interferers, with 7 interferers from each row on average. Each interferer transmits unicast packets (of payload length 3 bytes) to a destination randomly selected out of the other 4 interferers. The load of the interfering traffic is controlled by letting interferers transmit packets with a certain probability d whenever the channel becomes clear. In our experiments, we measure the broadcast and unicast reliability from the sender to its receivers when d is,.,.4,.7,.,.4,.7, and respectively. Thus the interference pattern is controlled by d in this case. (Phenomena similar to what we will present in Section. have been observed for other interfering traffic patterns, for instance, with different spatial distribution and different number of interferers.). Experimental results (a) Kansei (b) 4 7 grid Figure : Sensor network testbed Kansei tern enables experimentation with regular, grid topologies, as well as random topologies (e.g., by randomly selecting nodes of the grid to participate in experiments). XSM is an enhanced version of Mica [] mote, and each XSM is equipped with a Chipcon CC [] radio operating at 433 MHz. To form multihop networks, the transmission power of the CC radios is set at -4dBm (i.e., power level 3) for the experiments of this paper unless otherwise stated. XSM uses TinyOS [4] as its operating system. For all the experiments in this paper, the default TinyOS MAC protocol B-MAC [3] is used; a unicast packet is retransmitted, upon transmission failure, at the MAC layer (more specifically, the TinyOS component QueuedSend) for up to 7 times until the transmission succeeds or until the 8 Unlike in 8. networks where unicast and broadcast differ in a variety of ways such as MAC coordination method (i.e., whether data transmission is preceded by the RTS-CTS handshake) and transmission rate, the main difference between unicast and broadcast in mote networks (where TinyOS B-MAC or 8..4 MAC is usually used) lies in link layer error control. That is, upon transmission failure, a unicast packet is usually retransmitted at link layer to improve delivery reliability, whereas a broadcast packet is not retransmitted. This link layer retransmission affects the accuracy of estimating unicast link properties via those of broadcast beacons. The accuracy of beacon-based link estimation is also degenerated by trafficinduced interference. In what follows, we present our empirical results on the impact of link layer retransmission and traffic-induced interference on the accuracy of beacon-based link estimation. Impact of link layer retransmission. In beacon-based link estimation, unicast ETX (i.e., the expected number of physical transmissions taken to successfully deliver a unicast packet) along a link is estimated as P b, where P b is the broadcast relia-

3 bility along the link. Based on the measured data on broadcast reliability and unicast ETX in our experiments, Figure shows Error in beacon based est. (%) d = d =. d =.4 d =.7 d =. d =.4 d =.7 d = distance (meter) Figure : Errors in estimating unicast ETX via broadcast reliability. Note that, due to factors such as radio and environment variations, properties of the wireless links in a specific network setup are usually not monotonic functions of the senderreceiver distance [44]; accordingly, we will observe that the phenomena presented in this paper tend not to be monotonic with respect to distance. We have also observed that phenomena studied in this paper are complex and tend not to be correlated or monotonic with other parameters such as receiver-side SINR or link reliability. In this paper, we use distance mainly to identify individual links associated with a sender, and for clarity of presentation, we do not present confidence intervals unless they are necessary for certain claims of the paper. the errors in estimating unicast ETX via broadcast reliability, where the error is defined as the estimated unicast ETX minus the actual unicast ETX and then divided by the actual unicast ETX. We see that the estimation error tends to be large, e.g., up to 88.78%. The estimation error also changes with interference pattern, which makes it difficult to compensate for the estimation errors in practice since interference patterns may well be unknown and unpredictable. Given a certain interference pattern (more specifically, interfering traffic load in this case), one major cause for the significant errors in beacon-based link estimation is the temporal correlation among unicast-physical-transmissions, that is, the fact that the status (i.e., success or failure) of the individual physical transmissions of a unicast are correlated due to short-term, bursty variation in environment-induced fading and traffic-induced interference. As a result of this short-term temporal correlation, the probability of a physical transmission failure conditioned on an immediately preceding physical transmission failure is higher than the probability of a transmission failure at a random moment in time. Given that broadcast beacons are usually well separately in time when compared with the interval between consecutive physical (re)transmissions involved in a unicast, the reliability of unicast-physical-transmissions tends to be lower than that of broadcast beacons. This conjecture is corroborated by Figure 3 where we show the difference between the reliability of each unicast-physical-transmission and that of broadcast in different interference scenarios. As a result, beacon-based estimation tends to underestimate unicast ETX, which can also Here we only consider the ETX along one direction of a link. difference in delivery rate (%) d = d =. d =.4 d =.7 d =. d =.4 d =.7 d = distance (meter) Figure 3: The mean reliability of each unicast-physicaltransmission minus that of broadcast be observed from Figure. As we will discuss shortly, the temporal correlation among unicast-physical-transmissions is also affected by interference pattern, thus the error in beaconbased estimation changes with interference pattern as we see from Figure. To further understand the temporal correlation among unicast-physical-transmissions, we analyze the autocorrelation coefficient ρ among the individual physical transmissions of unicasts. Given the observations x, x,...,x n of a time series, the autocorrelation coefficient ρ(h) for lag h ( n < h < n) [7] is where ρ(h) = ˆγ(h) ˆγ() ˆγ(h) = n h n t= (x t+ h x)(x t x) x = n n t= x t For a link of length 9. meters (i.e., 3 feet), Figure 4(a) shows the autocorrelation coefficients for the status of unicastphysical-transmissions. We see that autocorrelation coefficient decreases as the lag h increases; this is because the correlation between individual unicast-physical-transmissions is due to short-term, bursty variation in environment-induced fading and traffic-induced interference. We also see that autocorrelation coefficient tends to decrease as interference load increases, partly due to the increased degree of randomization in traffic-induced interference as a result of increased load of random interfering traffic. For different links in different interference scenarios, Figure 4(b) shows the autocorrelation coefficients for lag 4. We see that autocorrelation coefficient also varies across links. (Similar phenomena are observed for other links and lag values.) The complex correlation patterns among unicast-physical-transmissions partly explain why it is difficult to address the difference between broadcast and unicast-physical-transmission and thus the significant errors in beacon-based estimation. Impact of traffic-induced interference. Figure shows the network conditions, in terms of unicast ETX, in different interference scenarios. We see that, as interference changes, unicast ETX changes significantly (e.g., up to 3.44), and unicast ETX tends to increase with increasing interference traffic load. In event-detection sensor networks [36], there is no data 3

4 ρ(h) h d = d =. d =.4 d =.7 d =. d =.4 d =.7 d = (a) Autocorrelation coefficient for a link of length 9. meters ρ(4).. d = d =. d =.4 d =.7 d =. d =.4 d =.7 d = Link length (meter) (b) Autocorrelation coefficient for lag 4 Figure 4: Autocorrelation coefficient for the status of unicastphysical-transmissions ETX d = d =. d =.4 d =.7 d =. d =.4 d =.7 d = distance (meter) Figure : Unicast ETX in different interference scenarios traffic in the network most of time, yet a huge burst of data packets are generated within a short period of time (e.g., a few seconds) once an event occurs. Therefore, network trafficinduced interference and thus unicast ETX tend to vary significantly depending on whether there is an active event. To reduce control overhead, broadcast beacons are usually exchanged at low frequencies (e.g., once every 3 seconds) in practice, thus the network conditions experienced and sampled by broadcast beacons may well reflect those in the absence rather than in the presence of bursty event traffic. Consequently, beacon-based estimation may well lead to significant sampling errors (i.e., broadcast beacons fail to sample the network conditions for data transmissions) in event-detection sensor networks. Summary. From the testbed based experimental analysis, we see that beacon-based link estimation introduces significant estimation errors in low-power wireless networks such as the XSM mote networks. Unlike in [4] where MAC feedback does not indicate the number of physical transmissions for a unicast, the MAC components of mote networks expose the number of physical transmissions for each unicast, and this enables us to derive the status of each unicast-physical transmission and thus the properties of unicast-physical-transmissions such as their reliability and correlation as shown in Figures 3 and 4. These new observations give insight into the complexity of temporal correlation among unicast-physical-transmissions and the impact of interference patterns, which represent inherent difficulties in beacon-based link estimation and thus motivate the necessity of estimating unicast properties via feedback information about unicast data transmissions themselves. 3 Methods of data-driven estimation In this section, we first identify the two representative methods (i.e., L-NT and L-ETX) of data-driven link estimation, then we comparatively study their estimation accuracy via mathematical and experimental analysis. 3. Different data-driven estimation methods MAC feedback for a unicast transmission usually contains aggregate information (e.g., MAC latency, number of physical transmissions, and transmission status) about transmitting the unicast packet as a whole, but from these information we can derive properties of the individual physical transmissions involved in unicasting. Accordingly, we can classify the methods of using MAC feedback depending on whether they directly use the aggregate information about unicast or they use information about the individual unicast-physicaltransmissions. In the literature, SPEED [], LOF [4], and CARP [6] use the aggregate information MAC-latency, yet protocols such as four-bit-estimation [7], EAR [3], NADV [7], and MintRoute[39] use the reliability information about individual unicast-physical-transmissions. In this paper, we mainly focus on the following two methods of using MAC feedback: L-NT: directly use feedback information on the number of physical transmissions (NT) to estimate the expected number of transmissions (ETX) required to successfully deliver a packet across a link; L-ETX: first use feedback information to estimate the reliability PDR of individual unicast-physicaltransmissions, then estimate ETX as PDR. The rationale for considering the above two methods are as follows: ) ETX is a commonly used metric in wireless network routing; ) the parameter NT is tightly related to MAC latency (e.g., MAC latency is approximately proportional to NT given a certain degree of channel contention) such that L- NT also represents protocols that directly use MAC latency in routing; (we will examine routing methods that use MAC latency in Section, and will show that they behave similar to In this paper, we use ETX to denote the property of a link. We also use ETX to denote a specific beacon-based link estimation and routing protocol. The context of its usage will clarify whether we mean the metric or the protocol. 4

5 L-NT.) 3) L-NT and L-ETX estimate the same link property ETX, which enables fair comparison between different datadriven estimation methods; and 4) ETX is also used in existing beacon-based routing protocols [], which enables us to compare data-driven link estimation with beacon-based estimation (as we do in Section 4). Based on this research design, L-NT represents the method used in SPEED, LOF, and CARP, and L-ETX represents the method used in four-bit-estimation, EAR, NADV, and MintRoute. 3. Analysis of L-NT and L-ETX In this section, we focus on the accuracy of estimating ETX for a wireless link via the commonly-used exponentiallyweighted-moving-average (EWMA) estimator. Given a time series {x i : i =,,...} where x i is a random variable with mean µ and variance σ, the corresponding EWMA estimator for µ is y = x y k = αy k + ( α)x k, α, k =, 3,... By induction, we have y k = α k x + ( α) k α k i x i, k In L-NT and L-ETX, the EWMA estimator tries to estimate ETX t and PDR t respectively, where ETX t is the ETX for the link and PDR t is the expected reliability of unicastphysical-transmissions along the link. In L-NT, the time series input {x i } to the estimator is {NT i : i =,,...}, where NT i is the number of physical transmissions taken to deliver the i-th unicast packet; in L-ETX, the time series input {x i } is {PDR i : i =,,...}, where PDR i is the packet delivery rate of the i -th window of unicast-physicaltransmissions with window size W (i.e., the average delivery rate of the ((i ) W + )-th, ((i ) W + )-th,..., and the ((i ) W + W)-th unicast-physical-transmission). Note that, for the i-th unicast packet, NT i is calculated based on MAC feedback on the number NT i of physical transmissions incurred and the status of the unicast transmission showing whether the packet has been successfully delivered or not. If the status shows success, then NT i = NT i ; otherwise, the feedback simply shows that the packet has not been delivered after transmitting NT i times, and we penalize NT i by setting i= it as NT i P ( < P u.i u.i < ), where P u.i is the average unicast reliability calculated based on the status information on transmitting the i unicast packets so far. In what follows, We first analyze the accuracy of EWMA estimator in general, then we compare the accuracy of L-NT and L-ETX. Accuracy of EWMA estimators. To measure the estimation error in the estimator ˆµ = y k, we define squared error (SE) as SE k = (y k µ) where CP k = ((α k x + ( α) k i= αk i x i ) µ) = (α k (x µ) + ( α) k i= αk i (x i µ)) = α k (x µ) + ( α) k i= α(k i) (x i µ) + CP k () = ( α) k i= α i (x µ)(x i µ)+ ( α) k i= k j i,j= αk i j (x i µ)(x j µ) Therefore, the mean squared error (MSE) is MSE k = E[SE k ] = α k E[(x µ) ]+ ( α) k i= α(k i) E[(x i µ) ]+ E[CP k ] Note that the expectation E[] is taken over x, x,..., x k. To get a closed form solution to MSE k and considering the fact the x i s in L-NT and L-ETX are well separated in time when compared consecutive unicast-physical-transmissions 3, we assume that that x i and x j are uncorrelated (i.e., E[x i x j ] = E[x i ]E[x j ]) if i j. (Despite this simplifying assumption, the insight we get about the relative estimation accuracy of L-NT and L-ETX can be corroborated by testbed-based experimental analysis as we show in Sections 3.3 and 4..) Then for i j, E[(x i µ)(x j µ)] = E[x i x j x i µ µx j + µ ] = E[x i ]E[x j ] E[x i ]µ µe[x j ] + µ = Thus, E[CP k ] = ( α) k i= α i E[(x µ)(x i µ)]+ ( α) k k i= j i,j= αk i j E[(x i µ)(x j µ)] = (3) From Equations and 3, we have MSE k = α k σ + ( α) k i= α(k i) σ = σ αk+ α+ +α To measure the degree of estimation error (DE) using estimator ˆµ = y k, we define DE k as MSE k µ. Thus DE k = MSE k µ = σ µ α k+ α+ +α = COV [X] α k+ α+ +α where COV [X] is the coefficient-of-variation (COV) of the x s, i.e., COV [X] = σ µ. Therefore, we have 3 From Figure 4(a), we see that temporal correlation tends to be high only for samples from a close, short time frame. () (4) ()

6 Proposition Assuming that x i and x j (i j) are uncorrelated, the degree of estimation error using EWMA estimator is proportional to the COV of the x s. Relative accuracy of L-NT and L-ETX. To compare the DEs of L-NT and L-ETX, therefore, we first analyze the COV of NT i and PDR i. To this end, we assume that each unicastphysical-transmission is a Bernoulli trial with a failure probability P ( P < ). (Despite this simplifying assumption, the insight we get about the relative estimation accuracy of L-NT and L-ETX can be corroborated by testbed-based experimental analysis as we show in Sections 3.3 and 4..) Then, we can derive the COV of NT i and PDR i as follows: L-NT: By definition, NT i can be modeled as following a geometric distribution with the probability of success P. Thus E[NT i ] = P, and std[nt i ] = ( P) P = P P. Therefore, COV [NT i ] = std[nt i] E[NT i ] = P ( P ) (6) L-ETX: Given a window size W >, the number N of successes in W physical transmissions can be modeled as following a Binomial distribution with the probability of success P and the number of trials W. Thus, E[N] = W( P ), and var[n] = W( P )P. Let PDR i = N W, then E[PDR i ] = W E[N] = P, var[pdr i ] = W var[n] = ( P)P W, and std[pdr i ] = var[pdr i ] = COV [PDR i ] = std[pdr i ] E[PDR i ] ( P)P W. Therefore, = P (7) W P Given that P <, we know from Equations 6 and 7 that COV [NT i ] > COV [PDR i ] (8) And this inequality holds even when W =. From Proposition and Inequality (8), we have Proposition Given an EWMA estimator, DE k (L-ETX) < DE k (L-NT); that is, L-ETX is more accurate than L-NT in estimating the ETX value of a link. Proof sketch: We first show that DE(L-ETX) DE(PDR) as follows. Let P t be the actual E[PDR], ETX t be the actual ETX and equal to P t, and P e be the estimated E[PDR]. Then the absolute estimation error ETX of ETX is as follows: ETX = P e P t = P t P t P e P e = ETX t P t P e P e Thus DE(L-ETX) = E[ ETX] = E[ P t P e ] DE[PDR]. ETX t In the mean time, we know from Proposition and Inequality 8 that DE k (L-NT) > DE k [PDR]. Therefore, DE k (L-NT) > DE k (L-ETX). P e From the above analysis, we see that, even though L-NT and L-ETX use the same MAC feedback information in a seemingly similar fashion (e.g., PDR i s assume, approximately, a form of the reciprocal of NT i s), the variability (more precisely, the COV) of NT i s is larger than that of PDR i s, and this difference in variability makes L-NT a less accurate estimator than L-ETX. We corroborate these analytical observations through experimental analysis in Sections 3.3 and 4.. Note that, even though they do not affect the relative accuracy of L-NT and L-ETX, the window size W used in L-ETX and the weight factor α of the EWMA estimator also affect estimation accuracy. In what follows, we briefly discuss the impact of W and α. Impact of W and α. From Equations and 7, we see that larger window size W will lead to smaller estimation error in L-ETX; on the other hand, a larger W leads to reduced agility for the estimator to respond to changes, which can negatively affect routing performance in the presence of network dynamics. In practice, we usually choose a medium-sized W as a tradeoff between estimation accuracy and agility, and W is set as for the study of this paper. α Let Q k (α) = k+ α+ +α, then, by Equation, DE k is proportional to Q k (α). Figure 6 shows the impact of α on Q k (α).8.6 k = k =.4 k = 4 k = 8. k = 6 k = 3 k = α.6.8 Figure 6: Q k (α) Q k (α) and thus DE k. We see that the optimal α value increases as k increases, and the intuition is that, as k increases, the contribution of the history data (i.e., x, x,..., x k ) to y k becomes more and more important compared with that of the most recent observation x k. On the other hand, the agility of the estimator decreases as α increases. After α exceeds certain threshold value, moreover, Q k (α) (and thus DE k ) increases as α increases further. In practice, therefore, we can choose a value that tradeoffs between estimation precision and agility, and we set α as 7 8 in our study. 3.3 Experimentation with L-NT and L-ETX Having shown that L-ETX is a more accurate estimation method than L-NT in Proposition, we evaluate the validity of the analytical result using the experimental data collected for unicast transmissions in Section, and we experimentally study the impact of estimation accuracy on the optimality and stability of routing. We have done the experimental analysis for different interference scenarios and observed similar phenomena. Due to the limitation of space, here we only present 6

7 data for the case when d =.. Estimation accuracy. COV.. NT PDR Figure 7 shows the COV of NT and distance (meter) Figure 7: COV[NT] vs. COV[PDR] when d =. PDR for different links. We see that COV[NT] is significantly greater, up to a factor of 7.78, than COV[PDR], which is consistent with our analysis as shown by Inequality 8. Accordingly, the degree of estimation error (DE) in L-NT is consistently greater than that in L-ETX, as shown in Figure 8 where DE L NT L ETX distance (meter) Figure 8: DE(L-NT) vs. DE(L-ETX) when d =. DE(L-NT) and DE(L-ETX) for different links are presented. Therefore, the experimental results corroborate the prediction of Proposition. To elaborate on the details of link estimation, Figure 9 shows, for a link of length 9. meters (i.e., 3 feet), the time Number of transmissions NT trueetx L NT L ETX Time series Figure 9: Time series of estimated ETX values in L-NT and L-ETX for a link of length 9. meters (i.e., 3 feet) series of the estimated ETX values via L-NT and L-ETX respectively. (Note: the figure is more readable in color-print than black-white print.) To visualize the accuracy of L-NT and L-ETX, we also show the NT values carried in MAC feedbacks and the actual ETX value for the link. To easily represent a unicast transmission failure (after 7 retransmissions), we present -8 as the NT value for the corresponding unicast transmission. We see that the estimated ETX in L-NT tends to deviate from the actual ETX value, especially in the presence of MAC transmission failures; the estimated ETX in L-ETX, however, is very close to the actual ETX value, even in the presence of MAC transmission failures. We also see that the estimated ETX value in L-ETX is pretty stable whereas the estimation of L-NT oscillates significantly, which has significant implications to routing behaviors as we discuss next and in Section 4.. Routing optimality and stability. To understand the routing behaviors in L-NT and L-ETX, we consider the case where the sender on the left end of the middle row of Figure (b) needs to select the best next-hop forwarder among the set of receivers in the middle row, and the destination is far away from the sender but in the direction extending from the sender along the middle row to the right. For simplicity, we assume that the sender uses a localized, geographic routing metric ETD (for ETX per unit-distance to destination) in evaluating the goodness of forwarder candidates. The metric ETD is defined as follows: given a sender S, a neighbor R of S, and the destination D, the ETD via R is defined as { ETXS,R L S,D L R,D if L S,D > L R,D otherwise where ETX S,R is the ETX of the link from S to R, L S,D denotes the distance from S to D, and L R,D denotes the distance from R to D. We will show in Section that this local, geographic metric performs in a similar way as the global, distance-vector metric for uniform networks. In our case, the best forwarder is grid-hops away from the sender since the corresponding link has the minimum ETD value. To see how L-NT and L-EXT perform in selecting the next-hop forwarder, Table shows the forwarders (identified Method Forwarder Percentage (%) Cost ratio L-NT L-ETX Table : Forwarders used in L-NT and L-ETX in terms of their grid hop distance from the sender) used in L-NT and L-ETX respectively. To illustrate the optimality of different methods, we measure the percentage of time each forwarder is used, and the cost ratio of this forwarder to the optimal forwarder. We see that L-ETX is able to identify and use the optimal forwarder more than % of the time compared with L-NT. The average ETD of using L-ETX is 3.6% more than the optimal ETD, yet the average ETD of using L- NT is.34% more than the optimal ETD. We also measure the number of times that the sender changes its forwarder when using L-NT and L-ETX respectively, and we observe that the number of forwarder changes (9) 7

8 is 9 and 3 in L-NT and L-ETX respectively. Thus, L-ETX ensures much higher routing stability than L-NT, which is due to the fact that L-ETX is a more stable estimator than L-NT as can be seen from Figure 9. Higher routing stability helps improve the predictability of packet delivery performance in networks. 4 Routing performance Having discussed the significant impact that link estimation methods have on estimation accuracy and routing optimality in Sections and 3, we experimentally evaluate the performance of different link estimation methods in this section. We first present the methodology, then compare different data-driven estimation methods, and we finally compare data-driven link estimation with beacon-based approaches. 4. Methodology We use a publicly available event traffic trace for a field sensor network deployment [3] to evaluate the performance of different protocols. Since the traffic trace is collected from 49 nodes that are deployed in a 7 7 grid, we randomly select and use a 7 7 subgrid of the Kansei testbed (as shown in Figure (b)) in our experiments. To form a multi-hop network, we set the radio transmission power at -4dBm (i.e., power level 3). The mote at one corner of the subgrid serves as the base station, the other 48 motes generate data packets according to the aforementioned event traffic trace, and the destination of all the data packets is the base station. We also evaluate protocols with other traffic patterns, e.g., periodic data traffic, and other network setups, e.g., random networks. We observe phenomena similar to what we will present, but we relegate the detailed discussion to Section. Using the above setup, we comparatively study the performance of the following data-driven link estimation and routing protocols: 4 L-NT: a distance-vector routing protocol whose objective is to minimize the expected number of transmissions (ETX) from each source node to its destination. The ETX metric of each link (and thus each route) is estimated via the L-NT data-driven method. L-ETX: same as L-NT except that the ETX metric is estimated via the L-ETX method. L-WNT: a variant of L-NT where the input to the EWMA estimator is the average of NT values for every consecutive unicast transmissions. We study this protocol to check whether the performance of protocol L-NT can be improved by increasing the stability of the L-NT method through the window-based NT average. L-NADV: a variant of L-ETX where the window size W is and the EWMA estimator is used to estimate packet error rate (PER) instead of PDR. We study this proto- col mainly to examine the impact of W. L-NADV is also the distance-vector version of the geographic protocol NADV [7]. In the above data-driven protocols, periodic, broadcast beacons are never used. We use the approach of initial link sampling [4] to jump-start the routing process, where a node proactively takes 7 samples of MAC feedback (by transmitting 7 unicast packets) for each of its candidate forwarders and then chooses the best forwarder based on the initial sampling results. To understand the importance and benefits of data-driven link estimation and routing, we also study the performance of the following beacon-based protocols: ETX: a distance vector, beacon-based routing protocol whose objective is to minimize the ETX from each source to its destination. The ETX metric is estimated based properties of broadcast beacons. This is similar to the protocol proposed in []. RNP: same as protocol ETX except that the ETX metric is estimated as the required number of packets (RNP) [9] which tries to capture the temporal correlation in link properties. For each protocol we study, we ran the event traffic trace sequentially for 4 times, and we measure the following protocol performance metrics: Event reliability (ER): the number of unique packets received at the base station divided by the total number of unique packets generated for an event. This metric reflects the amount of useful information that can be delivered for an event. Number of transmissions per packet delivered (NumTx): the total number of physical transmissions incurred in delivering packets of an event divided by the number of unique packets received at the base station. This metric affects network throughput; it also reflects the energy efficiency of a protocol, since it not only affects the energy spent in transmission but also the degree of duty cycling which in turn affects the energy spent at the receiver side. 4. Different data-driven estimation methods Figures and show the event reliability and the average Event reliability (%) L NT L WNT L ETX L NADV Figure : Event reliability 4 In this paper, we sometimes use the same name for the protocol, the estimation method, and the routing metric. The context of its usage will clarify its exact meaning. Our experiments show that routing performance is statistically the same whether we use PER or PDR as the input to the EWMA estimator. 8

9 NumTx L NT L WNT L ETX L NADV Figure : Average number of transmissions per packet delivered number of transmissions required for delivering each packet in different data-driven protocols respectively, Figures and 3 show the average route hop length and route transmission Average route hop length L NT L WNT L ETX L NADV Grid hops Figure : Average route hop length for nodes different gridhops away from the base station L-NT vs. L-ETX. From Figure, we see that L-ETX achieves a significantly higher event reliability than L-NT. For instance, the median event reliability in L-ETX is 9.63%, which is.8% higher than that in L-NT. The higher event reliability in L-ETX is due to the facts that the routes used in L-ETX are shorter than those in L-NT and the reliability of the links used in L-ETX is higher than that in L-NT, as shown in Figure and Table respectively. Due to the same reason, L-ETX achieves significantly higher energy efficiency than L- NT, as shown in Figure. For instance, the average number of packet transmissions required in delivering a packet to the base station in L-ETX is.8, which is 3.7 times less than that in L-NT. From Table, we see that the links used in L-ETX are longer yet more reliable than those in L-NT. This implies that L-ETX enables nodes to choose better routes in forwarding data and thus leads to significantly better performance in data delivery. Variants of L-NT and L-ETX. Counterintuitively, Figures and show that L-WNT performs worse than L-NT, and Table shows that L-WNT chooses worse routes than L- NT does. Through careful analysis, we find out that the cause for the worse performance of L-WNT is that, even though L- WNT is a more stable estimator than L-NT, it is slower (i.e., less agile) in adapting to link property changes. The slow convergence in L-WNT further exacerbates the error in NT-based estimation and leads to larger estimation error compared with L-NT, especially in the presence of dynamics. This can be seen from Figure 4 which shows the time series of the esti- Average route txcount L NT L WNT L ETX L NADV Grid hop Figure 3: Average route transmission count for nodes different grid-hops away from the base station count respectively for successfully delivered packets coming from nodes at different grid-hops away from the base station, and Table shows the detailed information about the proper- Metric L-NT L-WNT L-ETX L-NADV Average per-hop geo-distance (meter) Average per-hop physical tx reliability (%) Average per-hop unicast reliability (%) Per-hop ETX Table : Per-hop properties in different routing protocols ties of the links used in different protocols. Number of transmissions NT trueetx L WNT L NT Time series Figure 4: Time series of estimated L-WNT and L-NT for a link of length 9. meters (i.e., 3 feet) mated ETX values in L-WNT and L-NT for a link of length 9. meters (i.e., 3 feet). As expected, L-NADV performs slightly worse than L- ETX, as shown in Figure, Figure, and Table. This is due to the larger estimation errors in L-NADV, especially in the presence of transmission failures. This can be seen from Figure which presents the time series of the estimated ETX values in L-NADV and L-ETX for a link of length 9. meters (i.e., 3 feet). Route stability. Table 3 shows the route stability measured by comparing the routes taken by every two consecutive packets. We see that L-ETX (and its variant L-NADV) is very stable and seldom changes route (only at a probability of.3%), yet L-NT (and its variant L-WNT) tends to be much 9

10 Number of transmissions 4 3 txcount trueetx L NADV L ETX Time series NumTx ETX RNP L ETX Figure : Time series of estimated L-NADV and L-ETX for a link of length 9. meters (i.e., 3 feet) Two consecutive routes L-NT L-WNT L-ETX L-NADV (%) Same Diff. routes but same hop length Increased hop length Decreased hop length.4.63 Table 3: Route stability measured by comparing the routes taken by every two consecutive packets more unstable. The fact that nodes seldom change routes in L-ETX also shows that initial link sampling is able to identify the best forwarders for most nodes in L-ETX. Stability in routing helps facilitate other control logics such as QoS-oriented structuring and scheduling, but detailed discussion of this is beyond the scope of this paper. 4.3 Data-driven vs. beacon-based estimation Having shown in the previous subsection that L-ETX is the appropriate data-driven routing protocol to use in practice, we compare the performance of L-ETX with that of beacon-based protocols ETX and RNP in this subsection. Figures 6 and 7 show the event reliability and the average number of transmis- Event reliability (%) ETX RNP L ETX Figure 6: Event reliability sions required for delivering each packet in different protocols respectively. We see that L-ETX achieves significantly higher event reliability (e.g., up to 8.7%) than both ETX and RNP. L-ETX also achieves higher energy efficiency than ETX and RNP, by a factor of.43 and.96 respectively. To explain the above observations and to understand the detailed routing behavior, we present in Figures 8 and 9 the average route hop length and route transmission count respec- Figure 7: Average number of transmissions per packet delivered Average route hop length.. ETX RNP L ETX Grid hops Figure 8: Average route hop length for nodes different gridhops away from the base station Averag route txcount 4 3 ETX RNP L ETX Grid hop Figure 9: Average route transmission count for nodes different grid-hops away from the base station tively for successfully delivered packets coming from nodes at different grid-hops away from the base station. We also present in Table 4 the detailed information about the proper- Metric ETX RNP L-ETX Average per-hop geo-distance (meter) Average per-hop physical tx reliability (%) Average per-hop unicast reliability (%) Per-hop ETX Table 4: Per-hop properties ties of the links used in different protocols. Figure 8 shows that the routes taken by successfully delivered packets in ETX are of similar hop length as those in L-ETX, and Figure 9 shows that the route transmission count for successfully delivered packets in ETX tends to be slightly smaller than that in

11 L-ETX; yet Table 4 shows that the links used in L-ETX are much longer than those used in ETX and the per-hop ETX in L-ETX is less than that in protocol ETX. This seemingly contradicting observations implies that packets that have been lost in ETX tend to go through long and unreliable routes. Similar phenomena are also observed in RNP. Table 4 shows that the average per-hop unicast reliability in L-ETX is significantly higher (e.g., up to 7.9) than that in ETX and RNP, which enables L-ETX to achieve a significantly higher event reliability. The facts that L-ETX achieves higher event reliability and that L-ETX uses more reliable links and thus fewer number of transmissions enable L-ETX to deliver packets with fewer number of transmissions on average. (Note: a counterintuitive observation from Table 4 is that links used in ETX have higher physical transmission reliability than those in L-ETX, yet the unicast reliability in ETX is lower than that in L-ETX. The reason for this is that the physical transmission failures are more separated and consecutive physical transmission failures are rarer in L-ETX so that they do not degenerate unicast reliability in L-ETX that much.) Table shows the route stability in ETX, RNP, and L-ETX. Two consecutive routes ETX RNP L-ETX (%) Same Diff. routes but same hop length Increased hop length Decreased hop length.7.87 Table : Route stability: data-driven vs. beacon-based estimation We see that the routes used in L-ETX are also slightly more stable than those in beacon-based routing protocols ETX and RNP. Discussion In Sections 3 and 4, we have mainly studied the two representative data-driven protocols L-NT and L-ETX, and the study is mainly based on the regular 7 7 event traffic trace and testbed configuration. In this section, we discuss two more data-driven protocols which are related to L-ETX and/or L-NT, and we comparatively study L-ETX, L-NT, and ETX with other traffic patterns and network configurations. We also compare the achievable throughput in L-ETX, L-NT, and ETX respectively. Other data-driven protocols. We implement the protocol L-ETX-geo that is a geographic-version of L-ETX and uses the ETD metric (as discussed in Section 3.3); we also implement the protocol L-ML which is the same as L-NT except for the fact the MAC latency carried in MAC feedback is used to measure link quality and the routing objective is to minimize the end-to-end MAC latency in packet delivery. L-ML is very similar to L-NT since, given a certain degree of channel contention, the number of transmissions determine the MAC latency. For the same testbed and traffic setup as in Section 4., Figure shows the event reliability, and Figure shows the av- NumTx Event reliability (%) L ETX geo L ML L ETX Figure : Event reliability L ETX geo L ML L ETX Figure : Average number of transmissions per packet delivered erage number of transmissions per packet delivered in different protocols. Even though detailed study of geographic routing as compared with non-geographic, distance-vector routing is a non-trivial research issue itself and is beyond the scope of this paper, we see that L-ETX-geo achieves similar performance (e.g., statistically equal median event reliability at 9% confidence level) as L-ETX in our testbed where nodes are uniformly distributed. Since L-ML is similar to L-NT, we see that the event reliability and energy efficiency in L-ML are lower than those in L-ETX. Other traffic load. Using the same event traffic trace as in Section 4., we control the set of nodes that actually generate source packets to imitate events of different sizes. We have experimentally compared L-ETX, L-NT, and ETX using two event size: 3 3 where the nodes in the farthest 3 3 subgrid from the base station generate packets, and where the nodes in the farthest subgrid from the base station generate packets. We observe similar phenomena for both event size configurations, and here we only present the data for the configuration only. For the same testbed setup as in Section 4., Figure shows the event reliability, and Figure 3 shows the average number of transmissions per packet delivered in different protocols. We see that, in the case of lighter traffic load, L-ETX still achieves higher event reliability and energy efficiency (as measured in the number of transmissions required in delivering a packet to the base station) than L-NT and ETX. The performance of L-NT is also significantly worse than that of ETX, and this shows the importance of using the right datadriven metrics in taking advantage of data-driven link estimation and routing. Compared with the case shown in Figure,

12 Event reliability (%) NumTx NumTx 3 Figure : : event reliability Figure 3: : average number of transmissions per packet delivered Figure : Periodic traffic: average number of transmissions per packet delivered traffic trace as in Section 4., Figure 6 shows the event reli- Event reliability (%) the event reliability of L-NT is lower because the average distance between the traffic sources and the base station is larger in this case, which implies longer routing hops. Periodic traffic. To understand routing performance in periodic data collection applications, we experimentally compare performance of different routing protocols in delivering periodic traffic. To this end, we let all the nodes except for the base station in a 7 7 grid periodically generate packets with an average inter-packet interval of seconds. For the same testbed setup as in Section 4., Figure 4 shows the packet delivery Figure 6: Sparser network: event reliability ability, and Figure 7 shows the average number of transmis- NumTx Packet delivery reliability (%) Figure 4: Periodic traffic: packet delivery reliability reliability, and Figure shows the average number of transmissions per packet delivered in different protocols. We see that L-ETX outperforms L-NT and ETX in delivering periodic traffic. Sparser network. To understand whether the relative performance among different protocols is consistent across different network densities (e.g., number of neighbors for each node), we change the radio transmission power to -7dBm (i.e., power level ). Then for the same node deployment and Figure 7: Sparser network: average number of transmissions per packet delivered sions per packet delivered in different protocols. We see that L-ETX achieves higher event reliability and energy efficiency than both L-NT and ETX. Compared with the case when the radio transmission power is -4dBm, the event reliability in a protocol is lower when the power level is -7dBm because the routing hop length increases as a result of the reduced power level. Random topology. Instead of deploying 49 nodes in a regular 7 7 grid, we deploy 49 nodes in a randomly selected set of 49 grid points from the 4 7 grid space as shown in Figure (b). Then for the same traffic trace as in Section 4., Figure 8 shows the event reliability, and Figure 9 shows the average number of transmissions per packet delivered in different protocols. We see that L-ETX achieves significantly higher event reliability and energy efficiency than L-NT and