Distributed Graph Routing for WirelessHART Networks

Transcription

1 Graph Routing for WirelessHART Networks Venkata P Modekurthy Department of Computer Science Wayne State University Detroit, MI, USA Abusayeed Saifullah Department of Computer Science Wayne State University Detroit, MI, USA Sanjay Madria Department of Computer Science Missouri University of Science & Tech. Rolla, MO, USA ABSTRACT Communication reliability in a Wireless Sensor and Actuator Network (WSAN) has a high impact on stability of industrial process monitoring and control. To make reliable and real-time communication in highly unreliable environments, industrial WSANs such as those based on WirelessHART adopt graph routing. In graph routing, each packet is scheduled on multiple time slots using multiple channels, on multiple links along multiple paths on a routing graph between a source and a destination. While high redundancy is crucial to reliable communication, determining and maintaining graph routing is challenging in terms of execution time and energy consumption for resource constrained WSAN. Existing graph routing algorithms use centralized approach, do not scale well in terms of these metrics, and are less suitable under network dynamics. To address these limitations, we propose the first distributed graph routing protocol for WirelessHART networks. Our distributed protocol is based on the Bellman-Ford Algorithm, and generates all routing graphs together using a single algorithm. We prove that our proposed graph routing can include a path between a source and a destination with cost (in terms of hop-count) at most 3 times the optimal cost. We implemented our proposed routing algorithm on TinyOS and evaluated through experiments on TelosB motes and simulations using TOSSIM. The results show that it is scalable and consumes at least 4% less energy and needs at least 65% less time at the cost of 1kB of extra memory compared to the state-of-the-art centralized approach for generating routing graphs. CCS CONCEPTS Networks Network protocols; Routing protocols; KEYWORDS Graph Routing, Wireless Sensor Actuator Networks, WirelessHART, Algorithm ACM Reference format: Venkata P Modekurthy, Abusayeed Saifullah, and Sanjay Madria Graph Routing for WirelessHART Networks. In Proceedings of 19th International Conference on Computing and Networking, Varanasi, India, January 4 7, 218 (ICDCN 18), 1 pages. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. ICDCN 18, January 4 7, 218, Varanasi, India 218 Association for Computing Machinery. ACM ISBN /18/1... $ INTRODUCTION Wireless Sensor and Actuator Network (WSAN) provides an efficient communication infrastructure for a broad range of industrial control applications [11]. Reliability of communication in a WSAN has a high impact on the stability of critical control applications like process control [16], smart manufacturing [27], smart grid [15], and data center power control [25]. Packet losses in such applications may lead to highly unstable systems. Feedback control loops in a WSAN therefore impose stringent depability requirements on communication between sensors and actuators. However, industrial environments make it difficult to meet these requirements because of frequent transmission failures due to channel noise, limited bandwidth, physical obstacles, multi-path fading, and interference from coexisting wireless devices [27]. Of late, WSAN has received a new impulse with the advent of industrial wireless standards such as WirelessHART [6]. WirelessHART was specifically designed to operate in highly unreliable environments. With approximately 3 million HART devices installed across the world, it is predominantly being used worldwide for process management [1]. To make reliable and real-time communication in highly unreliable environments, a key technique adopted in WirelessHART is reliable graph routing [6]. In graph routing, a packet is scheduled on redundant time slots, on redundant links on multiple paths leading to its destination, and on multiple channels based on TDMA (Time Division Multiple Access) for enhanced -to- reliability. Packets from all sensor nodes are routed to the controller through an uplink graph. For each actuator, there is a downlink graph from the controller through which control messages are delivered to it. While graph routing with such a high degree of redundancy is crucial to reliable and real-time communication, determining and maintaining such routes is challenging in terms of run time and energy consumption at resource constrained WSAN devices. In this paper, we address these challenges and study highly efficient and scalable graph routing for WirelessHART networks. While there are enormous routing algorithms for wireless networks [9], graph routing for WSAN has been studied recently [16, 3]. These existing algorithms use centralized heuristics and do not scale well with number of nodes. Emerson [2] and MOL[3] are targeting to deploy WSAN networks that span across thousands of node making scalability in WirelessHART networks a notable problem. Creating all routing graphs centrally and disseminating to all nodes is less suitable for large industrial WSANs particularly in presence of network dynamics. As wireless conditions change frequently in industrial environments, the central manager has to frequently reconfigure and re-disseminate the routing graphs. Frequent dissemination is highly energy and bandwidth consuming and can hinder critical control operations. To address this, we propose the first fully distributed protocol for graph routing for

2 WirelessHART networks, thus obviating the need for centrally creating and disseminating the routes. While a distributed protocol is highly scalable and responsive to network dynamics, developing such a protocol is challenging as it must guarantee convergence and achieve near optimal (in terms of routing cost) performance based on local computations only. We design the proposed distributed graph routing protocol by adapting the Bellman-Ford algorithm [2] for WSANs. In the Bellman- Ford algorithm a node broadcasts its routing table on path updates, which contains the cost of reaching from itself to all other nodes. Thus, each node updates its routing table according to those of its neighbors and generates all possible paths. This means that if a link on a route fails, a node can forward a packet through alternative paths which is the essence of graph routing. We leverage this feature of the Bellman-Ford algorithm to develop our distributed graph routing protocol. However, as the routing table at a node contains costs to all other nodes, message size and energy required to converge pose a key challenge for low power and low bandwidth WSAN nodes. To handle this, we use node clustering where the routing table at a node needs to contain link cost information of only the nodes in its own cluster and cluster heads of the other clusters, and generate routes only to the needed destinations. Another key feature of our protocol is that it can generate all routing graphs together using a single algorithm unlike existing approaches that use a separate algorithm for each. When the nodes in a cluster are single-hop away from its cluster head, least cost path on each routing graph yielded by our approach is guaranteed to have a cost (in terms of hop-count) at most 3 times the optimal cost. We implemented our proposed routing algorithm on TinyOS and evaluated through experiments on TelosB motes and through TOSSIM simulations. We determine the effectiveness of our distributed routing in terms of scalability, energy, and under network dynamics. The simulation results demonstrate that our protocol consumes at least 82.4% less energy and needs at least 66.1% less time compared to the state-of-the-art centralized approach [16, 3] for generating routing graphs. In the rest of the paper, Section 2 presents related work. Section 3 presents our network model. Section 4 gives an overview of graph routing. Section 5 describes our distributed routing protocol. Section 6.2 analyzes its performance. Section 7 presents our implementation and evaluation. Section 8 concludes the paper. 2 RELATED WORK Routing in wireless ad hoc and sensor networks has been studied extensively on issues like energy efficient [23], time-depent [18], hierarchical [19, 29] and multi-path [1, 22]. However, ensuring reliability of communication during node or link failures was mainly explored in multi-path routing. The multi-path routing protocols like AOMDV [22] provide multi-path routing to provide a certain degree of -to- reliability. More recently, Bee-Sensor- C [1] integrates clustering and agents to obtain multi-path routes. GDMR [31] presents a heuristic multi-path routing for optimizing network performance by load balancing traffic in IP networks. Two distributed multi-path routing algorithms were developed in [14] to let nodes efficiently find next-hops for each destination and guarantee a few metrics of QoS routing. The above multi-path routing protocols provide a few either node-disjoint or link-disjoint paths between a source and a destination, thereby providing a certain degree of -to- reliability. In contrast, graph routing is particularly designed to achieve a high degree of reliability in WirelessHART networks for real-time process monitoring and control applications in highly unreliable industrial environments. In each routing graph, a node has multiple neighbors to which it can transmit and retransmit a packet multiple times to be delivered towards the corresponding destination. The same holds for each node the packet passes through. Thus, a routing graph for a control loop can consist of an exponential number of routes between its source and destination; many of the paths may be overlapping and the packet follows one path based on the link conditions. For reliability, every node in a routing graph in a WirelessHART network should have at least two neighbors to provide redundant paths of routing. Thus, a routing graph with l nodes can have at least 2 l paths between the corresponding source and the destination. This provides a high-degree of reliability and existing multi-path routing cannot be adopted as it will have to generate an exponential number of paths between source and destination. Real-time scheduling for process control based on graph routing has received considerable interest over the last five years [16, 25 27, 32]. Specifically, real-time scheduling [13, 27], schedulability and -to- communication delay analysis [25], priority assignment [26], and localization [32], has been studied recently for WirelessHART networks based on graph routing. However, all these works assume that routing in the network layer is given. While there are enormous routing algorithms for wireless networks [9], graph routing for WirelessHART has been studied only recently [16, 3] and these algorithms use centralized heuristics to generate the uplink graph and each downlink graph separately at the expense of scalability. Emerson [2] and MOL[3] are targeting to deploy WSAN networks of thousands of nodes making scalability in WirelessHART networks a notable problem for centralized approach. Also, frequent dissemination of the routes to all nodes is highly energy and bandwidth consuming particularly in presence of network dynamics. In contrast, we propose a fully distributed protocol for graph routing for WirelessHART networks, thus obviating the need for centrally creating and disseminating the routes. 3 NETWORK MODEL WirelessHART, designed on the top of IEEE , operates in the 2.4GHz band. It forms a multi-hop mesh network consisting of a Gateway, field devices, and access points. The network manager and controller are at the Gateway. Field devices are wirelessly networked sensors and actuators which are equipped with a half-duplex omnidirectional radio transceiver that cannot both transmit and receive at the same time and can receive from at most one ser at a time. Access points provide redundant paths between the wireless network and the Gateway. The network involves feedback control loops between sensors and actuators through the controllers. Transmissions are scheduled based on a multi-channel TDMA (Time Division Multiple Access) protocol. The network uses the channels defined in IEEE Time is globally synchronized. Each time slot is of fixed length. A transmission and its acknowledgement (ACK) needs one time slot. For transmission between a 2

3 HOST APPLICATION v3 v9 NETWORK MANAGER v1 v6 v3 Access point v2 v5 6,7 3,4 9,1 (a) Network Topology (b) Uplink Graph v4 v6 Actuator v7 v8 (c) Downlink graph for v 6 c a b 1,2 Figure 1: An example of graph routing Figure 2: A schedule example receiver and its ser, a time slot can be either dedicated or shared for the link between the ser and the receiver. In a dedicated slot, only one ser is allowed to transmit to a receiver. In a shared slot, multiple sers can attempt to s to a common receiver. For enhanced reliability, the network adopts graph routing [6]. Graph routing allows to schedule a packet on multiple links using multiple channels on multiple time slots through multiple paths to deliver a packet to a destination, thereby ensuring high reliability in unreliable environments. The network is represented as a graph G = (V, E), where V is the set of nodes and E is the set of edges. We consider a set of destination nodes D V and a node v D represents a process controller or an actuator node. We use V, E, and D to denote the number of nodes, the number of edges, and the number of destination nodes in the network, respectively. 4 AN OVERVIEW OF GRAPH ROUTING A routing graph in WirelessHART network is a directed list of loop free paths between a source and a destination. It is thus a directed acyclic graph from a source to a destination, and each node in the graph route except the destination has a minimum of 2 neighboring nodes. Redundancy in paths at each node in the routing graph circumvents temporary link or node failures allowing retransmission through redundant links. Graph routing defines two types of routing graphs; uplink graphs and downlink graphs. Packets from all sensor nodes are routed to the Gateway through the uplink graph. For every actuator, there is a downlink graph from the Gateway through which the Gateway delivers control messages. In each routing graph, a node can have multiple neighbors to which it can transmit and retransmit a packet multiple times to be delivered to the corresponding destination. For a network shown in Fig. 1(a), an example of uplink graph routing and downlink graph for actuator v 6 are presented in Fig. 1(b) and Fig. 1(c) respectively. In a routing graph, a primary path is considered as least cost from source to a destination. All other paths in the routing graph are considered as back-up paths. A packet is usually delivered to the destination through the primary path. Upon failure of a transmission on a primary path, it is delivered through a back-up path of the routing graph. In the uplink graph shown in Fig. 1(b), for sensor node v 8 if the primary path on uplink graph is computed to be v 8 v 5 v 3, then all other paths, like v 8 v 5 v 9 v 3, are back-up paths. For a control loop scheduling in the uplink graph, on one path from the source to an access point, the scheduler allocates a dedicated slot for each device starting from the source, followed by allocating a second dedicated slot on the same path to handle a retransmission. Then, to handle failures of both transmissions along a link, it allocates a third shared slot on a separate path to handle another retry. Then the links in the downlink graph are scheduled similarly. Fig. 2 shows an example of transmission scheduling from node a to an access point (AP) for one control loop in a network of 4 nodes. Each link is labeled with the time slots in which it is scheduled. Link a b is first scheduled on dedicated slots 1 and 2; b AP is scheduled on dedicated slots 3 and 4; To handle transmission failures along a b on both slots, the same packet is scheduled on shared slot 3 along a c, and so on. 5 DISTRIBUTED GRAPH ROUTING In this section, we present our distributed graph routing protocol for WirelessHART networks. Our protocol is highly scalable, and adaptable to network dynamics when compared to existing solutions that rely on central manager to create routes and disseminate them to all nodes. In addition our approach is energy efficient for resource constrained WSAN nodes. The algorithm also can converge quickly since the number of message communications required for disseminating all routes by a centralized network manager is usually higher than the number of communications required by our protocol. Our proposed distributed graph routing is an adaptation of Bellman-Ford algorithm [2, 24] for WirelessHART networks. Bellman-Ford algorithm [2, 24] has been used widely in the Internet for distributed routing. In the Bellman-Ford algorithm, a node maintains a routing table containing the costs from itself to all other nodes in the network through all of its neighbors. A node broadcasts its routing table whenever there is a change and neighboring nodes update their routing tables accordingly. As a result, all nodes maintain a least cost path to a destination. The Bellman-Ford algorithm gives more information that we exploit to yield routing graphs. Specifically, a node knows all neighbors that can route a packet to a destination. This means that for temporary link failures, a packet can be forwarded on alternative paths. In graph routing, routes do not need to be recomputed for short term link failures as alternate paths provide reliable communication between a source and a destination pair. However, under sustained link failures, nodes can re-broadcast the routing tables and update their least cost paths. We leverage on Bellman-Ford algorithm s capability to recover from link failures to develop our 3

4 distributed graph routing protocol. The least cost path obtained from the Bellman-Ford algorithm is considered as the primary path in the routing graph and all other paths as back-up paths. Since the routing table at a node contains costs to all other nodes, the energy required to converge on a solution and message size for communicating the routing table in this approach pose a key challenge for low power and low bandwidth WSAN nodes. In addition, Belmman-Ford algorthm does not scale well with number of nodes and consumes O( V E ) messages to converge to an optimal solution. To address these challenges, we propose a novel distributed graph routing protocol based on the Bellman-Ford algorithm for WirelessHART networks as described in the following sub-section. 5.1 Protocol Description To address the challenges on energy overhead for WirelessHART networks, we propose to adopt the distributed Bellman-Ford algorithm through node clustering. That is, the nodes across the network will be divided into clusters C 1,C 2,,C q where each C i has a cluster head denoted by CH i. Our objective of clustering is to minimize the energy required to generate and maintain routes using the Bellman-Ford algorithm, not to solve the problem through a hierarchical networking approach. There exist many algorithms for distributed clustering in wireless sensor networks such as [8, 17] and our proposed approach works with any clustering algorithm. For the sake of simplicity, we assume that the network is clustered during network initialization. After clustering the network, we use the Bellman-Ford algorithm at each node to generate routes to nodes within the same cluster and all cluster heads. That is, each node in the network will have least cost path to all nodes within its cluster, v C i and to all other nodes through their respective cluster heads, CH j where j i. Hence, creating k clusters reduces the energy, memory, and convergence time requirement when compared to the Bellman-Ford algorithms as much as by a factor of O(k). After the convergence of the Bellman- Ford algorithm, cluster-heads broadcast the information of nodes within its cluster to all other nodes in the network. In the proposed approach, a packet with its destination within the same cluster is forwarded on the routing graph to its destination. For a packet with destination v j in another cluster C i, a node forwards the packet on routing graphs to the cluster headch i. Since nodes in a cluster have routes to all other nodes within the same cluster, upon entering the cluster C i, the packet will be forwarded to the destination node v j rather than the cluster head CH i. To further reduce the energy usage, we generate routes and cluster head informaiton to only the destination nodes, that is controller and actuator nodes. This reduces the number of message communication and energy consumption in the proposed algorithm to O( D E ), where D < V. Routes and cluster head information to nodes other than the destination nodes are not generated as intermediate nodes can not process the information in the packet. In section 6.1, we analyze the convergence and optimality property while considering just destination nodes. In the proposed method, we adopt DSDV [24] to avoid generating loops in the system. Our approach can support any other technique like Poison Reverse [2] but we choose DSDV for the sake of simplicity. 4 Algorithm 1: Graph Routing Algorithm Input :destinationnode, nodeid Output :routetable, clusterheadinfo Initially, routetable NULL clusterheadinfo NULL clusterhead = generateclusters() if destinationnode then s routetable if nodeid = clusterhead then s routetable sleep until algorithm converges s clusterheadinfo if received message then for row in routetable do if row is for destination node within same cluster or row is for cluster head then update routetable s routetable if received message and contains clusterheadinfo then update clusterheadinfo s clusterheadinfo The pseudo code of our proposed distributed graph routing algorithm is presented in Algorithm 1. Each node executes this algorithm to construct routing graphs and cluster head information. The algorithm takes a boolean value if the node is destination node or not (destinationnode) and ID of a node (nodeid) as input and generates routing table and cluster head information as output. Given the inputs, nodes first run a clustering algorithm (Leach in our case) to generate and identify cluster heads. After clustering, destination nodes and cluster heads initiate Bellman-Ford algorithm by sing routing tables. If the received message at a node contains route to a cluster head, then the node updates the cost in its routing table with the received new cost value. Upon updating the routing table with new cost value, each node re-broadcasts the routing table to all other nodes. If a node receives a message from a destination node, it updates the cost only for destination nodes within its cluster. If the destination is from a different cluster then routing table update message is ignored and is not re-broadcasted. Once the Bellman- Ford algorithm converges, each cluster head broadcasts cluster head information to all nodes in the network. Cluster head information packet from a cluster head contains node IDs of destination nodes that are a part of same cluster as the cluster head. Route Maintenance. While distributed graph routing is most suitable for distributed transmission scheduling of WirelessHART networks, our proposed distributed graph routing can be used for centralized scheduling also. In such case, routing is maintained through management superframe and periodic maintenance communication (that WirelessHART devices already do) with neighbors. Specifically, when any reconstruction of routes is needed, communication for reconstructing the routes can happen through management superframe and periodic maintenance communication. Since graph route provides redundant paths for packet delivery, reconstruction of routes may not be needed after some path breaks due to

5 v2 v3 v4 v v v Routing Table v6 v7 Cluster Head Info. v1 v2 v v v v Routing Table v6 v7 Cluster Head Info. v3.9 v v1.5 v6.6.7 v2.8 v v7.8 v5.9 v8 v4 1.5 v5 v7 v4 - v v v Routing Table v3 v2 Cluster Head Info. v7 v6, Figure 3: Example of routing tables v 3.9 v v v v 2.8 v v 7.8 v v 8 4.9, v2 Cluster Head v3 Destination v 7, v 2 Cluster Head v 6, v 3 Destination Figure 4: Example of intra-cluster routing v 3.9 v v v v 2.8 v v 7.8 v v v 7, v 2 Cluster Head v 6, v 3 Destination Figure 5: Example of inter-cluster routing link or node failure. Thus, reconstruction of routes are not frequently needed. Future distributed schedulers (where both routing and scheduling are distributed) which are required for large-scale WirelessHART networks [2, 3] will be specially benefitted by our distributed graph routing. 5.2 Example Fig. 3 illustrates our distributed graph routing for a simple topology with two clusters C 1 (with CH 1 = v 2 ) and C 2 (with CH 2 = v 7 ). It shows the final routing table at node v 1,v 4 and v 8 created through our protocol. Each row in the table gives cost to a destination node or cluster head through each of the neighboring nodes. Route to destination node v 3 is only present at nodes v 1 and v 4 as v 3 is within their clusters. However, nodes v 5 and v 8 do not contain entries to node v 3 as v 3 is not in the same cluster as v 5 and v 8. Consider v 3, as shown in Fig. 3, as the destination for packet at v 4. From its routing table, v 4 is aware of an optimal route through v 2 hence v 4 forwards the packet to v 2. If the link to v 2 or node v 2 fails, node v 4 can also forward the packet to v 1. Similarly, both nodes v 1 and v 2 can forward the packet through an optimal path to v 3 and in case of failure, they can transmit on an alternate path. In this example, least cost path (or primary) from source to destination is v 4 v 2 v 3 while other listed paths are back-up paths. Consider v 6 as the destination for a packet originating at v 1. Since v 6 and v 1 are in different clusters, node v 1 does not have a route to v 6 in its routing table. However, from cluster head information stored at node v 1, it can identify the cluster head for node v 6 is v 7. From its routing table, node v 1 is aware of a path to v 7 and 5 forwards the packet to v 4 and in case of failure of transmission it can forward on an alternate path through v 2 or v 3. Similarly node v 4 forwards packet on an optimal path to node v 5. Node v 5 however is aware of an optimal path to v 6 through node v 9 and chooses to forward the packet to node v 9. Node v 9 then forwards the packet on an optimal path to node v 6. In this example, the primary path of routing graph from v 1 to v 6 is v 1 v 4 v 5 v 9 v 6. 6 THEORETICAL ANALYSIS in this section, we first discuss the convergence and sub-optimality of the proposed approach. We then discuss the performance analysis of the routes generated by the proposed algorithm when compared to an optimal solution generated by a centralized scheduler. Notations used in appix are defined in Table Convergence and Sub-optimality We first discuss the convergence of routes generated by the proposed distributed graph routing algorithm. In the proposed distributed graph routing approach, we limit Bellman-Ford algorithm to compute optimal routes to known destination nodes within each cluster. In this section, we show that limiting the number of nodes does not impact convergence and optimality of routes between a source and a destination. We discuss the convergence and optimality for both route generation phase and update phase. During the route generation phase, optimal routes to a single source is proved to converge in V 1 steps in [12]. As described below, exting this proof to multiple destinations proves that Bellman-Ford algorithm converges and generates optimal routes to a set of destination nodes. Bellman-Ford algorithm is known to generate all possible paths from a source to destination. These paths can be viewed as a directed acyclic graph rooted at the source. Suppose a optimal path or primary path of the graph route is given as v 1 v 2 v k, where v 1 and v k are source and destination nodes, respectively, a link failure between nodes v i and v j updates the cost on link v j to. The, routing table update at node v i changes the local optimal path to v k based on the new minimum value in its routing table. After the update, node v i re-broadcasts its routing table to all of its neighbors. Similar to node v i, node v i 1 updates its routing table based on the received routing table from v i and the new minimum value in its routing table is chosen as the local optimal path. A similar operation is carried out at all nodes between v i 1 and v 1, and new local optimal paths are generated at each node. Since all paths in the directed acyclic graph are rooted at node v 1 and local optimal cost is the lowest cost among all other paths, the local optimal path is the global optimal path. Therefore, optimality property for Bellman-Ford algorithm under generation and update phase is preserved even when only destination nodes are considered. 6.2 Performance Analysis For a source and destination pair, cost on the primary path of a routing graph generated by our distributed graph routing may not be optimal. In this sub-section, we discuss the approximate ratio of primary path cost obtained by the distributed graph routing compared to the optimal cost on the primary path of a routing graph. We consider the cost on a path for a pair of source and

6 destination nodes as the number of hops between them. Note that the cost obtained on the primary path deps on the clustering algorithm used to cluster the nodes. In addition, cost also deps on the position of the nodes in the network. If v i and v j are in the same cluster then our distributed graph routing generates a least cost path from v i to v j which gives the approximation ratio as 1. When v i and v j are in different clusters, v i forwards packets on a path to the cluster head (CH (v j )) of v j. If the shortest path from v i to cluster head CH (v j ) passes through v j then packet reaches the destination in the optimal cost as it is sent to destination within the cluster. However, cost is maximum when shortest path to CH (v j ) does not pass through destination v j. We derive an approximation ratio for hop-count on the primary path in Theorem 1. Theorem 1. Let L(v i,v j ) be the cost of the primary path of the routing graph between a source v i and a destination v j generated by our distributed algorithm, when cost is considered as the number of hops between v i and v j. Then L(v i,v j ) is at most R(v i,v j ) + CH (v j ) 2, where CH (v j ) is the cluster head of v j, R(v i,v j ) is the optimal path cost between v i and v j, and R(v i,ch (v j )) is the optimal path cost between v i and CH (v j ). Proof. When v i and v j are in the same cluster, the hop-count on the primary path is the same as that of an optimal path between them. When v i and v j are in different clusters, the hop-count on primary path from v i to CH (v j ) is less than the sum of hop-count on path from v i to v j and the hop-count on path from v j to CH (v j ), that is R(v i,ch (v j )) R(v i,v j ) + R(v j,ch (v j )). This is due to the fact that, the Bellman-Ford algorithm generates a least cost path from a source and destination. Therefore, the maximum value of hop-count from v i to CH (v j ) is R(v i,ch (v j )) = R(v i,v j ) + R(v j,ch (v j )). Hence, the hop-count on the primary path of the routing graph between v i and v j L(v i,v j ) R(v i,v j ) + (2 R(v j,ch (v j ))). (1) The maximum distance between any two nodes in a cluster is considered as the diameter of that cluster. Assuming that all nodes can generate a graph route, i.e. nodes inside a cluster can form a 2- connected graph, the maximum value of diameter can be expressed as R(v j,ch (v j )) = C (v j ) 2 1. Therefore, Equation (1) can be expressed as L(v i,v j ) R(v i,v j ) + C(v j ) 2. (2) Since hop-count is depent on the clustering algorithm, for this section, we consider all nodes in a cluster are one hop away from their cluster head. For this assumption, we show in Corollary 1 which follows from Theorem 1 that the cost (in terms of hop-count) on the primary path is at most 3 times the optimal cost. Corollary 1. The cost on the primary path, L(v i,v j ), is three times the optimal cost, i.e. L(v i,v j ) = 3 R(v i,v j ), when cost is considered as hop-count and clustering algorithm generates clusters with a one-hop radius, where L(v i,v j ) is the cost on primary path obtained by our distributed graph routing and R(v i,v j ) is the cost on optimal path between v i and v j. Proof. Under the assumption that all nodes in a cluster are one hop away from a cluster head, R(v j,ch k ) R(v i,v j ) 2. The maximum value of diameter of the cluster is 2 and the fraction is maximum when 6 R(v i,v j ) = 1. Hence, R(v j,ch k ) = 2 R(v i,v j ). Thus, by Equation (2), the cost of primary path of routing graphs generated by our approach is at most 3 times that of an optimal path. 7 EVALUATION In this section, we present the evaluation of our distributed graph routing protocol through testbed experiments and simulations. We evaluate the effectiveness of the proposed routing in terms of scalability, energy, and under network dynamics, and compared against existing centralized graph routing approaches for WirelessHART. Energy (mj) Convergence time (s) # of control loops # of control loops (b) Convergence time Figure 6: Experiments under varying number of clusters 7.1 Experiment We implemented distributed graph routing protocol in TinyOS 2.2 [5] and evaluated using TelosB motes [7] through an indoor deployment. Each TelosB device is equipped with Chipcon CC242 radios compliant with the IEEE standard. Note that the physical layer of WirelessHART is based on physical layer. We used LEACH[17] clustering algorithm to randomly create cluster heads and request nodes to join a cluster. After the network is clustered, cluster heads and destination nodes initiate the Bellman- Ford algorithm using signal strength at the receiver as the cost between the two nodes. All nodes in the network generated routing graphs to all cluster heads and destination nodes within their cluster. After converging to a solution, each cluster head broadcasted a list of destination nodes that are in its cluster to all nodes in the network and each node updated its cluster head information accordingly. We assumed that all nodes compute the signal strength prior to the start of execution of the distributed graph routing algorithm. The proposed distributed graph routing algorithm is executed using the default CSMA-CA MAC protocol in TinyOS 2.2. For lower layers of implementation, we use the implementation provided in [28].

7 Our deployment formed a 3-hop network consisting of 11 nodes in the Computer Science Building of Wayne State University. The network is deployed such that each node is in communication range with a minimum of 2 neighbors and the average of number of neighbors in the network is 3. We evaluated our protocol under varying number of control loops and presented the result for an average of 1 experiments performed for each value of control loop. For each experiment we randomly selected a base-station or controller and two cluster heads. The reason for choosing a random setting is to vary the length of routes for each run such that the results represent a larger spectrum of scenarios. During the experiment (results shown in Fig. 6(a) and Fig. 6(b)) we have observed that initial difference in the average number of message communications per node between the two protocols is very small. However, energy consumption of centralized algorithm increases as the number of control loops increases. This is due to the fact that in centralized algorithms number of messages increases with increase in number of required graph routes. We also observed that energy consumed for distributed algorithms also increases due to increase in routes generated. However, the energy consumption for distributed algorithm is smaller than centralized algorithm. We observed that, the distributed protocol uses simultaneous communication to minimize the convergence time while centralized algorithm has to wait till each message is passed to all nodes in the network. Hence, convergence time for distributed algorithm is almost constant while convergence time for centralized algorithm increases linearly. In this experiment, we have also observed that memory consumption for all cases is approximately the same with distributed algorithm consuming 4 bytes of more information (on average) compared to centralized algorithm [16]. 7.2 Simulations For large scale evaluations, we perform simulations on 148 nodes in TOSSIM [21]. We use the testbed model from [28] of 74 nodes to generate the network. In order to scale with the number of nodes, we assume all nodes are placed in a grid structure and replicate this grid. We then add edges between neighboring grids to generate a connected graph. In simulation, we use distributed vertex coloring algorithms to generate a schedule. For our simulations, we consider one percent of the node as access points and nodes with highest degree of neighbors were selected to be access points. Default value for parameters used is this simulation is given in Table 1. Symbol Description Default Value V # of nodes 148 D % of control loops 4 Φ power level of transmission 5 dbm q % of clusters 5 Table 1: Parameters used in evaluations Metrics. We used energy consumed by each protocol to generate and dissipate routes to each node in the network as a metric for comparison. To estimate the energy consumption, we observed that each transmission takes an average of 6ms and using the data sheet [7], we have calculated that CC242 radio requires.38mj of energy. We used this energy consumption per transmission and the number of transmissions from simulation to compute the energy consumed at a node. Note that average energy consumption of each 7 node also includes energy consumed to cluster the network. We used convergence time of the algorithm as another metric. We define convergence time as the time difference between the deployment and generation of routing tables at all nodes in the network. Execution time of the centralized algorithm includes time taken to perform network discovery, time taken to generate routes centrally, and dissipate routes to all nodes in the network. Execution time of distributed routing algorithm includes time taken to discover neighbors and generate all routes. We use average memory required at each node as a metric for comparison. In addition, we use number of broken routes as a metric for comparison. We define number of broken routes as the number of routes with disconnected path in either dedicated or back-up paths. We compare our distributed graph routing () with centralized graph routing () [16] and energy-aware graph routing () [3]. In this section, we discuss the performance of each protocol in terms of energy and convergence time under varying number of nodes, control loops, transmission power, and link failures. We use an average of 5 random test cases for each parameter to obtain our results. For each test case, we randomly generate the sensor, actuator and priority for control loops Performance under Varying Number of Nodes. We now show the performance of the proposed distributed algorithm under scalability of number of nodes. In this simulation, we varied the number of nodes from 74 to 74 while keeping the density of the network constant. We observed that energy consumption of distributed algorithm is effected by increase in number of destination nodes and number of cluster heads in the network. Since in this simulation percentage of destination and percentage of cluster heads are kept constant, the number of destination nodes and number of cluster heads increase linearly with increase in number of nodes. Thus, there is a linear increase in the energy consumption by distributed algorithm. However, centralized algorithm need more energy as each message is propagated to all nodes in the network. algorithms theoretically need an exponential increase in energy consumption in the order of V 2 and our simulations verify the theoretical result. In addition, our simulations (as shown in Fig. 7(a)) show that our distributed algorithm consumes less energy than centralized or energy-aware algorithms. We have observed that distributed graph routing saves a minimum of 65% of energy consumption when compared to energy-aware centralized algorithm and 8% of energy consumption when compared to centralized algorithm. Similar to energy consumption, execution time increases linearly by a factor of V in distributed algorithm and V 2 in centralized algorithms as both are depent on the number of messages in the network. Thus, Fig. 7(b) shows a similar tr as Fig. 7(a) and we observed that distributed algorithm saves a minimum of 68% in execution time. Fig. 7(c) shows the average memory required per node in the network. The distributed algorithm generates all possible paths between a source and a destination and each node maintains a route to all destinations through all of its neighbors. In addition, each node maintains cluster head information of all destination nodes in the network. Therefore, memory used at each node increases linearly with increasing number of nodes in the network. However, for centralized algorithms, only nodes that are a part

8 Energy (J) Convergence time (m) Memory (KB) # of nodes # of nodes # of nodes (b) Convergence time Figure 7: Performance under varying number of nodes (c) Memory consumption Energy (mj) Convergence time (s) Memory (B) # of control loops # of control loops # of control loops (b) Convergence time (c) Memory consumption Figure 8: Performance under varying percentage of control loops of a graph route require memory. Fig. 7(c) shows that distributed algorithm consumes 8B additional memory than the centralized algorithms. Nevertheless, additional memory requirement posed by distributed routing is very less compared to available memory of the WirelessHART or TelosB devices (16kB [4]). We have also observed that for centralized algorithms, most of the nodes in the network use little to no memory and some nodes have very high memory consumption whereas in distributed algorithm, memory consumption is distributed evenly across all nodes in the network. Traffic concentration, in centralized algorithms, at some nodes can lead to early node failures in the network which can lead to network disconnections. These results show that memory overhead of the proposed distributed algorithm is minimal compared to the available memory at nodes Performance under Varying Number of Control Loops. In this simulation, we varied the number of control loops between 5 and 7 in a network of 148 nodes. Since the percentage of cluster heads is constant throughout this simulation, energy consumption of distributed algorithm is depent only on the number of control loops in the network. We observed that, increase in number of control loops results in a small increase in the average energy consumption of all nodes in the network. This is due to the fact that routes for actuator nodes are only computed within a cluster and not at all nodes. For centralized algorithms, increasing number of control loops increases the number of routing tables at each node and thereby the number of messages. This causes a sharp increase in energy consumption for centralized algorithm when compared to distributed graph routing, as shown in Fig. 8(a). We have observed that distributed graph routing saves a minimum of 4% when compared to energy-aware centralized algorithm and 65% when compared to centralized algorithm. Fig. 8(b) shows the execution time required by the two approaches. Similar to energy 8 consumption, execution time for the distributed approach deps on the number of control loops which results in a steady increase in execution time when compared to a sharp increase in centralized algorithms. Fig. 8(b) shows an average decrease of 4% in execution time of distributed graph routing. These results show that distributed algorithm is scalable in terms of number of control loops when compared to centralized algorithms. The effect of number of control loops on memory is shown in Fig. 8(c). algorithm requires cluster head information for all actuators in order to maintain graph routes from all nodes in the network. Memory required by cluster head information is proportional to the number of control loops. This causes a linear increase in memory consumption of distributed algorithm with increase in number of control loops. Nevertheless, centralized algorithm only consumes memory for all nodes in the routing graph. We observe that for 1% source and destination nodes in the network, distributed graph routing consumes 4B additional memory than centralized. Since required additional memory is very less compared to available memory, we can conclude that memory has little effect on scalability of distributed algorithm Performance under Varying Power Level. We varied the transmission power level of all nodes to change the density of the network and we considered 4 power levels 15, 1, 5 and dbm. Fig. 9(a) shows the energy consumption with increase in power levels. As expected, energy consumption of distributed algorithm increases linearly as the number of neighbors increase. However, energy consumption for centralized algorithms decreases due to the presence of new shorter paths and smaller path lengths of each route. We have observed that the difference in energy consumption is still significantly large and in the order of 1mj. Similar to energy consumption, convergence time of distributed graph routing increases gradually with increasing power level for transmission

9 Energy (mj) Convergence time (s) Memory (B) Transmission power (dbm) Transmission power (dbm) Transmission power (dbm) (b) Convergence time (c) Memory consumption Figure 9: Performance under varying transmission power levels Energy (mj) % of link failures Convergence time (s) % of link failures # of broken routes % of link failures (b) Convergence time (c) Relibility comparison Figure 1: Performance under varying percentage of link failures due to increasing number of edges in the network as shown in Fig. 9(b). For centralized algorithms, convergence time decreases as the number of messages decrease. This decrease is minimal as time required to collect the topology and compute the routes at the network manager is constant. These results conclude that the effect of network density on distributed algorithm is negligible and similar performance can be observed from centralized algorithms. The effect of varying node density on memory is shown in Fig. 9(c). As expected, the memory consumption of the distributed algorithm increases linearly with power level due to increasing the number of neighbors of nodes however this increase is very small Performance under Link Failures. The proposed distributed algorithm is naturally adaptive to network dynamics. For example, when links break, nodes recalculate routing tables and updates neighbors. Similarly, under volatile link conditions, deping on duration of failures, either optimal routes have to be re-computed every time a link becomes bad/good or existing routes have to be used. In this simulation, we evaluate the performance of our distributed algorithm under link failures in terms of additional energy required to compute optimal routes, convergence time required and number of broken routes when algorithm is not executed. We consider randomly link failures in the network while assuring the network is connected at all times. The effect of percentage of link failure on number of broken routes is shown in Fig. 1(c). As expected for centralized algorithm, increase in link failures results in larger number of disconnected path and disconnected number of control loops. This causes the steep increase in number of broken routes as number of link failures increase. Since distributed algorithm generates all possible paths from a source to destination, the number of broken routes does not increase by more than 1% of the total routes. These results show 9 Energy(J) Increase in area size Figure 11: Performance under varying area size that under link failure, distributed algorithms performs better in terms of reliability by offering multiple paths as back-up paths. Fig. 1(a) shows the additional energy consumption at each node to generate optimal graph routes. Additional energy consumption is proportional to the number of paths that are broken in the network. For centralized algorithms, this number is high for even a 1% of link failures and keeps increasing. This causes the steep increase in additional energy consumption. However for distributed, additional energy required was very less as the number of updates in link quality is less when compared to centralized. This causes the small value of energy consumption for distributed algorithm. We have observed a minimum of 8% of energy saving in our distributed algorithm compared to the centralized ones. Similar to energy, convergence time also deps on the number of broken paths and this value is high for centralized algorithms as shown in Fig. 1(b). We observed a minimum of 8% saving in convergence time Performance under Varying Area Size. We evaluated the performance of distributed graph routing under varying density for the same number of nodes by increasing the area. We used random placement in areas ranging from times of the original area