Evaluation of Ethernet based Control Network used in the Distributed Control System

Transcription

1 Proceedings on the 15th CISL Winter Workshop Kushu, Japan February 2002 Evaluation of Ethernet based Control Network used in the Distributed Control System Jae Young Choi Control Information Systems Lab., School of Electrical Engr. and Computer Science, Seoul National University, Seoul, , Korea Abstract: Distributed control system(dcs) has been used widely in industrial environments. This paper proposes new network system named ERCNet(Ethernet based Real-time Control Network), which satisfies all strict requirements of industrial environment like nuclear power plant. The proposed network, ERCNet uses ring topology, Token passing mechanism and physical media of Fast Ethernet. Proposed network has newly invented device, TCS(Topology Conversion Switch) to adopt ring topology using Fast Ethernet physical media and to enhance the network performance. In this paper, numerical analysis and performance evaluations about this network will be done. Performance enhancement effects of TCS are evaluated by experiments. Keywords: Distributed control system(dcs); Control network; Ring topology; Topology Conversion Switch(TCS); Medium Access Control(MAC) 1 Introduction As digital automation technologies became available in the 1960s, they were used to improve and enlarge the industrial automation systems. The concepts of distributed computer control systems were introduced in the field of industrial automation, and the use of the communication networks has had an almost explosive growth.[1, 2] As the industrial automation systems become large and the number of automation devices increase, it becomes very important for industrial automation to provide high performance network which make it possible to communicate many automation devices and stations in a rapid way. Although there are many network protocols to apply to the industrial network of DCS, they have several their own defects not enough to be applied to high performance DCS like current nuclear power plant. Hence, this paper can be the reference for high performance DCS network architecture. In this paper, a new communication network is proposed and evaluated to fit to the current trend of high performance DCS(Distributed control system). This is named as Ethernet based Real-time Control Network(ERCNet). ERCNet is developed to be able to be implemented as a control network of high performance DCS especially in nuclear power plant. ERCNet can be instrumented in DCS network of nuclear power plant digital I&C system. Today, distributed control systems are being developed using commercial-off-the-shelf (COTS) network products to reduce product development cycle and cost and achieve system interoperability.[4, 5] In the process control industry, Ethernet is becoming a de facto choice for open control network strategies. In accordance with this trend and for the sake of maintainability, ERCNet uses fiber optic physical media of 100Mbps Fast Ethernet. And ERCNet has ring topology with token passing mechanism. To overcome the large broadcasting delay that is one of the major defect of ring topology network, special hardware device that is named TCS(Topology Conversion Switch) is developed and used. And ERC- Net adopts replicated memory DCS system somewhat different from existing replicated memory product like SCRAMNet. The main purpose of this paper is focused on the design and evaluation of DCS network that can satisfy

2 ! \\[[ ZZgffee! jjiihhuttss!! =< ;!! NNMMLLYX > > KJII! $ $ ##"" /. - the tight requirements of industrial environment, such as nuclear power plant. This paper proposes and evaluates the new design and method of high performance DCS network. This paper is organized as follows.in Section 2, the concept and architecture of ERCNet will be described. Section 3 shows the performance enhancement of ERC- Net by implementing TCS. In Section 4, experimental result about the reduction of delay time as the number of nodes is suggested. Finally, conclusions and summarization are made in Section 5. 2 Architecture of ERCNet Hub sensor Motor Valve! ]^_`abcd g! klmnopqr u!! :; <=!! OPQRSTUVW XY ABCDEFGH JK Information Network Gateway Control Network Field Network! %&'()*+,-./ ERCNet is high reliability network that can be used as protection or control instrumentations. ERCNet should be faithful to communicate signal data to control instruments fast and correctly. For this reason, very high speed data exchange and storage should be achieved. ERCNet must consider reliable and swift data exchange mechanism. Also, ERCNet must take into account maintenance. Mostly, DCS is composed of many components. The main components of the DCS are controllers, input/output modules, communication network, sensors, actuators, operator interface stations (OIS), and engineer interface stations (EIS). Among these components, communication network is one of the most important components because all of components are connected by communication network.[6] Generally, communication network in DCS is designed hierarchically. Most DCS communication network is designed as 3 levels, that are information network, control network, and field network like Figure 1. Information network interfaces with users and gets data from control network. Control network transfers data to information network and gathers data from field network. Field network collects data from input/output modules and transfers it to control network. ERCNet corresponds to control network. ERCNet adopts ring topology for the ease of maintainability. Also, it uses Fast Ethernet media, which is used widely and its performance is verified for a long time, to enhance the reliability. It uses fiber-optic cables to reduce the error rate of lines. There are more features in ERCNet. Many features of ERCNet is covered below. 2.1 Replicated Memory method DCS Many DCS system uses distributed shared memory (DSM) scheme for its simplicity and flexibility in software programming. There are two currently existing representative DSM models, that is shared mem- Figure 1: General Architecture of DCS ory and replicated memory.[7] First, typical writeinvalidate protocol model works as follows: Each node has several copies of shared pages, and when a node comes to a pagefault, it sends a request for that page to the other nodes, who send back the page contents through the network. If a node already has a copy, or if it has received a copy form another node, it can access the copied page just the same way. As it accesses pages of normal machine memory(which we will call local memory). When a node tries to write to a page of shared memory, on the other hand, it sends a message to the other nodes in order to invalidate copies of the same page. Second, characteristics of replicated shared memory(called broadcast memory with FIFO)[8, 9] are considered to be almost opposite to those of shared memory. Replicated memory works as follows: Each node has its own copy of the entire contents of shared memory stored in a so-called global memory. A read access to shared memory is performed by reading the contents of its own global memory. When a write access to shared memory is performed, the written data is broadcast to all other nodes on the network to update their own copies. The written data is first stored in a FIFO buffer and sent to the network late, so that the node can resume its work without waiting for acquisition of the network. ERCNet uses replicated memory method network. In shared memory method, there is delay overhead at each memory access of every node and this is a bottleneck of high speed network. To reduce this bottleneck, cache can be used. But in case of cache miss, there comes serious problems. ERCNet uses new replicated memory method DCS and specific behavior is described as follows. All data is updated using broadcasting method with token pass mechanism. Basically, because every node

3 should contain the information of all node, broadcasting is efficient as a data transfer method. When each node receives token, this station(ts) broadcasts its own data. Token is transferred to the next station(ns) to make the next station(ns) broadcast its own data after all of data of TS is transmitted. All of data is updated like this scheme. These procedures can be seen like Figure 2. _`/Z4 _`/Z4 _`/Z4 ; / _`/Z4 ; _`/Z4 : _`/Z4 : _`/Z/ _`/Z D v wyx z { ~ w z Q 2.2 Token D D D E v wyx z {} ~ w z E E E D E v w x z ~ w z {} v w x z {} ~ w z ƒ ˆ ƒ ˆŠŒ ŽŒ Œy y Œ ŒŠ ƒ Figure 2: Data Transmission Process ERCNet uses general Fast Ethernet frames to enhance maintainability. To express token frame, ERCNet uses Type field of Fast Ethernet frame format. The frame format of ERCNet is shown in Figure 3. The first bit of Type field of Ethernet frame is named identifier. When identifier bit is set, it means this frame is token frame.and when identifier is cleared, it means this frame is general data frame. This Type Field is remained for network layer protocols by IEEE standards. /F_ZZc : /Y Z ` \ _ / _ /F _ Z Z c /Y Z /Z /5 / _ c X ] / X X / _ /Z /Z / ] X c 7 a8i _ ỹ šœ œ yž X c / c X ^ / a8 M c X ^ Y ] c X ^ / Z _ X Z ` _ / ` c / X X \ / I _ ] Z ^ Z c D c D c b _ /E : /E /E /E ; /E /E Figure 3: Token frame of ERCNet Since ERCNet doesn t have need to use network layer, this Type Field is not used in ERCNet. ER- CNet contains application layer just on the data link layer. Therefore ERCNet can make use of this field : Z^ freely. ERCNet understands token frame if identifier is set or data frame if identifier is not set. 2.3 TCS In the ring topology network, delays are generated at each node because each node must process every data frame whether frame is destined to itself or not. It is fundamental defects of ring topology network. TCS functions as reducing this delays. Because TCS connects reception line and transmission line in all reception nodes, received data are transmitted immediately and transferred to CPU simultaneously. So TCS can reduce the delay at each node. This scheme reduces CPU processing time at each node. When node receives token and wants to transmit its data, TCS controller disconnects the connection between transmission line and reception line and transmits its data. 100Base-FX Fast Ethernet network should use star topology. However, ERCNet adopts ring topology though physical media of 100Base-FX is used. TCS(Topology Conversion Switch) is the device that makes ERCNet have ring topology with physical media of 100Base-FX. TCS changes topology of network in a moment when station wants to transmit its data. At the moment station want to transmit data, TCS disconnects reception line from transmission line. In this situation, all network can be seen as bus topology, and transmission node broadcasts its data. Figure 4 is block diagram of TCS. Also, TCS connects reception line and transmission line in all the other reception nodes. Using these devices, ERCNet can use Ethernet physical media without hub or switch. ª ««ª OZ : 2 ²y ¹ ½ ² ± Ó ³y ± ± Ø LK L ± ² ³µ ¹ º» ³ ¼ ½ ¹ Ì Íy«yÎÃ Ï Ð ÑyÒ À ÁàÄÆÅ Ç È É Ê Ê Ê Ë Ì Í «Îà ÏÖÐ ÑyÒ À ÁàÄÔÅ Ç È É Ê Ê Õ Ë ¾ ¾ Ÿ Ÿ y ª ««ª c X _ ^ 9 2 Ì Í «¹ ½ ± ± Ø ¾ Z ²y ²y± Î Ï Ð ÑyÒ Ó ³ L LK ± ² ³ÃÓ ¹ ºy» ³ ½ ¹ Ì Í «yîã Ï Ð ÑyÒ OZ/ ^ ` c X _ ^ Z / ^ ` Figure 4: Block Diagram of TCS TCS transmits received frames just after switching analog signal not converting digital signal. It is much faster than converting digital signal. This data transition process can be appeared like Figure 5. In Figure 5, CPM means communication processing module in CPU(Central Processing Unit). Analog switch used to retransmit received data is ADG711 by analog device company. This device is often used in the Ethernet ¾ Ÿ Ÿ

4 switch or hub. This device can be switching at 200MHz bandwidth. Owing to this device, TCS can use ethernet physical media with ring topology. ð ñyò MJ/////////// ñ ó ô õ ö ó ø ö c_/cz c ZZX] ð ñyò MJ/////////// ñ ó ô õ ö ó ø ö c_/cz c The developed hardware in the experiment was a VME 6U board with MPC8260 CPU(Central Processing Unit). Operating system was not installed and all program codes were written in C language. It uses GNU compiler(gcc) and 66Mhz clock. The MPC8260 is integrated communication microprocessor designed for the telecommunications and networking devices. Ù Ú Ú Û Ü Ý Þ ß à á âyã ä Ý å æ Ü ç è Þ é å Ü ê ë Ü ì Ý è å í í Þ ë D:: Ù Ú Ú ÛŒÜ Ý Þ ß à á â ã ä Ý å æ Ü ç è Þ é å Ü î Þ æ Þ å ï Þ ë `_c`]/ Z_X] D _ X]` Ù Ú ÚyÛ Ü Ý Þ ßŒà á âyã ä Ý å æ Ü ç è Þ éyå Ü ê ë Ü ì Ý è å í í Þ ë D:: Ù Ú Ú Û Ü Ý Þ ß à á â ã ä Ý å æ Ü ç è Þ é å Ü î Þ æ Þ å ï Þ ë `_c`]/ Z_X] 3.3 Analysis This section describes the analysis of ERCNet. Numerical modeling of ERCNet is proposed and performance difference in case of using TCS or not is also suggested. Parameters used in this model are like this : c X _ ^ Z /J` O Z /J` Amount of packets assigned to node i : Q i (bytes) Figure 5: Signal Transition Diagram of TCS TCS controller is needed to make TCS turn analog switch on and off as shown in Figure 4. This TCS controller is implemented in the communication processing module of MPC8260 CPU. TCS controller generates 1 or 0 using CPU ports. 1 is generated when data is received, 0 is generated when data is transmitted. This signal connects and disconnects transmission line and reception line. 3 Performance Evaluation In this chapter, performance enhancement features of TCS in ERCNet is evaluated by experiments. Also, numerical analysis about the architecture of ERCNet is done. 3.1 Evaluation Properties ERCNet contrives to enhance the performance using TCS. In a general ring network, one node of network receives data and copies it to the local memory and then retransmits it to the next node. But in ERCNet, receiving and retransmitting data occurs simultaneously using TCS. In this chapter, reduction of delay by this architecture is experimented. The variance of delay as nodes increase is shown. Test software is compiled by GNU compiler(gcc) and monitored by serial communication. Many functions like token mechanism or data transmission is developed. 3.2 Experiment Environment All experiments are performed on a testbed network consisting of three MPC8260 boards connected together with fiber optic cable. All boards communicate each other using fiber optic cables. Transmission speed : 100Mbits/sec (Fast Ethernet) One-bit time : T b The number of nodes : N Authorization rotating time : T rot Propagation delay : P i (5ns/m in case of 100Base- FX) Header size (OH) : 22 bytes Length of CRC (CRC) : 4 bytes Data frame payload length : F d ( 46 bytes F d 1.5 Kbytes) Optical-to-electrical conversion delay : D oe Electrical-to-optical conversion delay : D eo TCS processing delay : D tcs Cable length between node i and i+1 : L i (L max = 2000 m) Authorization processing delay : D A Interframe gap : D g The number of packets that the node i sends : m i The number of frames assigned to i-th node m i is m i = Qi The payload length of the last frame of i-th node r i is Fd r i = Q i Fd(m i 1) The transmission time of the last frame T f is

5 Tf = (r i +OH + CRC) Tb The transmission time of the usual frame T n is Tn = (Fd +OH + CRC) Tb The authorization passing time of i-th node A i is Ai = (Tn + Dg)(m i 1) + Tf +(Deo + Pi + Doe) + DA + Dtcs where P i = 5L i (5 us/km) in case that the optical fiber is 100Base-FX[10] The authorization rotating time in the whole network T rot is Trot = n {(Tn + Dg)(m i 1) + Tf +(Deo + Pi + Doe) i=1 + DA + Dtcs} The authorization passing time of i-th node A i means processing time to acknowledge the fact that token is received and to transmit its own data. And authorization rotating time means time that token passes through the whole network. In experiments, D tcs term is removed and D A term is increased in case of not using TCS. If TCS is not used, CPU itself has to copy data from receive buffer to transmit buffer. This increases D A value. If TCS is used, D A value is decreased and D tcs term exists. But D tcs is so small because D tcs is just composed of analog signal switching delay. 4 Experiment Results This experiment built test network with 2, 3, and 8 nodes and measured frame rotating time in each case. Frame rotation time is defined as the time from sending data frame to the other nodes to receiving its own data frame. In this experiment, 1Kbytes data frame is transmitted and frame rotation time is measured using two 32bit timers provided by MPC8260 CPU. In case that TCS is used, all frames are switched and transmitted soon after received. In case of no TCS, DMA(Direct Memory Access) controller reads receive buffer and copies it to the transmit buffer using DMA. Figure 6 and table 1 show the experimental result. Two frame rotating times, in case of using TCS in ER- CNet and in case of not using TCS and data is copied using DMA are measured as node increases. X-axis means number of node and Y-axis time with microsecond scale. All data was gathered as the average of enough iteration. This result shows that frame rotation time of ERCNet with TCS increases slightly as Table 1: Frame Rotation Time(us) Number of Node TCS Without TCS Figure 6: Experimental Result-Authorization Rotation Time node increases in comparison with the case of not using TCS in ERCNet. In ERCNet, frames are retransmitted just after switching received analog signals. Frame rotation time of ERCNet with TCS can be smaller than data rotation time of FDDI. If TCS is not used in ERCNet, ERCNet shows result worse than the case of using TCS. It is because data retransmission is processed by software. CPU copies data from receive buffer to transmit buffer. It takes much more time than frames are retransmitted by analog switching. Although this experiment shows good result, TCS can make signal distortions because TCS switches analog signal directly. So it is probable that signals should be corrected and amplified appropriately in case that node increases. 5 Conclusion In industrial environment, the amount of data to be processed is increasing continuously and users want more convenience. Considering these situation, existing network protocols have some difficulties in operating current distributing control system. It is necessary that higher performance network should be developed. As a high performance network for these requirements, ER-

6 CNet is proposed in this paper. ERCNet is developed gathering the advantages of existing high speed networks, and it has enough performance to satisfy strict requirements of industrial environment. ERCNet is high performance network with high maintainability and speed as much as 100Mbps. Maintenance and repair is easy because it uses widely-used Fast Ethernet physical media. Wiring is simple because it uses ring topology. In general, ethernet network is implemented as star topology. But TCS device is developed to be capable of ring topology based upon ethernet physical media. Also, TCS have an effect on reducing time delay at each node, that is one of the most significant defects of ring topology network. We make network behave deterministically using token passing mechanism. Token passing mechanism is implemented by means of making use of unused field of Ethernet frame format. Performance enhancement effect of TCS in ERCNet is measured by experiment. This experiment shows that frame rotation time is slightly increased as node increases in comparison with not using TCS. It is because the time CPU copies data from receive buffer to transmit buffer is reduced. TCS retransmits frames just after switching analog signal that comes from 100Base-FX receiver. At this moment, distortion of signal may happen. Effect of distortion as number of node is not yet evaluated. Experimental evaluation about the signal distortion by TCS is needed. ERCNet can be used as the network system demanded by distributed control system. It is designed to satisfy strict requirements that most of distributed control system needs. Especially it is developed to meet the strict requirements like nuclear power plant. ER- CNet is planning to be used as DCS network in the nuclear power plant. Also, it can be adapted to almost industrial environment. evaluation, IEEE International Performance, Computing and Communications Conference, pp , Feb [5] A. Koyama, L. Barolli, and Z. Cheng, Performance evaluation of an adaptive ring access protocol for super high-speed networks, Preceedings of 15th International Conference on Information Networking, pp , [6] J. Cao, R. Steele, L.J. Yao, and W. Jia, Design and simulation of a reliable token-ring protocol for realtime communications, Proceedings of IEEE International Computer Performance and Dependability Symposium, pp , [7] M. Oguchi, H. Aida, and T. Saito, A proposal for a DSM architecture suitable for a widely distributed environment and its evaluation, Proceedings of the Fourth IEEE International Symposium on High Performance Distributed Computing, pp , [8] T. Saito, H. Aida, Oguchi M., and M. Kuroiwa, Implementation and Evaluation of a Shared Memory Using Broadcast Memory with FIFO, Transactions of the Institute of Electronics, Information and Communication Engineers, vol. J75- D-I, no. 12, pp , December [9] Oguchi M., H. Aida, and T. Saito, A Study of Synchronization Mechanisms for Ring-Connected type Replicated Shared Memory, IFIP Conference on South East Asia Communications, pp , October [10] B. Albert and A.P. Jayasumana, Performance analysis of FDDI LANs using numerical methods, IEE Proceedings-Computers and Digital Techniques, vol. 144, no. 3, pp , May References [1] Hong ju Moon, Performance analysis and design of a communication network for industrial automation : a case study of the Mini-Map network, Ph. D Thesis, Seoul National University, [2] John R. Freer, Computer Communications and Networks, Plenum Press, [3] John W. Bernard, CIM in the Process Industries, Instrument Society of America, [4] J. Huang, S. Song, L. Li, P. Kappler, R. Freimark, J. Gustin, and T. Kozlik, An open solution to fault-tolerant Ethernet: design, prototyping, and