Presented at the ODVA 2011 ODVA Industry Conference & 14 th Annual Meeting March 1-3, 2011 Phoenix, Arizona, USA

Transcription

1 Minimizing Contributions to System Delay from NIC Packet Processing in CIP Motion and Other Real-Time Control Applications By Jordon Woods Chief Technical Officer, Innovasic Semiconductor Presented at the ODVA 2011 ODVA Industry Conference & 14 th Annual Meeting March 1-3, 2011 Phoenix, Arizona, USA Abstract Motion control is one of the most demanding applications of Real-Time control within factory automation. Performing complex motion control using standard Ethernet infrastructure requires that packet latency and jitter be minimized across the network. CIP motion technologies deliver the promise of time-synchronized, multi-axis, distributed control over a standard Ethernet infrastructure. Among the challenges to such applications is the requirement for high-speed, deterministic updates from all axes on a CIP motion network. The total number of axes supported on such a network is constrained by the ability of the nodes or drives on the network to provide input data and the ability of the controller to plan, coordinate and communicate updates to networks drives during the scheduled update interval, a period on the order of 1 ms with messaging jitter less than 1 us. Therefore, any contribution to latency and jitter within the network must be minimized to optimize performance and increase the maximum number of axes on the network. This paper proposes a software-based architecture designed to minimize the contribution of the NIC to jitter and latency. This architecture can be implemented on any standard microprocessor. This paper also proposes enhancements to the hardware architecture of the NIC which provides additional QoS discrimination and minimizes contributions to latency and jitter associated with Real-Time Operating Systems (RTOS) and Media Access Controller (MAC). Finally, this paper will offer results from a proof of concept experiment which implements the proposed architecture and can readily be applied to Real-Time EtherNet/IP and CIP motion applications. Introduction At the 2009 ODVA annual meeting, Mark Chaffee presented a number of considerations for implementation of CIP motion control. In particular, the concept of a Network Interface Controller (NIC) as the means to connect the Automation Controller, CIP enabled Motion Drives and other network components was defined. The NIC typically consists of a CPU to run the stack, a MAC to manage the Ethernet media and a PHY to provide the hardware interface. With that definition in mind, the maximum number of axes that can be controller on a given CIP motion control network can be calculated as follows: Max Axis = 1 + {1/3 * Connection Update Period (Drive Transmission Delay + (m + 1) * Ethernet Transmission Time + m * Switch Latency + NIC Packet Processing Delay + Bus Interface Delay)}/NIC Packet Processing Delay (Where m = # of switches) As the above equation makes clear, any system delay has the potential to degrade the performance of the system. Quality of Service (QoS), as outlined in Volume II: EtherNet/IP Adaptation of CIP, chapter 9, section 9-5, provides a means to minimize contributions to system delay due to switch latency. Cost-effective switch technology supporting these recommendations is readily available. However, few offerings address the issues of packet processing delay in the Network Interface Controller (NIC) which provides communication between drives and the EtherNet/IP network. Instead, industrial control equipment manufacturers typically develop their own nodes using 2011 ODVA Industry Conference ODVA, Inc.

2 standard architectural approaches. These architectures often introduce latency and jitter which can be the major factors in limiting the determinism of the network. As embedded switch technology become more common in drives and other end devices the need to minimize contributions to latency and jitter in these devices become paramount. These issues can be easily demonstrated with the simple test setup shown in figure 1. A simple NIC card is connected to a PC acting as a scanner through a commercially available switch. The scanner application establishes a class 1 connection with a requested packet interval (RPI) of 1 ms. The NIC consumes these requests and produces a response at the same packet interval. A separate PC running Wireshark collects packet from the NIC data for later analysis. There is no additional traffic on the network, so this is effectively point-to-point communication. Figure 1 - Test Setup for measuring Packet Response Interval Using the IENetP performance analysis tool to post-process the packet data gathered by Wireshark yields the plot shown in figure 2. While the majority of NIC responses are very close to the RPI, there are significant variations. Jitter on the order of 100% is evident. Clearly, such variations, introduced at each NIC in a CIP motion network would severely limit the performance of said network ODVA Industry Conference ODVA, Inc.

3 Figure 2 - Performance Analysis Results For these reason, it is clear that an End-to-End QoS scheme, as proposed by Chaffee, should be implemented throughout a CIP motion network. In other words, QoS must be implemented not just in the network infrastructure, but in the NIC itself. Sources of Latency and Jitter: Why do such substantial variations exist? Consider a typical NIC such as that shown in Figure 1. Functions allocated to hardware are shown in gray while those allocated to software are depicted in white ODVA Industry Conference ODVA, Inc.

4 Figure 3 - Block diagram of a typical NIC The first and most obvious problem is the Ethernet MAC. With no discrimination between high and low priority Ethernet packets, all packet processing must be handled sequentially. Incoming packets arriving at the MAC are first checked to make sure that they are destined for this device. They are then buffered in memory to allow the CPU to process the packets at its processing speed. Imagine a scenario where a NIC is currently processing a low priority packet when a high priority, time-critical packet arrives on the wire. The high priority packet arrives at the MAC and is then placed into buffer memory. Since it is a high priority packet, ideally the CPU would immediately stop processing the low-priority packet and start processing the high priority data. However, in the conventional architecture shown in Figure 3, the CPU cannot switch over to process this packet until it has finished processing the entire low priority packet through the non re-entrant TCP/IP stack. If the low priority packet happened to be a very long packet, the delay could cause serious problems particularly in hard real-time control systems such as closedloop motion control. In addition, a Real-Time Operating System (RTOS) provides task scheduling, task switching, time management and memory management support. The RTOS introduces several sources of uncertainty. The RTOS constantly switches between various threads (tasks), as it shares the microcontroller CPU and resources across those tasks. For time critical applications, the variability of switching between tasks, and the inability to switch during processing of nonreentrant tasks (such as a TCP/IP stack) can cause latency to vary widely. Of course, the processor is likely not dedicated to simply processing Ethernet packet. Application software is needed to implement our motion control application or to transfer data to an upstream host processor. Such an application may disable interrupts as it carries out its task, effectively preventing the CPU from recognizing the arrival of Ethernet traffic. Depending on the task at hand, this delay may vary widely, introducing yet another source of uncertainty. Testing of a system running a standard TCP/IP stack, a standard EtherNet/IP stack and a simple application showed that the CPU spent more than 5% of its available execution time simply enabling and disabling interrupts in the stacks ODVA Industry Conference ODVA, Inc.

5 Quality of Service As shown, an immediate improvement can be made to this architecture by simply adding a software layer to sort packets based upon QoS. The architecture assumes that processor has adequate DMA and memory resources to add an additional data buffer to support queuing of high and low priority packets. Any time-critical packets that arrive at the NIC are transferred to the high priority buffer and the microcontroller is interrupted so that it can begin processing these packets as soon as possible. Of course, this approach does not address issues of uncertainty due to the RTOS, stack sharing or application blocking. Furthermore, a software-based approach to QoS filtering is not as efficient as the hardware-based approach utilized by most switches. Figure 4 - Embedded NIC with QoS functionality The basic problem with software-based QoS filtering is that, ultimately, the QoS sorter must share CPU resources like any other piece of software. If the programmer chooses to mask interrupts when servicing other tasks, then application blocking can still take place. If interrupts are enabled, a high priority packet can be interrupted by the arrival of a low priority packet, if only to determine that the arriving packet is indeed low priority. Each time an interrupt is received the RTOS must complete the current instruction, save the current state and switch to the new task. Silicon-based QoS filtering A preferable approach would include hardware to examine the TOS or DSCP header and automatically sort packets into high and low priority buffers. Most commercially available switch products include silicon-based QoS filtering and separate buffers; however, few MACs integrated with microcontrollers do so. With this scheme the processor need only be interrupted when an entire packet has been received. The entire process of sorting packets occurs without processor intervention ODVA Industry Conference ODVA, Inc.

6 Figure 5 - Embedded NIC with silicon-based QoS filtering Custom or programmable hardware can provide additional filtering of Ethernet packets. For instance, IEEE-1588 time stamp packets are used by all nodes for time-synchronization (CIPSync ). By necessity, these packets are tagged with the highest available QoS priority so they will propagate through the Layer 2 infrastructure quickly. However, once these packets are received by the NIC and properly time-stamped by hardware, they need not be processed by the CPU with the same urgency. In the case of EtherNet/IP, we would know that the most critical time sensitive data for any given node would be a UDP packet targeted to a specific Port Number. By sorting on this criterion in addition to the high priority QoS tag, we would restrict the high priority data buffer to handling only these critical packets with minimal delay. This Layer 3 QoS discrimination can easily be performed in silicon along with the conventional QoS tag filtering. Stack Sharing The architecture in Figure 5 shows the high and low traffic sharing a single TCP/IP and EtherNet/IP stack. Typically, such stacks are non-reentrant. Reentrant code allows multiple simultaneous, interleaved, or nested invocations which will not interfere with each other. Although you can interrupt the execution of a non-reentrant stack, you cannot start processing a new packet until the stack has completed processing the current packet. So if the stack begins to process a low priority packet, any new high priority packet that arrives at the node has to wait until the low priority packet has been processed by the stack ODVA Industry Conference ODVA, Inc.

7 Figure 6 - Embedded NIC with separated stacks A software-based architecture, such as that shown in Figure 6, supporting separate stacks for high and low priority traffic addresses this uncertainty. Further, since the high priority stack need only process those packets sorted by the Layer 3 QoS discriminator, this stack can be optimized, further improving processing efficiency. Application Blocking and RTOS Context Switching The uncertainty introduced by the RTOS and task-switching is more difficult to solve. Typically, an RTOS is a complex, third-party piece of software design for generic applications. Optimizing the RTOS for stringent requirements of high-performance networking represents a significant investment on the part of the designer, even assuming he has access to the RTOS source code. However, the contributions to latency and jitter due to task switching can be minimized by shifting some of the basic functions of an RTOS from a software-based architecture to a hardware-based architecture. Consider the architecture depicted in Figure -7. Several of the functions normally associated with an RTOS and now controlled in the hardware blocks called Context Control and Interrupt Control including: Preemptive Schedule Time Partitioning/Management Memory Management Time management Interrupt Control 2011 ODVA Industry Conference ODVA, Inc.

8 Figure 7 - Embedded NIC with silicon controlled context switching The system includes separate memory, stack and register resources for a number of tasks. These separate hardwarecontexts preserve the individual state of a given task and obviate the need to push state information onto the stack when switch between hardware contexts. In the case of a CIP motion NIC, a hardware context would be required for: The High Priority Stack The Low Priority Stack Any Application Software Master Control/Supervisor This architecture enjoys several benefits. First, it is now possible to physically separate the processing of high- and low-priority packet streams and to switch between them in a single clock cycle (Figure 4.). This single cycle, predictable context switching becomes important for highly deterministic real-time systems. Even non-re-entrant TCP/IP stack processing can be handled by this approach with single cycle switching because the context state is automatically preserved. This architecture provides highly deterministic response with consistently low jitter regardless of the incoming network traffic. Second, with interrupt handling under hardware control, it is now possible to implement a unified priority scheme across all processor resources including DMA. Such a model ensures that the highest priority task will always get the processor resources, effectively eliminating application blocking. In addition, by implementing hardware control of memory management and thread timing, we ve ensured that no thread will corrupt the memory resources of another and that no context will overrun its allotted execution time. Finally, these features taken together provide unparalleled fault tolerance in a single processor. Any attempt by a given thread to access the resources of another or to overrun its execution time will result in a fault. The unified 2011 ODVA Industry Conference ODVA, Inc.

9 priority scheme ensures that the supervisor context gets the processor resources in the event of a fault. The supervisor context can then be programmed to handle the fault in accordance with system requirements (i.e. send an alert; ensure high-priority traffic continues to be processed; etc.). Proof of Concept 1 An independent assessment compared the jitter tolerance of this architecture against that of a standard NIC. As shown in Figure 8, each NIC was placed on a network. For the purposes of this demonstration, standard Ethernet was used for all traffic. High and low priority packets were distinguished by UDP type. A PC was used to generate high and low priority packets. In addition, a traffic generator produced background traffic to further burden NIC resources. Each NIC was programmed to receive the UDP packets and then turn around and retransmit them back to the PC. Turnaround time of high priority UDP packets (the time required to receive and resend high priority packets in the presence of other network traffic) was measured. The difference between minimum and maximum turnaround times provides a measure of the jitter associated with each NIC. Figure 8 - Test configuration The results are shown in Figure 9. Note that the node jitter in the silicon controlled architecture is nearly 7x better than with the conventional software and RTOS controlled architecture. In addition, as shown in Figure 10, it was possible to cause the conventional architecture to experience a stack pointer violation that generated a fatal fault which completely crashed the application. However, in the case of the silicon architecture, when the LOW priority stack and application crashed, the HIGH priority packets could continue to be processed with no adverse affect on jitter. In this situation, it was even possible to gracefully recover by automatically restarting the LOW priority stack and application, again with no impact on HIGH priority packet jitter. This feature provides the ability to design an intrinsically secure operating mode for HIGH priority packets even under the most adverse operating conditions ODVA Industry Conference ODVA, Inc.

10 Figure 9 - Test results comparing jitter in the two nodes Figure 10 - Test results with fatal fault generated in the LOW priority stack These results clearly demonstrate this architecture s ability to eliminate uncertainty due to application blocking, RTOS switching time, and stack sharing. Notice that the latency of the proposed NIC architecture is also substantially lower than that of the standard NIC. However, this experiment does not address the ability of the architecture to scale in order handle multiple or, in the case of motion control, axes ODVA Industry Conference ODVA, Inc.

11 Proof of Concept 2 To address the concept of scalability, consider the system depicted in Figure 11. Programmable Automation Controllers (PAC) of this type are available from a variety of suppliers. In general, these systems consist of an intelligent controller, a NIC and a backplane to allow various modules to be added to the system. Such systems involving many modules are of particular interest, because the pushing of data from to the stack and retrieval of data from the stack are very time consuming processes. Modules NIC Backplane slots for up to 32 additional Modules Figure 11 - Standard Industrial PAC with Multiple/Scalable Slots In such systems it is necessary to minimize the pushing data onto the stack and retrieving data from the stack. While the NIC architecture proposed herein provides many advantages, it is possible to minimize these advantages through inefficient application or stack architecture. Context switching within the path must be minimized. Innovasic Semiconductor has been working in partnership with customers to optimize such systems utilizing the proposed NIC architecture. For instance, one system utilizing yet another popular Industrial Ethernet protocol, in an application with a single, 8-bit Input module and a single 8-bit output module yielded the following performance: o With the stack in its original configuration, the setting/getting of data took approximately 400 us (or about 40% of a 1 ms frame). o Following optimization to minimize context switching as described above, the output path required approximately 11.3 us and the input path approximately 6.6 us. While this was a substantial improvement, for a device with 32 modules you still have approximately 627 us of overhead (or 62.7% of processor bandwidth on a 1 ms frame). Keep in mind that other parts of the system (scheduler, receive path, transmission path, etc.) are using approximately 530 us (or 53%) ODVA Industry Conference ODVA, Inc.

12 Clearly, more optimization was needed. This particular stack implementation validated an connection each time data was exchanged. By performing this validation only upon initial establishment of a connection, overhead was reduced to approximately.31 us per iteration for output data and approximately 0.18 us for input data (or 16.7% of processor bandwidth on a 1 ms frame). While this type of problem does not exist in the EtherNet/IP stack we ve evaluated, it is demonstrative of the type of inefficiency that can be introduced by the stack architecture. Some further optimization along this line is possible (though not yet pursued) by moving some of the stack's data structures to internal RAM. An initial estimate of this approach indicates that we could reduce the 16.7% overhead described above to 7%-10%. Our next step was to improve QoS discrimination for the high-priority queue. While this particular protocol does not utilize standard IEEE 802.1D/802.1Q QoS, it does distinguish between high and low priority traffic by Ethertype. By simply filtering those Ethertypes which require minimal latency through the layer 2 infrastructure, but which do not immediately require CPU resources, we are able to further reduce the burden on the high priority queue. At the time of this writing, this optimization is not yet operational so no measured values are available. An estimate based on timing analysis indicates that such optimizations would save 170 us per iteration on the receive path, or 17% of throughput for a 1 ms frame time. If the estimate is accurate, we should see the total overhead for a 1 ms frame go from 53% (47% available for and the user) to 36% (64% available for and the user). The optimizations describe above are targeted for a real-world customer system. We expect this system to be fielded in the next year. Our investigation indicates that it is possible to further reduce the cycle overhead to approximately 100 us or less, leaving 80% of the frame for user with a comfortable margin for processing lower priority tasks. This approach would also provide the possibility of raising the maximum cycle speed to 500 us or, for a subset of applications, 250 us. Application to EtherNet/IP Innovasic Semiconductor believes that the architectures and optimizations described herein have direct applicability to EtherNet/IP. However, as of this writing, we have not implemented this architecture with an EtherNet/IP stack. Why? Several challenges have presented themselves: The implementation of the QoS object is a comparatively recent addition to the EtherNet/I/P standard. The EtherNet/IP stack we ve evaluated, even those that implement the QoS object, are single threaded and non-reentrant, making it architecturally challenging to create a high priority data path through the stack. Further work is needed to identify the mechanism for additional QoS discrimination (Port number, Ethertype, etc.) Despite these challenges, we remain convinced that EtherNet/IP is ideally suited for this architecture. The EtherNet/IP protocol provides several advantages with respect to this architecture, particularly for motion control applications: An approach for time-synchronized, multi-axis, distributed control over a standard Ethernet infrastructure (CIP motion, CIP sync) An approach for rapid ring recovery Object-based connections 2011 ODVA Industry Conference ODVA, Inc.

13 References: Chaffee, Mark. "CIP Motion Implementation Considerations." Proc. of ODVA 2009 Conference & 13th Annual Meeting, Howey-in-the-Hills, Florida USA. Lee, Edward A. "Absolutely Positively on Time: What Would It Take?" Computer 38.1 (2005): Prettyjohns, Keith, and David Alsup. "Enhancing Determinism, Fault-Tolerance and Reliability in EtherNet/IP Systems." Proc. of ODVA 2009 Conference & 13th Annual Meeting, Howey-in-the-Hills, Florida USA. ************************************************************************************** The ideas, opinions, and recommendations expressed herein are intended to describe concepts of the author(s) for the possible use of CIP Networks and do not reflect the ideas, opinions, and recommendation of ODVA per se. Because CIP Networks may be applied in many diverse situations and in conjunction with products and systems from multiple vendors, the reader and those responsible for specifying CIP Networks must determine for themselves the suitability and the suitability of ideas, opinions, and recommendations expressed herein for intended use. Copyright 2011 ODVA, Inc. All rights reserved. For permission to reproduce excerpts of this material, with appropriate attribution to the author(s), please contact ODVA on: TEL FAX WEB CIP, Common Industrial Protocol, CIP Motion, CIP Safety, CIP Sync, CompoNet, CompoNet CONFORMANCE TESTED, ControlNet, ControlNet CONFORMANCE TESTED, DeviceNet, EtherNet/IP, EtherNet/IP CONFORMANCE TESTED are trademarks of ODVA, Inc. DeviceNet CONFORMANCE TESTED is a registered trademark of ODVA, Inc. All other trademarks are property of their respective owners ODVA Industry Conference ODVA, Inc.