Making the Switch to RapidIO

Transcription

1 QNX Software Systems Ltd. 175 Terence Matthews Crescent Ottawa, Ontario, Canada, K2M 1W8 Voice: Fax: info@qnx.com Web: Using a Message-passing Microkernel OS to Realize the Full Potential of the RapidIO Interconnect Paul N. Leroux Technology Analyst QNX Software Systems Ltd. paull@qnx.com Introduction Manufacturers of networking equipment have hit a bottleneck. On the one hand, they can now move traffic from one network element to another at phenomenal speeds, using line cards that transmit data at 10 Gigabits per second or higher. But, once inside the box, data moves between boards, processors, and peripherals at a much slower clip: typically a few hundred megabits per second. To break this bottleneck, equipment manufacturers are seeking a new, high-speed and broadly supported interconnect. In fact, many have already set their sights on RapidIO, an open-standard interconnect developed by the RapidIO Trade Association and designed for both chip-to-chip and board-to-board communications. Why RapidIO? Because it offers low latency and extremely high bandwidth, as well as a low pin count and a small silicon footprint a RapidIO interface can easily fit

2 into the corner of a processor, FPGA, or ASIC. RapidIO is also transparent to software, allowing any type of data protocol to run over the interconnect. And, last but not least, RapidIO addresses the demand for reliability by offering built-in error recovery mechanisms and a point-to-point architecture that helps eliminate single points of failure. Of course, other potential interconnect standards exist, including HyperTransport, Infiniband, Fibre Channel, Gigabit/10G Ethernet, and PCI-based fabric interconnects. But RapidIO doesn t seek to supplant any of these. Rather, it is complementary, offering high performance up to 64 Gbit/s per port over the short distances needed to connect processors, memory, and local I/O within a single system. An Emerging Standard RapidIO is gaining momentum. Already, Motorola and IBM have begun to implement RapidIO interfaces on next-generation processors. Meanwhile, Altera has released a RapidIO solution for programmable systems-on-a-chip, and Tundra Semiconductor has announced both switch chips and host bridge chips, including PCI/PCI-X to RapidIO bridge chips for migration from existing designs. RapidIO has also gained acceptance from leading vendors such as Alcatel, Cisco, Ericsson, Lucent, Nokia, and Nortel in fact, most of these companies are active members of the RapidIO trade association. RapidIO itself continues to mature. Just recently, the RapidIO Trade Association released a serial specification that, combined with the existing RapidIO parallel specification, allows systems designers to standardize on a single interconnect technology. For example, semiconductor and ASIC developers bringing serial chips to market can now reuse much of their original RapidIO parallel design. Locked Out by Default: The Problem with Conventional Software Architectures While RapidIO provides a hardware bus that is both fast and reliable, system designers must find or develop software that can fully realize the benefits of its advanced features especially its support for distributed, multiprocessor systems. The problem isn t with RapidIO itself, which is generally softwaretransparent, but with software that can t easily migrate to high-speed distributed designs. Sometimes, the problem lies with application software that is written to be hardware- or protocol-specific. All too often, however, the problem lies not with the application, but with its underlying operating system (OS). For instance, consider what happens whenever an application needs to access a system service, such as a device driver or protocol stack. In most OSs, such services run in the kernel and, as a result, must be accessed by OS kernel calls. Since kernel calls don t cross processor boundaries, these services can be accessed only by applications on the local node. It thus becomes difficult, if not impossible, to distribute an application and its related services across multiple nodes they re effectively locked together on the same processor. Distributed by Design: The Advantage of Microkernel Architecture To make distributed systems much simpler to implement, the QNX Neutrino RTOS uses a message-passing microkernel architecture. In a microkernel OS, only the most fundamental OS primitives (e.g. threads, mutexes, timers) run in the kernel itself. All other services, including drivers, file systems, protocols, and user applications, run outside of the kernel as separate, memory-protected processes. Page 2

3 In this architecture, process are inherently distributed. That s because they can use just one mechanism, synchronous message passing, to communicate with any other application or service, either local or remote. For instance, let s say an application process needs to send a message to a device driver. To do this, it simply issues a POSIX open() call on a symbolic name identifying that driver. The underlying C library will then convert the call into a message, and the OS will route the message to its destination (the driver). If the driver is local, the OS microkernel will route the message directly; if the driver is on another node, an OS service called the QNX micronetwork will transparently forward the message to that node. In effect, it doesn t matter whether the driver is local or remote the application uses the exact same code either way. Consequently, any process can, given appropriate permissions, transparently access virtually any resource on any other node, as if that resource were local. The QNX Neutrino RTOS thus eliminates much of the complexity in building a distributed system, whether that system is based on RapidIO or any other form of interconnect. Moreover, existing software can migrate easily to RapidIO. Because of the clean division of labor provided by microkernel architecture, applications don t effectively care what protocol or media they send messages over it can be an Ethernet LAN today or a RapidIO backplane tomorrow. As indicated, software developers can implement distributed message passing using industry-standard POSIX calls such as open(), read(), and write(). But note that developers can also choose to access the messaging framework directly, using three simple calls: MsgSend(), MsgReceive(), and MsgReply(). Achieving Predictable Response in a Distributed System When implementing remote calls in a distributed system, the software developer typically has to deal with complex synchronization issues and ensure that the calls return in a timely fashion. Remote calls can be especially problematic for a realtime application, MASTER BOARD Qnet POSIX App LINE CARD Qnet Flash Fsys TCP/IP VoIP SNMP Ethernet TCP/IP HA TCP/IP HA Flash Fsys Ethernet Driver Microkernel POSIX App ATM Ethernet Message Passing LINE CARD Microkernel POSIX App TCP/IP VoIP Qnet SS7 ATM Figure 1 In the QNX Neutrino RTOS, the kernel contains only a small set of core primitives. All other system services drivers, file systems, protocol stacks are provided by separate, memory-protected processes that can be stopped and started dynamically. To achieve this modularity, QNX Neutrino uses message passing as the fundamental means of IPC for the entire system. Microkernel POSIX App POSIX App HDLC Figure 2 QNX message passing integrates a network of individual nodes into a single logical machine. As a result, an application can access resources on any other node transparently, without having to invoke remote procedure calls. Page 3

4 since they typically provide no guarantee that requests going to and from other nodes will be processed in priority order. Predictable response times become difficult, if not impossible, to achieve. The synchronous message passing provided by QNX Neutrino helps eliminate these issues. First, it automatically coordinates the execution of cooperating threads and processes. For instance, when process A sends a message to process B, process A will immediately stop running, or become blocked, until it has received a reply from B. This model ensures that the processing performed by B for A is complete before A can resume executing. It also eliminates the need to hand-code and debug synchronization services in either process. QNX message passing also ensures predictable response times in a multi-node system by providing distributed priority inheritance. For instance, if a lowpriority process sends a message to a driver, asking it to perform some work, the receiving thread in the driver will inherit that process s low priority. This ensures that the work done for the low-priority process doesn t execute at a higher priority than it should, thereby preventing a higher-priority process from accessing the CPU (a condition known as priority inversion). The important thing to remember is that, with QNX message passing, this priority RECEIVE blocked This thread Other thread MsgSend() MsgReceive() MsgReply() or MsgError() READY MsgSend() MsgReply() or MsgError() SEND blocked MsgReceive() REPLY blocked inheritance applies whether the message originates from a process on the local node or on another node. This feature is key to ensuring that a system will respond predictably even when its applications and services are distributed across a number of CPUs. In fact, QNX synchronous message passing generally makes designs simpler to implement and easier to maintain: it eliminates the need to keep track of complicated queuing behavior; it makes it easy to map a logical design to an actual implementation; and it fosters better encapsulation and informationhiding by restricting interactions to well-defined, message-based interfaces between processes. 1 Redundant Links for Greater Throughput and Fault Tolerance To enable fast, predictable response times, the RapidIO architecture offers both determinism (achieved through multiple message-priority levels) and low latency. This low latency can be maintained under even heavy network loads, thanks to several features. For instance, if a link is busy, a packet doesn t have to return to the original source device; rather, it simply waits for the link to become available. And if the network load is particularly heavy, RapidIO allows the system designer to implement multiple links that, together, provide extremely high bandwidth. These links can also provide greater fault-tolerance: if one link becomes unavailable, data can be re-routed over one or more of the remaining links. Nonetheless, conventional OS architectures don t offer built-in support for multiple, redundant links between processors whether those links are all based on RapidIO or some combination of RapidIO, fiber, Ethernet, serial, and so on. Consequently, it s up to the software developer to implement support Figure 3 QNX message passing simplifies the synchronization of processes and threads. For instance, the act of sending a message automatically causes the sending thread to be blocked and the receiving thread to be scheduled for execution. 1 Besides message passing, QNX Neutrino also provides a full complement of conventional synchronization services, including mutexes, conditional variables, sleepon locks, and semaphores. Page 4

5 for each link, a task that must be done by hand on an application-by-application basis. Each application may have to specify the primary link, any alternate links, and how to respond to those links. To complicate matters, the number of links, the kinds of links, and, the policy for each link (e.g. use link x for failover, link y for load-balancing, etc.) can change from device to device, or even from installation to installation. In fact, a different policy may be required for each pair of processes talking across the links. To address this problem, Qnet, the process that implements the QNX micronetwork, provides inherent support for multiple links. Again, applications require no special coding to take advantage of this feature, thanks to the network abstraction enabled by QNX message passing. Importantly, system designers can control how the redundant links are used, by choosing from the following classes of service: Load-balancing Queue packets on the link that will deliver them the fastest, based on current load and link capacity. When this policy is in effect, Qnet uses the combined service of all links to maximize throughput and allows service to degrade gracefully if any link becomes unavailable. Once a failed link recovers, it can automatically resume sharing the workload. Preferred Send out all packets over the specific link until it becomes unavailable, at which point use the second (or third or fourth) link. Qnet will automatically reroute traffic back to the preferred link once it has rejoined the pool of available links. Locked Only use the specified link. This policy is useful when only one of the available links is appropriate to certain traffic. Of course, a policy appropriate for one application or type of traffic may not be appropriate for another. Consequently, Qnet can support multiple policies simultaneously for different connections across processes. Seamless Routing To further simplify distributed processing, QNX Neutrino makes full use of the device ID addressing used by RapidIO technology. In RapidIO, no need exists for an address map or other addressing Load-balancing: Use the combined service of all links to maximize throughput. Preferred: Use an alternate link only if the preferred link becomes unavailable. Locked: Use only the specified link Figure 4 Qnet provides multiple classes of service, allowing redundant links to increase throughput, fault tolerance, or both. Page 5

6 scheme, since addressing is done using the individual device s ID. QNX Neutrino can use the device ID as its own addressing scheme, making routing in a distributed system seamless to the application. Using Standard Protocols For more conventional networking, the QNX Neutrino RTOS also supports TCP/IP protocols between processors on the RapidIO interconnect. Supported stacks include: NetBSD TCP/IP stack Supports the latest RFCs and functionality, including UDP, IP and TCP. Also supports forwarding, broadcast and multicast, routing sockets, ARP, ICMP, and IGMP. To develop applications for this stack, programmers use the industry-standard BSD socket API. Enhanced NetBSD stack with IPSec and IPv6 Includes functionality targeted at the new generation mobile and secure communications. Provides full IPv6 and IPSec support through the KAME extensions, as well as support for VPNs over IPSec tunnels. Also includes optimized forwarding code for additional performance. Tiny TCP/IP stack for memory-constrained systems Despite its small footprint (<80k), this stack provides complete support for IP, TCP, and UDP over Ethernet and PPP interfaces. To develop applications, developers use the BSD socket interface; in fact, developers can switch from the NetBSD stacks to the tiny stack without having to modify or recompile code. Mutually Compatible As we ve discussed, QNX Neutrino s messagepassing microkernel architecture provides a single, unified development model for creating both uniprocessor and distributed applications: The same, simple message-passing paradigm that works on a single CPU works identically across a cluster of CPUs. It is, as a result, much easier to take advantage of RapidIO s distributed capabilities. Still, this only touches the surface of how QNX Neutrino and RapidIO can, together, improve the design of networking equipment. For instance, QNX Neutrino s microkernel architecture also makes it possible for a system to recover intelligently from errors in almost any software module even a faulty device driver can be dynamically restarted, without a system reset or user intervention. This software faulttolerance complements RapidIO s ability to recover from hardware errors, again without intervention. As a result, systems using both QNX Neutrino and RapidIO can achieve significantly greater uptime with little additional effort on the part of the systems designer. There are yet other ways in which QNX Neutrino and RapidIO work in concert to simplify the design of high-availability, high-performance network elements. For a summary of these complementary benefits, see the attached Affinity Table. Page 6

7 Affinity Table: RapidIO Architecture and the QNX Neutrino RTOS RapidIO and the QNX Neutrino RTOS are both designed for high-performance communications equipment. As a result, they complement and extend each other in numerous ways. For instance, both provide inherent support for loosely coupled multiprocessing (clusters) and for tightly coupled, symmetric multiprocessing (SMP). Both offer low latency, deterministic response, automatic error recovery, and compatibility with a large pool of existing software. And both employ a modular architecture to keep hardware costs to a minimum. The following table summarizes the various benefits that QNX Neutrino and RapidIO can, in combination, bring to a network element. Benefit How RapidIO contributes How QNX Neutrino contributes Scalability No practical limit. Architecture can address over 64,000 attached devices, using extended transport addresses. Support for thousands of memoryprotected processes per node, with no practical limit on number of nodes. (Note that nodes can share resources transparently; see Distributed processing, below.) Reliability and fault-tolerance (See also Redundant links between nodes ) Can recover from all single-bit and most multi-bit errors, without software intervention. Can also notify software of severe errors, allowing software to redirect traffic around failed device. Point-to-point architecture helps eliminate single points of failure in hardware (e.g. a pin failure in the backplane). Error recovery techniques include: performing error detection and correction on each link in the path of a data transfer correcting corruption by using separate CRCs for the header and the data payload Can recover gracefully from software faults, even in low-level drivers and protocol stacks, without system reset. Microkernel architecture helps eliminate single points of failure in software by running every application, driver, and protocol stack in a separate memoryprotected address space. Faults are typically isolated to the process in which they occur. Automatic error recovery enabled by the QNX High Availability Toolkit (HAT), which checks for early warning signs that an error is occurring and allows the developer to specify recovery actions (e.g. stop and restart faulty driver; connect database to backup file system; and so on). Page 7

8 Benefit How RapidIO contributes How QNX Neutrino contributes Distributed processing (cluster computing) RapidIO provides a unified processor bus, allowing multiple devices to act as peers. Any device can communicate directly with any other device. Built-in peer-to-peer networking allows any node to transparently access the resources of any other node. Uses RapidIO s source-based addressing scheme to identify targets of transactions in a distributed system. Networking is integrated into the heart of QNX message-passing primitives, making local and network-wide interprocess communication (IPC) one and the same. Thus, an application can access hardware resources on any other node without having to invoke remote procedure calls. Programs are inherently network-distributed. Redundant links between nodes Supports arbitrary topologies, allowing the systems designer to implement multiple redundant links for faulttolerance and extremely high bandwidth. Allows applications to communicate transparently across multiple redundant links. Links can be a combination of various media and protocols RapidIO, fiber, Ethernet, and so on. Designers can control how traffic flows across redundant links to boost throughput, fault-tolerance, or both. Tightly coupled, symmetric multiprocessing (SMP) Supports SMP through an optional distributed shared-memory extension. Supports true shared everything SMP through optional SMP microkernel. Microkernel allows any thread to run on any available CPU for maximum performance and flexibility. Applications, drivers, and OS modules can move unmodified from singleprocessor to SMP systems. No need to hardcode SMP awareness into each software process. Page 8

9 Benefit How RapidIO contributes How QNX Neutrino contributes Fast, predictable response times Provides determinism through multiple message-priority levels, which ensure that high-priority transmissions reach their destination in a fast, predictable manner, even under heavy network loads. Offers low latency through techniques such as: allowing packets to wait until a link becomes available (no need for packets return to their origin) performing automatic error recovery in hardware Offers lower latency than bus technologies like PCI and PCI-X, while using a smaller silicon footprint. Provides deterministic response times, both at the application level and within all subsystems. As a result, any timecritical task, such as a routing table update, can be processed in a fast, predictable manner, even when many other processes demand CPU time. Predictable response achieved through techniques such as: priority-driven preemptive scheduling nested interrupts priority inheritance for all processes very fast interrupt latencies and context switches (less than one microsecond per context switch on a Motorola MPC7450 processor) Supports network-distributed priority inheritance so that requests from multiple nodes are processed in priority order. As a result, a system can respond predictably even when its applications and services are distributed across a number of CPUs. Small footprint, lower hardware costs Modular specification OEMs can include only those functions needed by application, minimizing silicon footprint. Interface can fit in a corner of a processor, FPGA, or ASIC. Uses few signal pins (e.g. 8-bit parallel version uses only 40 pins per bi-directional port, less than that required for PCI). Modular architecture The QNX microkernel and its accompanying Process contain only the most fundamental services: signals, scheduling, process creation, etc. All other system services (file systems, protocol stacks, drivers, etc.) exist as optional, add-on processes. OEMs can include only those OS modules required by the application, minimizing memory costs. Page 9

10 Benefit How RapidIO contributes How QNX Neutrino contributes Small footprint, lower hardware costs (cont d) Simplified thin client computing network transparency provided by QNX message passing allows even the most resource-constrained node to access resources on other nodes: disks, network cards, protocols, databases, etc. Support for open standards Compatibility with existing hardware and software Hardware independence Developed through a cooperative effort involving Alcatel, Cisco, Lucent, Motorola, Nortel, and other major networking companies. Freely available to the industry. Transparent to application software. From software perspective, the RapidIO interconnect looks like a traditional microprocessor and peripheral bus. Also bridges easily to PCI and PCI-X, allowing designers to use legacy PCI chips. The RapidIO logical specification, which defines the protocol and packet formats, can be transmitted over any interface (serial, parallel, etc.) and any media (copper, fiber, etc.) OS engineered from ground up to support industry-standard POSIX APIs. Allows OEMs to leverage large Unix/Linux developer community. Graphical Integrated Development Environment (IDE) based on Eclipse, an open, extensible framework that allows developers to integrate tools from multiple vendors. Standard APIs allow developers to easily reuse code developed for other POSIX/Unix/Linux OSs. Popular Internet software like Apache, Perl, GateD can run natively, without code changes. Supports wide variety of processors and boards used in communications products: PowerPC, MIPS, x86, SH-4, ARM, StrongARM, XScale. Transport independence QNX message passing hides details of underlying network media and protocols from applications. As a result, new media and protocols can be introduced without application recoding. Processor independence QNX Neutrino allows many applications and drivers to be source-code identical across processor families. Page 10

11 Benefit How RapidIO contributes How QNX Neutrino contributes Ease of extensibility Specification is partitioned to support future protocol extensions. Designed so that switch fabric devices don t have to interpret packets, enabling forward compatibility. Since most system services (drivers, file systems, protocols, etc.) are implemented as memory-protected processes, extending the OS is simply a matter of writing new user-space programs. Programmers can create OS extensions, using standard source-level tools and techniques no kernel programming or debugging required. Drivers, protocols, and applications can also be stopped and started dynamically, allowing operators to extend a live system on the fly. No need for system resets or service interruptions. Low transaction overhead to take full advantage of available bandwidth Source routing Ensures that only the path between the sender and receiver is burdened with the transaction. No need for the transaction to be regenerated by a host controller. Since packet headers are as small as possible, and organized for fast assembly and disassembly, control overhead is minimal. Efficiency increases as data in each packet increases. Direct copying of messages Message-passing is a direct operation between the sender and receiver only. No need for intermediate copying if messages are exchanged on local node. Enables message passing to perform on a par with conventional IPC. Since most QNX messages are quite tiny, the amount of data moved around the network can be far less than with network-distributed shared memory. To further conserve network bandwidth, QNX RTOS supports combine messages, which package multiple messages into a single message, thereby reducing the number of transactions QNX Software Systems Ltd. All rights reserved. QNX, Momentics, Neutrino, Photon microgui, and Build a more reliable world are registered trademarks in certain jurisdictions, and Qnet is a trademark, of QNX Software Systems Ltd. All other trademarks and trade names belong to their respective owners. Page 11