Overview. ATCA Reference Architectures Real-time Acquisition and Recording. Reference Architectures. Inside contents. AdvancedTCA Technology

Transcription

1 SAN Technology, Inc. ATCA Reference Architectures Real-time Acquisition and Recording Reference Architectures Magma PCI Express 10Gb ATCA platforms Overview Inside contents Overview 1 Recording Architectures 2 Local Storage 2 Shared Storage 3 Virtualization 4 Equipment summary 5 Summary 6 Real-time data acquisition and recording is a key part of numerous military applications such as radars, radio, wireless or virtually any task that involves digitizing analog signals. Once digitized, data is sent to computers for real-time processing or archived to disks for post processing. Sometimes customers want to play back data through D-A converters, reproducing the original analog output signals at full bandwidth. When decomposing the functional building blocks of such applications, we found significant overlap in wish-list feature requirements that included: Need recording/playback speeds 500 to 2000MB/s Preference for using COTS acquisition cards (A/D converters, FPGA etc) Wanted modern OS development environments (Windows 2012, Linux, VMware) Wanted Moderately rugged operating environments (0-55C) This paper explains how to combine a commercial-off-the-shelf (COTS) technology called ATCA with virtualization and standard PCI-express cards (for application specific ) to fulfill the technical requirements of many data acquisition applications, while also meeting Department of Defense (DoD) goals for Modular Open Systems Approach (MOSA) designs that reduce R&D costs and accelerate time to market. AdvancedTCA Technology ATCA is an open standards specification that defines a moderately rugged blade computing environment with literally hundreds of suppliers, 10 years in the making. ATCA blades aren t anemic, struggling to fit enterprise class computing into spaces too small to accommodate the technology. Rather, these blades feature Intel s newest 64-bit XEON CPUs, terabytes of storage, 10 Gb Ethernet networks and modern operating systems. System designers can construct custom platforms by combining multiple compute, network or storage blades to create scalable platforms with as few as two blades or groups of many dozens. ATCA was originally conceived to fulfill the computing needs of the US and European telecommunications Industry, replacing monolithic proprietary systems with reusable COTS products that enable faster development, easier technology refresh and to foster a competitive supplier environment to keep costs down. ATCA platforms offer redundancy, remote monitoring and consume less space, weight and power than an equivalent set of commercial desktops or rack servers. It is these same innate attributes that attracts DoD projects seeking acquisition reform, lower development costs, and 7+ year life cycles to satisfy the technology demands of 21 st century net-centric warfare. 1 ATCA Advanced Telecommunication Architecture ( ) 1

2 ATCA Recording Architectures BOX2 The recording architectures discussed in this paper can be built with the same two building blocks installed in a single 19 rack chassis. This two-box setup supports a variety of recording requirements and system architectures that share a common physical appearance. The first box contains a backplane with slots for ATCA compute and storage blades. The second box, manufactured by Magma, provides PCI-Express expansion slots to accommodate application specific cards. Cables are used to interconnect ATCA blade rear transition modules (RTM) with the Magma enclosures, bridging standard PCI Express cards with ATCA host blades. Magma product details are presented later in this paper. BOX1 BLADE RTM PCI-Express Cable System architects select and combine blades in an arrangement that best fits their performance, scalability and cost goals. There are several system design considerations and trade-offs which are discussed in the next sections. Local Storage Architecture When an application collects sampled data (bit streams) from acquisition cards, it can be stored to local (private) disk pools, or to networked disk pools. In testing, local disk pools delivered the most bandwidth and were integrated with the CPU slot saving space and power. An example of this architecture, shown below, uses one SANBlaze ATCA blade (ATCA2000) which combines a processor (4 CPU), 16 GB of memory and 3.8TB of RAID storage. Acquisition ATCA Storage Blade PCI-Express 4 Core XEON 16 GB DDR3 3.8TB SSD The blades disk performance was measured by installing Linux and configuring a RAID0 volume with (8) eight Intel SSD, Model 520. The RAID volume was then split into two logical volumes lva0 and lvb0. Linux dd command was used to simulate 2 data acquisition bit-streams; measured results appear below. Bit streams Measured Read bandwidth Measured Write bandwidth MB/Sec 1200 MB/Sec MB/Sec 1660 MB/Sec 2 One bit stream write/read: echo noop > /sys/block/sdb/queue/scheduler; echo noop > /sys/block/sdc/queue/scheduler blockdev --setra /dev/sdb; blockdev --setra /dev/sdc sync; echo 0 > /proc/sys/vm/drop_caches; dd of=/dev/vgb/lvb0 if=/dev/zero bs=1m count=30k oflag=direct sync; echo 0 > /proc/sys/vm/drop_caches;dd if=/dev/vgb/lvb0 of=/dev/null bs=1m count=30k iflag=direct Two bit streams write/read: sync; echo 0 > /proc/sys/vm/drop_caches dd of=/dev/vgb/lvb0 if=/dev/zero bs=1m count=30k oflag=direct & dd of=/dev/vga/lva0 if=/dev/zero bs=1m count=30k oflag=direct sync; echo 0 > /proc/sys/vm/drop_caches; dd if=/dev/vgb/lvb0 of=/dev/null bs=1m count=30k iflag=direct & dd if=/dev/vga/lva0 of=/dev/null bs=1m count=30k iflag=direct 2

3 Shared Storage Architecture Networked disk pools provide the ability to share storage capacity and bandwidth between several blades using 10Gb Ethernet connections. All applications can access the storage pool, but you control how much capacity they each get. With Networked storage you can amortize the disk cost across several applications, but maximum bandwidth is limited by 10Gb Ethernet connections (~1100MB/Sec). An example of this architecture appears in the figure below. A single Magma box holds all acquisition cards, but electronically isolates expansion slots so each card is exposed to only one PCI-Express computer master (root complex). The compute blades each contain two Intel Westmere processors (12 cores) and 192GB of memory. All of the compute blades connect to a single storage blade, sharing its capacity and bandwidth. Acquisition PCI-Express ATCA Compute Blade ATCA Network Blade ATCA Storage Blade 12 Core XEON 192GB DDR3 10Gb switch 4 Core XEON 16 GB DDR3 6TB SSD PCI-Express 12 Core XEON 192GB DDR3 The recording performance of this configuration was measured by installing Linux on each of compute blades. These connected to the SANBlaze storage blade (ATCA2000) using iscsi over switched Ethernet connections. The storage blade contained (8) eight Intel SSD Model 520, configured in a RAID0, and then sub-divided into two targets. Linux dd command was used to simulate 3 data acquisition bit-streams; measured results appear below. Bit streams Measured Read bandwidth Measured Write bandwidth MB/Sec 366 MB/Sec MB/Sec 689 MB/Sec MB/Sec 1366 MB/Sec (Two 10Gb ports) 3 One bit stream write/read: iscsiadm -m node --targetname Target iqn com.sanblaze:atca2000-d06f42.target0 --portal :3260 login iscsiadm -m node --targetname Target iqn com.sanblaze:atca2000-d06f42.target1 --portal : login echo noop > /sys/block/sdc/queue/scheduler; echo noop > /sys/block/sdd/queue/scheduler blockdev --setra /dev/sdc; blockdev --setra /dev/sdd dd of=/dev/vga/lva0 if=/dev/zero bs=1m count=30k oflag=direct dd if=/dev/vga/lva0 of=/dev/null bs=1m count=30k iflag=direct Two bit streams write/read: dd of=/dev/vga/lva0 if=/dev/zero bs=1m count=30k oflag=direct & dd of=/dev/vgb/lvb0 if=/dev/zero bs=1m count=30k oflag=direct & dd if=/dev/vga/lva0 of=/dev/null bs=1m count=30k iflag=direct & dd if=/dev/vgb/lvb0 of=/dev/null bs=1m count=30k iflag=direct & Four bit streams write/read: dd of=/dev/vga/lva0 if=/dev/zero bs=1m count=30k oflag=direct & dd of=/dev/vga/lva0 if=/dev/zero bs=1m count=30k oflag=direct skip=30k & dd of=/dev/vgb/lvb0 if=/dev/zero bs=1m count=30k oflag=direct & dd of=/dev/vgb/lvb0 if=/dev/zero bs=1m count=30k oflag=direct skip=30k & dd if=/dev/vga/lva0 of=/dev/null bs=1m count=30k iflag=direct & dd if=/dev/vga/lva0 of=/dev/null bs=1m count=30k iflag=direct seek=30k & dd if=/dev/vgb/lvb0 of=/dev/null bs=1m count=30k iflag=direct & dd if=/dev/vgb/lvb0 of=/dev/null bs=1m count=30k iflag=direct seek=30k & 3

4 ATCA and Virtualization ATCA combined with virtualization represents a strategic opportunity for consolidation and cost reductions at a system level. Data acquisition and recording may represent only one subsystem function of a larger project. With ATCA, a single computer blade with 12 CPU core can provide the same processing power as two or three computers designed 5 years ago. It becomes reasonable for embedded designers to consider hardware consolidation by running multiple tasks on a single ATCA blade. However, the system architect will need answers to some practical implementation questions: 1) How does someone map multiple standalone subsystems onto one physical computer platform? Solving this is fundamental to scaling applications that can fully utilize processor technology capabilities. 2) How does someone physically add many peripheral functions into single ATCA computer without designing form factor specific hardware? The ideal solution would accommodate high volume COTS PCI-Express cards. 3) As a single computer takes on more tasks, how can someone add tolerance and reliability? Virtualization addresses all these concerns. Consider for example, an existing rack containing 6 VME backplanes, dedicated to 6 monolithic tasks. With virtualization packages from VMware, it s possible to port these tasks to virtual machines (VM) with resources equal to the original VME source computers, but utilizing physically fewer ATCA blades. A VMware hypervisor called ESXi is loaded onto the raw ATCA blade(s). The hypervisor abstracts physical computer resources (CPU, memory, networking etc.) and makes them available to a community of users. Using a simple GUI, users define their dream machine with processors, memory, storage, and install an OS like Linux or windows on the virtual machine. Once complete, users can interact with the virtual machine in the same fashion as a physical one. They can ping it, open remote shells, and run applications. Legacy Virtualized VME Backplane #1 ATCA Backplane MAGMA CHASSIS Processor VM TB RAID 0,1,5,6 Dual 10Gb switch Domain 1 PCIe PCIe Processor VM 2 12 Core XEON 192GB DDR3 RTM VM X-1 VM X Domain x Virtual machines boot from a shared ATCA storage blades with up to 12TB of total storage; this storage pool uses RAID algorithms to boost performance and protect user data. Pooled storage also makes it easier to deliver right sized disks to each virtual machine and its application(s). Data recording is accomplished by installing data acquisition cards in a Magma PCI-Express expansion box, and assigning their status to pass-through using the VMware GUI interface. In this way, a VM can own the card(s). By utilizing magma expansion technology, users can utilize standard PCI express cards and save the R&D costs of developing form-factor specific solutions. Virtualization did not impact storage performance; Measured Bit stream recording and playback was equal to the nonvirtualized shared storage results presented in the previous section. 4

5 Recorder Equipment Details Magma PCI-Express Technology Overview Magma develops and manufactures rack mount PCI Express Expansion Systems which are central to SANBlaze data recorder architectures. The product extends PCI Express signals from multiple ATCA blades over a cable to a chassis containing multiple card slots and installed peripherals. (see Magma expansion products incorporate PCI-Express switch technology that interconnects up to 16 slots. Users can install PCI-Express cards (end points) in these slots, noting that some slots are wired to support x16 traffic lanes, while others support only x8. Magma software can group slots into domains. Domains are managed by a single ATCA computer called a root complex. Each ATCA server connection consumes one slot, and all the cards assigned to the domain. In the topology example below, the Magma box is configured to present three domain trees, each with a single root, terminating at the ATCA blade. Blade 1 Blade 2 Blade 3 Domain color Grey Red Green Example PCI-Express Slot assignment 1 host, 2 cards 1 host, 1 cards 1 host, 1 cards SANBlaze Storage Blade Overview SANBlaze developed the ATCA2000 as a dual purpose product that operates either as an endpoint in an iscsi storage area network (SAN) or as a CPU with direct attach storage (DAS). If deployed in the latter configuration, users can install any OS which executes on its 64-bit Intel 4 core XEON processor. Examples include Windows, Linux or VMware. Other standard features include an Intel dual 10Gb Ethernet controller, a MegaRAID SAS controller from LSI and eight (8) userserviceable 2.5 SSD. In a recorder application, the blade sustained over 1600MB/Sec when recording bit streams and over 2200 MB/Sec for playback. 4 core XEON processor 6 TB SSD max 16GB DDR3 memory SANBlaze Compute Blade Overview SANBlaze developed the ATCA7300 to tackle more processor intensive applications. The dual processor design features 12 XEON cores, and up to 192 GB of DDR3 memory, making it an ideal blade for virtualization. These blades run commercial OS including windows, Linux and VMware. SANBlaze Network Blade Overview The SANBlaze ATS1936 features two separate 20 port Ethernet switches, one operating at 10Gb (fabric), and other 1Gb (base). Two switch blades provide a fully redundant star fabric that keeps connections available, even if one blade fails or is removed. 5

6 SANBlaze Technology, Inc. One Monarch Drive Suite 204 Littleton, MA Phone (978) Fax (978) Web Summary Numerous embedded applications utilize acquisition cards which create bit-streams of data that are processed by computers and often stored to disk. Often times, the target operating environment means commercial hardware cannot be used because it cannot meet requirements for operating temperature, mobility or ruggedness. To address these needs, designers turn to embedded computing products. ATCA is an embedded computing technology, with over 300 suppliers. Today, customers can choose from a large selection of computer, storage and network blades and interconnect them with 10Gb Ethernet fabrics. 10Gb Ethernet provides a ubiquitous, reliable, very fast, and cost effective method for moving data between computer and storage. Ethernet is also ideal for managing serial bit streams. The final challenge is adapting acquisition functions to the ATCA form factor, ideally without designing new daughter cards or accessory modules. Magma PCI-Expansion products can be used for this purpose. They enable customers to connect standard PCI-express cards with ATCA embedded computer with a cable. This paper explains how the two box system (ATCA and Magma) fits neatly into a 19 rack, providing the hardware basis for numerous applications including data recorder and playback with sustain speeds exceeding 2200MB/Sec. For more information please visit the SANBlaze web site at: or send info@sanblaze.com SANBlaze Technology, Inc. is a leading provider of storage, networking and multifunction solutions for embedded systems. SANBlaze embedded computing products include a complete line of ATCA storage and compute blades, multifunction RTMs for ATCA blades, and AMC storage and networking controllers and modules. Additionally, the company provides fully configured and integrated ATCA systems and services. Copyright 2012 SANBlaze Technology Inc. All rights reserved. Referenced products are trademarks or registered trademarks of their respective owners UC 125 r00 6