High Speed Optical Interconnect

Transcription

1 High Speed Optical Interconnect May08-06 Draft Design Client Lockheed Martin Advisor Dr. Ahmed Kamal Team Members Bader Al-Sabah Dave Feely Adam Jackson Richard Jones Nick Ryan DISCLAIMER NOTICE! DISCLAIMER: This document was developed as part of the requirements of an electrical and computer engineering course at Iowa State University, Ames, Iowa. The document does not constitute a professional engineering design or a professional land surveying document. Although the information is intended to be accurate, the associated students, faculty, and Iowa State University make no claims, promises, or guarantees about the accuracy, completeness, quality, or adequacy of the information. Document users shall ensure that any such use does not violate any laws with regard to professional licensing and certification requirements. Such use includes any work resulting from this student-prepared document that is required to be under the responsible charge of a licensed engineer or surveyor. This document is copyrighted by the students who produced the document and the associated faculty advisors. No part may be reproduced without the written permission of the senior design course coordinator. December 12,

2 Table of Contents 1. Terms and Definitions Requirements Specification Problem Statement Need Statement Market and Literature Survey Requirements Functional Non-Functional Environmental Constraints, Limitations, and Risks Constraints Limitations Risks Risk Management System Description User Interface Description Strategy Technology research Market research Design Implementation Testing Documentation Deliverables

3 3. Project Plan Resource Requirements Work Breakdown Structure Project Schedule Results of Market Research utca Hardware Viability Hardware Availability PCI-E Hardware Viability Hardware Availability Engineering Specification Basic System Design Hardware Specification Switch Transmission Medium Software Specification Qcheck Ethereal IP Traffic Test & Measure LAN Traffic V Test Specification Bandwidth Measurement Time Measurement

4 Latency Measurement Quality of Service Measurement Bandwidth Efficiency Measurement Summary Appendix A Iowa State University (ISU) High Speed Optical Interconnect (HSOI) Senior Design Project Introduction ISU HSOI Team Statement of Work IRP Design Review Summary of Issues Does our intended software take into account overhead, buffer size, etc? How long does test run for? Can we keep the buffer full? How long? More concise definitions in general - what does "multiple" mean? Is 10Gb theoretical? Actual? Building to theoretical and testing for actual

5 1. Terms and Definitions Term Definition AMC ATCA COTS Gbps Micro TCA QoS PCI-E PICMG System Advanced Mezzanine Card Advanced Telecom Computing Architecture A component of high speed fiber networks that interprets control signals at a very high rate Commercial Off The Shelf Gigabits per second Micro Telecom Computing Architecture A point of connection to the system. Quality of Service PCI Express PCI Industrial Computer Manufacturers Group A component that enables connections via switching The sum of all nodes and physical connections. 2. Requirements Specification 2.1. Problem Statement At this time, the maximum real-world throughput of 10 Gbps network interface cards is unknown Need Statement Lockheed Martin (LM) needs a test plan designed and executed to measure the maximum realworld throughput of a 10 Gbps network composed of COTS components Market and Literature Survey ATCA and Micro TCA are new technologies governed by PICMG, which is a consortium of over 450 companies who collaboratively develop open specifications for high performance telecommunications and industrial computing applications. These technologies are intended to be the backbone for new high speed telecommunications hardware. ATCA is being implemented in newer system architectures. Its goal is to use standardized components to create core telecommunications systems. Micro TCA aims to accomplish the - 5 -

6 same goals as ATCA, but for low end systems with lower cost. Both cards are hot swappable, enabling the user to switch out boards without shutting the network down. ATCA and Micro TCA cards are mounted on a backplane. There are different physical backplane configurations available; the two main configurations being a cube architecture and a 19 inch rack mount enclosure. Each backplane can be configured to support different subsystems running on it simultaneously, with each system having a virtual carrier controller managing it. Both ATCA and Micro TCA cards are becoming widely available through well known vendors. PCI-E is a standard that has been around for approximately 3 yrs, and is now widely used. There are PCI-E NICs that are capable of 10 Gbps, but the PCI-E bus cannot handle that amount of data. If we were to use PCI-E NICs, we would have to work around this problem 2.4. Requirements Functional The prototype network shall be implemented with 10 Gbps components Advanced or Micro TCA components shall be considered first If Advanced or Micro TCA components will not work for us, we will explore other options The team shall test the network to determine real-world bandwidth, bandwidth efficiency, switch time, latency, and quality of service Non-Functional The solution shall be implemented and tested before the Senior Design class completes in May The solution should determine whether or not fiber optic components can be used within budgetary constraints Environmental The solution is assumed to be implemented in an environment where the following will affect the final design: Weight Temperature Shock sensitivity 2.5. Constraints, Limitations, and Risks Constraints There will be a limited budget available for the purchase of components Limited availability of 10 Gbps test equipment Limitations Solution must obtain a speed of 10 Gbps - 6 -

7 Each node must contain interface a network interface Risks A team member could leave the project (sickness, internship, etc.) No component could be offered within budget Component failure due to improper handling or usage Risk Management Multiple team members will be assigned to each major project task to minimize the negative effects of a team member leaving If no components can be found within budget, the system will be redesigned with slower and cheaper components The first few system configurations designed by one team member will be checked for correctness by another team member to avoid improper handling or usage of components 2.6. System Description The system will be comprised of at least three nodes that are connected to each other. All the nodes do not have to be connected directly; however a node must be able to reach any other node following a certain path. An example of the proposed network architecture is shown in Figure 3. Each of the connected nodes must have a switching fabric, a control processor, and a number of interface units. A possible node organization is shown in Figure 1. Figure 1 Example of a System using utca Figure 2 Example of a Possible Test Network Topology using utca - 7 -

8 2.7. User Interface Description The interface for the test design should be a command line interface that will run the test with the type of data the customer wants to use, and will provide an output file with various measurements of throughput. The output will focus on the maximum throughput, but may also include other factors such as minimum and average throughput for the same time period to provide checking information regarding network traffic at the time of the test Strategy Technology research The team shall find and read documents pertaining to 10 Gbps, focusing on ATCA and MicroTCA The team shall contact faculty members with expertise in relevant areas of networking technology Market research The team shall to find companies that offer products in line with what our technology research shows is the best implementation The team shall contact companies that offer components relevant to our chosen implementation to determine cost and availability The team shall decide whether copper or fiber optic interconnect will be the best solution based on transmission speed, number of nodes, and budgetary constraints Design The team shall design a node consisting of: A network interface Required controlling and switching components The team shall design an effective method for connecting nodes Implementation Based on market research results, the team shall purchase components required to implement this design The team shall use the purchased components to build a prototype network Testing The team shall test the prototype network for basic functionality The team shall design and execute a test scenario to determine real world throughput and bit error rate of the switching fabric using multiple 10 Gbps data streams Documentation The team shall document the technology and market research results The team shall document the design process

9 The team shall document problems encountered while implementing the chosen design The team shall document the test results The team shall document any conclusions or suggestions Strategy Block Diagram Figure 3 Block Diagram of Project Strategy 2.9. Deliverables The team shall present the chosen design solution to LM for approval before implementation The team shall provide weekly reports to LM The team shall present the test results to LM The team shall deliver a final report and all documentation to LM The team shall deliver the prototype to LM at the completion of this project All deliverables shall be presented to Rick Stevens on or before April 25, Project Plan - 9 -

10 3.1. Resource Requirements The team will need to purchase Advanced or Micro TCA components. The team needs availability of fiber optic cabling or the ability to purchase cabling if necessary. Test equipment, including a 10 GHz fiber optic scope and access to the high speed networking lab, will be needed. The team will also need to receive guidance and technical knowledge from our advisors Dr. Kamal and Dr. Smith

11 Personnel Meetings Research Design Implementation Testing and Debugging Documentation Website Totals 3.2. Work Breakdown Structure Adam Bader Nick David Richard Total

12 3.3. Project Schedule

13 4. Results of Market Research 4.1. utca Hardware Viability The results of our market survey lead us to believe that utca would work to implement our design. The standard allows for speeds up to 12.5 Gbps. utca systems allow for multiple nodes to be mounted on backplanes of different sizes, so our design would have scalability Hardware Availability 4.2. PCI-E After calling companies asking for their available 10 Gbps NICs, we were unable to find any options available on the market today. What are available are backplanes that are able to handle 10 Gbps. This makes 10 Gbps NIC implementation possible, but no companies had yet to offer them. A few of the companies we talked to said they were planning to offer 10 Gbps NICs within 6 months 1 year, which would not work for our time frame Hardware Viability PCI-E will allow transfer rates up to 8 Gbps through its interface using high end cards. We have looked into using cards with 4 Gbps. Neither option is fast enough for us to test 10 Gbps directly, but we plan on using memory buffers to achieve 10 Gbps Hardware Availability PCI-E is readily available and has COTS NICs available for use right now. The cost is within our budget, and so we shall move forward using PCI-E 5. Engineering Specification 5.1. Basic System Design The first implementation will be a simple point to point connection. The 1 transceiver will be connected to the 2 receiver, and vice versa. Figure 4 Two- Directly Connected Network

14 The final design will be to add a third node and a switch. All three nodes will connect to the switch. Two of the nodes will transmit and the third will receive. Figure 5 Switched Network Configuration 5.2. Hardware Specification Computer System The computer system will need a motherboard with one PCI-E slot for each Network Interface Card (NIC) that will connect to it Dell s PowerEdge SC440 was selected as the computer system because of its support for the x8 PCI-E standard connection required for the selected NIC Network Interface Card Switch The NIC will be PCI-E form factor capable of connecting to the base system. This card will have optical transmit and receive ports with specified sending and receiving rates of at least 10 Gbps Netxen s NXB-10GXxR Intelligent NIC was selected for this component The switch will be a fiber optic switch capable of supporting at least 3 connections and 10 Gbps speeds The TigerSwitch 10G 8-Port Standalone XFP 10Gigabit Ethernet switch manufactured by SMC Networks, Inc was selected for the switch Transmission Medium The transmission medium will be made with fiber optic cabling supplied by the Department of Electrical and Computer Engineering. These transmission lines will be used to connect nodes and the router in the above systems Software Specification Qcheck

15 Qcheck will be used to make any measurements related to bandwidth, and will be used to generate traffic. This will include the raw bandwidth measurement and also the bandwidth efficiency measurement. Qcheck is able to vary the OSI Layer 4 protocol between UDP, TCP, and SPX. It is also capable of varying the OSI Layer 3 protocol between Internet protocol (IP) and Internetwork Package Exchange Ethereal Ethereal is a packet capture program. It logs all packets sent and received by a network card. This program will be used in testing to determine the amount of Layer 1 data sent over a certain period of time. This will be used to determine the bandwidth efficiency of the network. Ethereal has many additional capabilities that may be used as tests are executed. It will certainly be a valuable debugging tool IP Traffic Test & Measure Software testing suite used to generate TCP and UDP traffic. User can specify parameters to shape and control traffic load. The testing suite has on-line or off-line analysis of many QoS parameters (send and receive) including throughputs, inter-packet delay, packet erasure rate (PER) and packet transit delay LAN Traffic V2 Software used to generate traffic on IP networks using TCP, UDP, or ICMP. Allows user to view traffic statistics, configure IP equipment, and manage the network bandwidth. Must have individual licenses for each machine Test Specification After assembling the system the following tests will be performed Bandwidth Measurement Description This measurement will focus on channel capacity, or the tightest upper bound on the amount of information that can be reliably transmitted Testing Plan Throughput will be tested to determine the maximum amount of information that can be transmitted in a certain time interval. These measurements will be taken between two directly connected nodes using Qcheck software Presenting Results Compare link usage under varying workloads Time Measurement Description

16 This measurement is the time it takes for the Ethernet switch to process and output the data. time effectively measures the amount of time that it takes for the switch to buffer the data and route the data to the correct receiver Testing Plan This measurement will be taken by first measuring the latency between two directly connected nodes. After this, the latency between the same two nodes in a switched configuration will be measured. The difference between these two measurements will provide a reasonable estimate of the actual switching time Presenting Results Compare switching times for varying link loads and number of nodes connected Latency Measurement Description This measurement is the time delay from one node to another Testing Plan This measurement will be performed using a switched network to determine oneway trip time between the output of one node and the input of another node. Qcheck and ping will be used to implement this test Presenting Results Compare latency for varying loads and nodes connected to the network Quality of Service Measurement Description This is a measurement of how the data will be handled when two nodes transmit to the same node Testing Plan This measurement will be taken in the switched, 3-node network. The test will be performed by fully loading each link with traffic then measuring the percent of data each endpoint node receives from other nodes. In a perfectly balanced network, this will be 50% for each endpoint node Presenting Results Show data received from each node over time Bandwidth Efficiency Measurement Description

17 The measurement of how much real data is transmitted, in other words, the amount of information that is transmitted versus frame formatting, coding, etc. For this measurement, Cyclic Redundancy Checking (CRC) will also be considered lost (inefficient) bandwidth for this measurement Testing Plan Bandwidth efficiency will be measured with a direct connection between two nodes. Testing for this quantity will require measuring the raw OSI Layer 1 data sent between nodes. At the same time, OSI Layer 7 data must also be quantified for comparison. The efficiency will then be the ratio of OSI Layer 7 to OSI Layer 1 data. Ethereal will be used to calculate the Layer 1 data sent over the measuring period, and Qcheck will provide the measurement of the Layer 7 data sent Presenting Results Compare OSI layer 1 Data sent and OSI Layer 7 data generated will be used to present the results of this measurement. 6. Summary The project is feasible as outlined above. Our schedule changed slightly, only to reflect that we researched PCI-E after we realized we could not use utca. This did not set us back; it only changed what we were researching for the remainder of the research period. It had no other effects on the schedule. We will run the tests as planned on the PCI-E hardware. The largest risk we face is that Lockheed Martin may not be able to purchase the switch and transceivers. There is no way for us to mitigate this risk, and if it were to occur, our project would not be able to continue

18 7. Appendix A Iowa State University (ISU) High Speed Optical Interconnect (HSOI) Senior Design Project Introduction 9/21/2007 Lockheed Martin (LM) is interested in research in the area of high speed scalable networking architectures which connect multiple nodes. LM is investigating the use of ATCA and microtca standard routing and switching equipment. These two standards define a high speed backplane interconnect for the routing of data between multiple I/O and processing interfaces. Multiple modules are combined into a chassis to construct a routing and switching node. LM is interested in using 10 Gigabit Ethernet () interfaces to interconnect multiple nodes. Plug-in modules provide the physical interface. The final system would be interconnected by fiber optic cable. In the ATCA form factor, a single card may provide both the switching fabric as well as the I/O interfaces. MicroTCA cards are much smaller, so the switching function is performed by a single card called a MicroTCA Carrier Hub (MCH). The rest of the I/O interfaces are provided by additional plug-in modules called Advanced Mezzanine Cards (AMCs). Figure 1 is a block diagram of a typical node and Figure 2 shows one topology for how these nodes could be interconnected. Figure 3 - Block Diagram of a Single Network

19 Figure 4 - One example of a possible network topology in the delivered system LM has done some preliminary investigation into available ATCA and MicroTCA modules. Initial candidates are listed below: ATCA form factor o ATCA Gb Switch/Router (Radisys) MicroTCA form factor o CEN-MCH + CEN-FAB-E 10 GigE MicroTCA Carrier Hub module (CorEdge) o KSI8560 module (Emerson) The following are some related web sites that provide example modules and information on the ATCA/MicroTCA standards:

20 ISU HSOI Team Statement of Work As part of the ISU HSOI Senior Design Project, LM suggests the following tasks for the design team: 1. Perform a thorough trade study on currently available ATCA and MicroTCA modules and systems, including prices, lead times, and features. 2. According to the outputs of Task 1, select and purchase a set of modules that will allow testing of the throughput of a switching fabric with multiple 10 Gbps interfaces. A minimum of three 10 GbE interfaces into the system is desired. Optical interfaces on the I/O modules are preferred, but copper interfaces are sufficient if costs are prohibitive. 3. Build and check the test bed for basic functionality. 4. Design and implement a test of the maximum real-world throughput of the switching fabric. This will include identifying necessary test equipment for generating multiple 10 Gbps data streams. 5. Write and deliver a report detailing the findings of the study. 6. Provide regular status updates to LM

21 8. IRP Design Review Summary of Issues 8.1. Does our intended software take into account overhead, buffer size, etc? Operating system overhead will not be taken into account by the chosen software. The team will use a Linux operating system whenever possible to because it has a lesser overhead than Windows. The buffer size will be another constraint, but the team hopes to access the driver on the NIC to setup a circular buffer to avoid having the host system itself slow down the transfers How long does test run for? Can we keep the buffer full? How long? Due to the speed of the network, it is possible the computer won t be able to keep the memory full of data to be sent. Solutions to this issue will be investigated. The team will look into host systems with 4 G of memory. One idea is to send the data, but not remove it from the buffer, so that it can then be resent. Theoretical time that we can run will be calculated, and tested to confirm actual times. If a solution to keeping the buffer full is found, this will be tested over a course of days to determine that speeds can be maintained More concise definitions in general - what does "multiple" mean? Each test will be run a multiple of times. In multiple we mean approximately Once we have this much data we'll be able to identify outliers and data points that seem skewed from actual distribution. Also, this high number of data will allow us to approximate our results with the normal distribution and thereby establish a confidence interval of the parameter we're seeking. We'd like to establish a 95% + confidence interval for our testing value Is 10Gb theoretical? Actual? Building to theoretical and testing for actual. This system is not designed to achieve 10 Gbps, but only to test components rated at 10 Gbps