Flexible Hardware Support for Interworking Systems. Till Harbaum Detlef Meier Matthias Prinke. Martina Zitterbart

Transcription

1 Flexible Hardware Support for Interworking Systems Till Harbaum Detlef Meier Matthias Prinke Martina Zitterbart fharbaum meier prinke Institute of Operating Systems and Computer Networks TU Braunschweig, Germany Abstract Todays FPGA technology allows recongurable hardware to be integrated into standard PC hardware. With this kind of hardware it is possible to change hardware functionality on-the-y and to remove and insert complete subsystems just by rebooting the congurable parts. Hardware like this has new demands on software that often can't be fullled by the classical device driver concept used in todays operating systems. This paper presents a exible and on-the-y recongurable hardware subsystem for interworking systems together with software integration that allows fast and ecient reconguration of the hardware. It provides a exible integration into a Unix environment. 1 Introduction With the increasing complexity of todays semiconductors, new hardware concepts are evolving. For a long time, processing power was a limited factor in desktop computers and the main focus on hardware enhancement was on the performance of the system CPU. Nowadays, high performance versions of main components like CPU and memory are available even to the low end users (e.g., PC users). Further improvement of the overall system performance requires the integration of new hardware concepts into standard computer systems. Several approaches are currently discussed and investigated to increase system performance: Usage of multiple main CPUs (SMP). By increasing the number of CPUs, the overall performance can be raised signicantly. However, system limitations like memory bandwidth will limit the maximum number of CPUs to a very small number [4]. The integration of special purpose hardware (e.g., 3D accelerators, video encoders and decoders) can increase the performance of dedicated tasks, but is of limited or of no use for other applications [3]. By inserting special purpose CPUs and DSPs, eective support for very CPU intense operations (e.g., video and audio lters) can be added to the system [1]. The integration of special purpose subunits into the system CPU itself (e.g., MMX, 3D-Now...) forms a similar approach. It does not allow independent scaling of CPU and special purpose hardware. Moreover, it often limits/aects the functionality of the main CPU (e.g. MMX and 3D-Now aect the Floating- Point-Performance) [2, 5].

2 Multiple Special seperate proprietary re-congurable main CPUs purpose CPUs and CPU hardware (SMP) hardware DSPs extensions (FPGAs) Performance high very high high high high Scalability very low low high very low very high Flexibility high very low high low very high Table 1: Current technologies to increase system performance Reprogrammable hardware (FPGAs) can perform various tasks at a very high performance, but have limited storage capacity and always lack performance in comparison with special purpose hardware [6]. However, they provide more exibility than dedicated hardware because they can be reprogrammed and, thus, be exibly adapted to a specic task. Most of the approaches listed in table 1 are already available to end-users. Each of it has its limitations. A exible high end solution for interworking systems will try to combine the advantages of several very exible dierent hardware technologies into a single system [7]. This paper presents FHiPPs, a exible general purpose hardware for use in standard PC systems. Since FHiPPs was built to be as exible as possible, it includes a CPU with local memory, a DSP and several FPGAs to combine the most exible and scalable hardware currently available. An important issue in this context is the integration of new hardware concepts into the hosts operating system. The classic hardware integration is based on a device driver. This approach works well for the integration of the pre-dened functionality of a special purpose hardware into the kernel. However, the classic device driver concept is limited with respect to functionality. It usually supports hardware with a predened functionality. Moreover, the installation of device drivers often includes manual interaction with the user and a reset of the whole system. It, thus, cannot be loaded during regular operation of the system. A exible hardware that can be recongured on-the-y needs a software integration that supports hardware re-conguration without user interaction. Furthermore, installation and removal of hardware specic functions need to be supported without interference to the normal operation to the system. Seamless reconguration and operating system integration of FHiPPs into the Linux operating system forms a main part of the presented paper. Applications for FHiPPs hardware range from hardware support of active networking [9, 12] over dedicated hardware support for classic router bottle necks [10, 8, 11] to support of hardware and software co-design of protocol engines. The paper is structured as followed. In section 2 the FHiPPs hardware is outlined. A exible software integration of FHiPPs is described in section 3. The paper closes with the outlook in section 4. 2 FHiPPs - Flexible High Performance Platform The exible hardware platform FHiPPs is an approach that combines the exibility of standard microprocessor technology with the performance potential of FPGA designs and digital signal processors. FHiPPs is composed of several units that can operate concurrently. Each unit comprises an individual bus as well as interfaces to the other units (cf., gure 1). The current FHiPPs implementation consists of two separate PCI boards: The i960 based main unit with the FPGA board and ATM interface. A DSP based unit. 2.1 The i960 unit The i960 (cf., gure 2) forms the basis of FHiPPs. It has a local i960 I/O processor to

3 SRAM FPGA 0 Switch Control FPGA 4 ATM NET TI DSP TMS32C080 FPGA 1 FPGA 2 FPGA 5 FPGA 6 1MB RAM ATM interface DSP unit FIFO FPGA 3 FPGA board glue logic MACH4-256 FIFO FPGA 7 NEC SAR Controller ATM interface PCI bus i960 bus PCI based host system PCI bus PLX-PCI9060 i960 based main unit intel i960 io processor 32MB DRAM Figure 1: The exible Platform perform general setup and control tasks for the other units integrated into the system. The i960 is connected to the PCI bus via the PLX9060 PCI interface chip. This chip allows various types of interaction between the FHiPPs and the host system. The main interface features are: Direct mapping of the i960 address space into the host address space. Interrupt handling from i960 to PCI and from PCI to i960. Several dual-port mailbox registers for simple bidirectional communication. Interrupt driven I/O ports (doorbell registers) from PCI to i960 and from i960 to PCI. The PLX9060 implements fast buered I/O in both directions. With the address mapping capabilities it enables the host to directly access the address space of the i960 unit, including the i960 memory and all other subunits (FPGA, ATM,... ). The advanced interrupt functions allow fast asynchronous communication between FHiPPs and the host system without appreciable delay. The interrupt functionality of the PLX9060 can be shared between all units attached to the i960 bus and the squall interface (cf., gure 1). It allows each unit to transfer data to and from the the host without interaction with the i960. Therefore, the i960 can be used to setup the subsystems and initialize direct communication between host and subsystem. The squall bus interface of the i960 unit allows an easy expansion. This 32 bit bus interface operates at 33 MHz and oers slave bus mastering as well as burst operations with a data rate of up to 160 Mbytes/s. The squall bus is mapped into the i960 address space and can be used transparently via the i960 bus and the PCI bus, allowing direct access of the host CPU to squall slaves and master bus accesses from the squall slave units to the host address space (e.g. direct transfers from the ATM interface into the hosts main memory). Another main feature of the i960 board is a local 32 Mbytes DRAM array that can be used for program and data storage for the i960 as well as for the subunits (FPGA, ATM). It is possible to map this memory directly into the user space of the host machine. This enables direct and simple interfacing with standard applications running on the host. 2.2 The FPGA board During the development of FHiPPs, the main focus was on the design of the FPGA board (cf., gure 3). It supports multiple concurrently operating FPGAs with local bus interface, I/O

4 Squall bus (to FPGA/ATM) Squall interface DRAM/IO controller RAS/ CAS 16 MB DRAM bank 0 16 MB DRAM bank 1 DRAM 33Mhz / 32 bit i960 local bus PCI bus (to host/dsp) PLX9060 PCI interface i960 CPU Figure 2: The i960 unit memory and buer memory each. The FPGA board consists of nine Xilinx 4020e (optional Xilinx 4044ex) FPGAs. Each FPGA is connected to the central switch matrix via two synchronous unidirectional eight bit busses. The switch matrix is controlled by the switch controller which is one of the FPGAs. The switch controller is connected to each of the other FPGAs by several control lines. A FPGA requests a connection to a particular destination via these lines. The switch controller sets up the connections as soon as the required destination is available and acknowledges the connection which is then available to the requesting FPGA. The remaining eight FP- GAs can be exibly used for high performance processing of user tasks. To support fast and ecient I/O, each FPGA has a local dual-port RAM. It is used to buer incoming messages until the FPGA is ready to handle the received data. An additional 2 Mbytes of static RAM is available for further buering. This RAM may be used to support complex state machines inside the FPGA or to store frequently accessed data locally instead of storing it in the i960 main memory. The FPGA board interfaces to the i960 units squall bus. The translation between the synchronous 8 bit bus interface used with the FPGAs and the asynchronous 32 bit squall interface is done by a FIFO. It uses high speed DMA transfers to the squall interface and does the bus width adaption from 32 to 8 bits. The FIFO stores messages up to 2048 quadwords and is controlled by a DMA controller and the switch controller in a way similar to the FPGA interfaces. Thus, the FPGAs can request and use a connection to the squall interface the same way, they use connections between each other. Once a connection to the squall interface is established, the DMA interface transfers the data into the i960 main memory and informs the i960 and optionally the host processor of the arrival of new data. This design supports hardware/software co-design in a very comfortable way. Due to the generic interface programs running on FPGAs can transparently communicate with another FPGA, the i960 or the host CPU. The FPGAs are clocked by a phase locked loop (PLL). The clock rate can be programmed between 1 and 32 MHz. Data is transferred between the FPGAs at a rate corresponding to the FPGA clock rate. This results in a maximum throughput of about 2*32 Mbytes/s between the FPGAs and the squall interface. The squall interface uses DMA bus mastering and has a sustained data transfer rate of 133 Mbytes/s to the i960 bus. This way, the asynchronous CPU bus does not represent a bottleneck for hardware/software communication between the FPGAs and the i960 CPU. FPGAs have limited debugging capabilities. To support comfortable debugging, several debugging units were designed. The main debugging tool is the vampire unit (cf., gure 3). It attaches to all nine FPGAs and the printer port of another computer. The vampire allows to capture the state of all FPGAs at a specic moment and to transfer the state into the other PC. With the appropriate software annotation

5 Debugging Unit FPGA Unit 2MB static RAM FPGA 0 Destination ID 5 FPGA 8 Request Xilinx 4020 Acknowledge (Xilinx 4044ex) Data Transmit dual-port RAM Data Receive 8 8 Switch Controller Xilinx 4020 Buffer i/o buffer 10 x 10 Switch 2MB static RAM FPGA 7 Xilinx 4020 (Xilinx 4044ex) Switch Control Debug Controller dual-port RAM 8 8 DMA Controller output queue input queue mux 32 Bus Interface Debug Interface Figure 3: The FPGA board tools, the current state of all FPGAs can be reconstructed. The reconstructed state together with the FPGA design description allows reconstruction of all signals inside the FPGA and can be used like a debugger for the FPGA compiler. While this tool is useful for static analysis, the dynamic state changes within the FPGAs can be monitored via debug interfaces which connect the FHiPPs to a logic analyzer and a LC display. 2.3 The DSP unit and the ATM interface The DSP unit and the ATM interface are othe-shelf components. They are fully integrated into the FHiPPs unit. The ATM interface allows fast I/O directly from the FHiPPs and the DSP unit is mainly used for audio- and videoprocessing. 3 Integration of FHiPPs FHiPPs can be integrated into PCI-based hosts. A complete FHiPPs (cf., gure 4) can be run independent from the host system. Due to a serial RS232 and an extra power connector on the i960 unit it is possible to power the system externally and use it without any host system at all. However, FHiPPs was designed to support a host and, therefore, needs to be integrated in the hosts operating system. Due to its open source concept and its easy expand-ability, Linux was chosen as the operating system for a prototypical integration of FHiPPs. Several components are needed to integrate FHiPPs into the Linux system: Device drivers and their front-ends to program the FPGAs and to download code to the DSP and the i960. Compilers to generate code for the i960, DSP and FPGAs. Drivers to include support for new functions for the FPGAs into the kernel. The rst two categories are standard applications. The main focus with the FHiPPs is on the exible integration of new functionality into the system kernel during runtime. Standard hardware has limited capabilities and, therefore, a device driver can be written to support all these predened functions. In contrast exible hardware like FHiPPs or P4 [6] can be modied by re-conguring the FPGAs and the hardware can provide a completely new functionality. The classic device driver concept

6 Figure 4: The FHiPPs (without DSP unit) does not work in such an environment since the update of functionality should happen: Without user interaction (the user may implicitely request a new functionality by starting a new application, but retrieval and download of new hardware description code should not require manual user interaction). Without the need to reboot the host. Without limiting the usage of the system during hardware reconguration. Therefore, a concept is needed that supports re-conguration of the hardware on-the-y as well as exible integration of new device support routines. The proposed approach implements such a exible system integration. The basic concept is not limited to a special functionality. The current implementation, however, focuses on the support for active networking. It, therefore, was integrated into the service module concept of the AMnet approach [12]. 3.1 Hardware Applets Direct access to hardware requires system privileges. Therefore, new code segments must be integrated into the system kernel in order to setup and support a new hardware functionality. Setup of a new hardware functionality consists of three major steps: Installation or modication of hardware functionality by downloading FPGA description les and/or DSP and CPU code fragments to the hardware. Boot or reboot parts of the congurable hardware in order to setup CPU states, initialize memory or bring FPGA state machines into an initial state. Install drivers in the hosts operating system in order to allow transmission and reception of data to and from the hardware. The new concept allows to include all these functions into one le. This le is called a happlet (hardware applet). It contains all necessary code fragments and hardware description codes to install or modify the required hardware functionality. This part of the happlet is called the hardware code segment. The other part of the happlet contains the host code segment. This segment holds all information needed to transfer the hardware code onto the hardware, to re-initialize it and to install I/O routines for the new hardware functionality. Access to the hardware may be implemented directly by the

7 happlet or via some device driver. It will often be useful to have some generic functions for the exible hardware available in a standard device driver. These functions will for example be able to reprogram the FPGAs or to boot some program onto the CPU/DSP of the subsystem. Other more specic or timing critical functions are integrated directly into the happlets host code. This can be highly optimized for the hardware functionality implemented by this happlet. Other more general functions required by a happlet, such as I/O buering and the like are provided by the hardware manager. Several ways of implementing service routines for happlets are supported: Highly optimized routines are included into the happlet. These routines can be optimized for the specic purpose of the happlet. This will mainly be hardware I/O functions like interrupt control and fast data transfer. General hardware specic functions may be included into a device driver. These functions are unique for each hardware and are provided for all happlets using this specic hardware. This kind of code is used for initializing and resetting the hardware. Generic hardware independent functions are provided by the Hardware Manager. This code will be used for buer and I/O control and to interface with the software interface of the HW Manager. User space based I/O is supported via the Hardware Manager. This kind of code is used to start a code segment in user space to easily integrate existing software into the concept. The demo happlet uses an user space program to reset the FHiPPs hardware and to perform a self-test. 3.2 Hardware Manager The core of the exible hardware integration is the hardware manager (cf., gure 5). The hardware manager is a generic pseudo device driver that is included into the system kernel. It uses a simple software interface for data transfer between software-based functions and the hardware. It also supports exible insertion and removal of new happlets. Additionally the hardware manager monitors the usage of the dierent hardware units and optionally with a resource manager for proper system-wide resource management. The Hardware Manager is integrated into the Linux kernel. It implements a pseudo device interface (/dev/happlet). This interface can be used by programs to request and access dedicated hardware services. A program starts interaction with the Hardware Manager by opening the pseudo device and requesting a specic function. If this functionality is not already available, the Hardware Manager loads the requested happlet into the system kernel. It sets up a generic interface structure to the happlet and initializes I/O buering for this happlet. Finally the happlets initialization routine is executed. It is responsible for the following tasks: It checks the availability of the required hardware. It initializes and boots the hardware. It informs the Hardware Manager about the buer and I/O requirements of the happlet. And it informs the Hardware Manager about the implemented I/O functions realized in this happlet. After the hardware manager is informed about the buer and I/O capabilities of an happlet, the happlet installation is nished. The calling program may now set up buers and I/O ports to interface with the happlet. Allowing the calling program to allocate buers instead of delegating all I/O buering to the hardware manager has the advantage of reduced I/O overhead. Since all buers are allocated by the calling program (which is a user space program), all data can directly be transfered from the user space onto the hardware and back from the hardware directly into user space. No additional copying between kernel and user space is required.

8 User interface application 1 application 2 appl. area download job requests, data IO hw host code code happlet a Hardware Manager pseudo device /dev/happlet OS kernel hw code host code happlet b hw host code code preprocessing x (host code) preprocessing y (host code) happlet c download device driver 1 device driver 2 host hardware 1 hw module x (hw code) hardware 2 hw module y (hw code) hardware units Figure 5: The exible Platform When the setup call succeeded the calling program may start sending data to the Hardware Manager. All data from and to the hardware is forwarded through the hardware manager and, thus, may be scheduled in cooperation with a resource manager (like the AMnet Manager). Since hardware is a limited resource, it can be managed by the hardware manager individually for each application allowing eective hardware resource management. Therefore, the hardware manager constantly monitors the usage of all hardware subsystems and forwards information on the current load to the resource manager. 3.3 The calling application The concept requires the calling application to interact with various unknown hardware units. Therefore, it must not implement any hardware specic tasks, but needs to interface to the Hardware Manager via the pseudo device. To allow a specic task to be executed in hardware, the application needs to open the pseudo device and request a specic functionality via the ioctl system call. The Hardware Manager sets up all required interface structures and informs the application about number and kind of the established interfaces. Once the ioctl call succeeded, the application can be sure that the happlet was successfully installed and may send and receive data to the hardware and receive the output. The same hardware can be used by dierent happlets at once (e.g. eight dierent FPGA based happlets may be run simultaneously on the FHiPPs). Furthermore, each happlet may be used by several applications at once (e.g. block oriented lter operations will allow the happlet to accept lter jobs from dierent applications). Therefore, the Hardware Manager has the ability to re-congure portions (e.g. a single FPGA) of the hardware on-the-y without aecting other parts of the hardware (other FP- GAs) or other happlets. Each happlet is capable of simultaneously handling several requests. This feature was not included into the hardware manager since task scheduling and buering inside the happlet can be handled more ef- ciently by the happlet itself. E.g. complex lters/transcoders (as needed for video encoding) often do not work on a xed data block size and require detailed information on the data to be transformed for ecient scheduling and buering.

9 3.4 Installation of a happlet To allow the installation of a specic happlet, it must available on the local system (cf., gure 5) or on a happlet server. Download of happlets onto the system or setup of a connection to a happlet server requires manual user interaction at system setup time only. Happlets use a unique naming scheme, allowing to determine from an applications job request whether the requested happlet is already available on the local system. Advanced environments will allow to retrieve missing happlets from happlet servers. This load-on-demand over the internet will allow happlet providers to maintain an up-to-date pool of happlets and to update happlets without notication of the user. Once an application requests a happlet that is available either locally or on a happlet server, the Hardware Manager installs the host code of it on the host system. The happlets host code is now used to initialize the hardware, to perform some hardware tests and to install the hw code of this happlet on the hardware. Furthermore, information on the number of interfaces required for this happlet and the number and size of I/O buers is returned to the calling application. The application has to set up the required number of I/O connections and to allocate buer space in order to complete the installation of the happlet. After termination of all applications using a specic happlet, it is closed. Upon close the happlet may bring the allocated hardware resources into an idle state. All buers and the hardware are freed for further use by other happlets. 3.5 Performance considerations The goal of the exible hardware integration is to increase the overall system performance. Therefore, the software integration must support fast and ecient data transfers to the hardware units. The two basic I/O concepts for data transfer currently used in operating systems are synchronous and asynchronous transfer. While synchronous I/O is easy to implement and is supported by the classic device driver concept, a fast and ecient asynchronous approach allows for higher performance. The asynchronous approach allows a happlet to copy data directly from the hardware into the user space without delay. With FHiPPs, asynchronous communication as well as synchronous I/O are supported. Thus, it can be selected between easy happlet implementation or maximum performance. Additionally, for some hardware units (e.g. the FHiPPs i960 unit) it is possible to map the hardware memory into the user address space. This feature may be used to allow hardware and software service modules to operate on the same data. The current approach does not support this feature, since most service modules do not work on static data and require to copy the data, anyway. Furthermore, access to memory-mapped external hardware is much slower than access to the hosts main memory. Forcing software-based service modules to work on I/O mapped memory would decrease the overall performance instead of increasing it due to reduced data transfers. The exible support is currently available on the Linux platform. First tests show that the setup delay mostly depends on the exible hardware used. The FHiPPs contains i960- microprocessor based parts which can be recon- gured and rebooted in at most two seconds. Reprogramming of an FPGA takes about 35 milliseconds but sometimes requires to install new code on the i960 and, hence, takes up to two seconds to reboot the i960, too. After this time a locally available happlet can be installed into the host system within several milliseconds The time required for this installation depends on the mechanism used to integrate the happlet code into the systems kernel. The Linux version uses the kernel module loader which loads kernel modules within at most 100 milliseconds if the module is locally available. Future versions with access to a happlet server and the download of happlets thru the Internet will slow down the initial setup of a happlet signicantly. The consequences of this delay will depend on the application. The current implementation is used for active networking. The happlets installed for active networking (lters, transcoders,... ) are used for longer periods (e.g. video lters are used for the whole duration of the video connection) and setup times of a few seconds can be tolerated.

10 For applications which require fast setup, local availability of happlets is strongly recommended, a technique that allows to transfer new code segments to the i960 unit without rebooting the i960 may be required, too. This will reduce the setup delay to a few milliseconds even for happlets containing both, i960 and FPGA code. 4 Summary and outlook The FHiPPs hardware platform and its software integration were successfully integrated into the Linux system kernel. Todays operating systems (Linux) already include concepts like dynamic loadable device drivers that can be adapted and extended to support seamless integration of exible hardware by introducing an unied pseudo device interface. After the proposed approach has been implemented current work focuses on the integration of the FHiPPs hardware/software concept into the AMnet active networking project [12]. Other applications that are currently applied to the FHiPPs are hardware support for RSVP routers and the support of HW/SW co-design of protocols. Future applications of the FHiPPs will be general support for multimedia applications and further support for inter-working systems. References [1] R. J. Clarke. Digital Compression of Still Images and Video. Academic Press Inc., [2] Intel Corp. MMX Technology at a Glance. Technical report, June [3] S3 Corp. Savage3D Accelerator: A Comprehensive Architecture for Superior PC Video. Technical report, November [4] Alan Cox. An Implementation of Multiprocessor Linux. Technical report, [5] Advanced Micro Devices. Inside 3DNow! Technology. Technical report, February [6] I. Hadzic and J. M. Smith. On-the-y Programmable Hardware for Networks. In Proc. Globecom '98, Sidney, Australia, November IEEE. [7] W. H. Mangione-Smith, B. Hutchings, D. Andrews, A. DeHon, C. Ebeling, R. Hartenstein, O. Mencer, J. Morris, K. Palem, V. K. Prasanna, and H. A.E. Spaanenburg. Seeking solutions in congurable computing. In IEEE Computer Magazine, pages 38{43. IEEE, December [8] S. Nilsson and G. Karlsson. Fast address lookup for internet routers. In Proc. Broadband Communications, University of Stuttgart, April IFIP. [9] D. L. Tennenhouse, J. M. Smith, W. D. Sincoskie, D. J. Wetherall, and G. J. Minden. A Survey of Active Network Research. IEEE Communications Magazine, 35(1):80{86, January [10] T.Harbaum, D. Meier, M. Zitterbart, and D. Brokelmann. Hardware Assist for IPv6 Routing Table Lookup. In SYBEN '98, Zurich, Switzerland, May [11] M. Waldvogel, G. Varghese, J. Turner, and B. Plattner. Scalable high speed ip routing lookups. In Proceedings ACM SIGCOMM '97, Cannes, France, September [12] R. Wittmann and M. Zitterbart. AMnet: Active Multicasting Network. In Proc. of Intern. Conf. on Communications (ICC'98), Atlanta, GA, USA, June IEEE.