μtca-based Controller

Transcription

1 μtca-based Controller Aleksander Mielczarek, Dariusz Makowski, Grzegorz Jabłoński, Andrzej Napieralski, Piotr Perek, Paweł Prędki Department of Microelectronics and Computer Science Technical University of Lodz Lodz, Poland Abstract This paper presents the μtca-based Controller (μtc) hardware module that, was designed to meet he requirements for a cavity field stabilizing controller for standingwave linear accelerators and for a Fast Beam-Based Feedback Processor (FBBFP) hardware platform. The μtc is one of the several modules for High Energy Physics (HEP) designed according to the drafts of MTCA.4 specification. The necessary design requirements and implementation of the hardware module for the cavity electromagnetic field controller will be discussed. The results of tests performed in the laboratory and comparison with SimCon-DSP board will be presented. Index Terms Micro Telecommunications Computing Architecture, MTCA.4, RF Control System, Beam-Based Feedback, Digital Signal Processing I. INTRODUCTION The xtca for Physics subgroup of the PCI Industrial Computer Manufacturers Group (PICMG) is working on definition of new standards for instrumentation. The MTCA.4 specification, based on the Micro Telecommunications Computing Architecture (μtca), is developed taking into consideration the requirements and needs of the High Energy Tomasz Jeżyński, Frank Ludwig, Holger Schlarb Deutsches Elektronen-Synchrotron Ein Forschungszentrum der Helmholtz-Gemeinschaft Hamburg, Germany desyinfo@desy.de Physics (HEP) applications. The standard allows to design double-width Advanced Mezzanine Card (AMC) modules with Micro Rear Transition Modules (μrtm) and supports backplane distribution of additional signals used in HEP. The μtca-based Controller (μtc) board, shown in Fig. 1, was designed according to AMC standard and follows the guidelines of the drafts of MTCA.4 specification. It was designed to provide an universal hardware platform for implementation of the control systems for the standing-wave linear accelerators. II. SYSTEM REQUIREMENTS The specification for the μtc defines a number of formal requirements, that the hardware is supposed to met. These were driven by the intended application of the μtc in two systems: RF field controller and Beam Based Feedback. The most significant requirements are listed below: Figure 1. Assembled board of μtca-based Controller The board should be designed as a Double-Width, Mid-size AMC Module according to PICMG MTCA.4 specification (Revision 1.0 Draft 0.7), μtc shall provide enough computation power to implement a cavity field controller or a Fast BeamBased Feedback Processor, μtc shall supply non-volatile memory for Field Programmable Gate Array (FPGA), embedded soft processor and Digital Signal Processor (DSP), μtc shall provide RAM memories: QDR memory for fast data buffers and general purpose DDR2 memory, The configuration and control shall be performed using the PCIe interface on the backplane, A gigabit connection for main transmission protocol shall available on the backplane and on the front panel as an optical link, required latency below 200 ns, Module Management Controller (MMC) shall provide Intelligent Platform Management Interface (IPMI) functionality and a firmware upgrade feature, Sockets compatible with SFP standard for high speed serial connections, minimum 4 are required, The controller shall cooperate with a VM designed as a μrtm and provide it with minimum 10 bidirectional, differential pairs and proper management functionality.

2 III. MICROTCA AND RELATED STANDARDS A. Advanced Mezzanine Card The AMC standard defines a building block for the μtca and derivative specifications, and hence shall be described first. The standard focuses on providing a modular, cost-effective platform for various computational needs. It defines cards for extending the functionality of the base system that may come in several form-factors. All modules share the same set of connectivity, management, power and thermal properties and only differ in mechanical dimensions. At the same time the specification imposes virtually no restrictions on the module payload enabling e.g. implementation of high performance single board computers. The AMC module interfaces with the supervising system by means of a 170-pin edge connector. The connector provides the card with electrical power. The device receives two supply voltages: 12 V for data processing payload and 3.3 V for management purposes [1]. The module is allowed to consume up to 80 W. The data link is composed of up to twenty bidirectional differential serial communication channels for application use, which are able to operate with gigabit throughput. The supported communication standards include, but are not limited to: PCI Express, Gigabit Ethernet, Serial RapidIO, Advanced Switching. The front panel space of the modules can be used freely, with exception of the AMC handle and status LED locations. These two elements play an important role in the module management as they are required to provide the full hot-swap functionality. When the card is going to be extracted, the handle position sensing switch informs MMC, that the board should be deactivated. It also triggers the board activation process after the module is again safely locked in place. During normal board operation, the MMC is responsible for thermal and power supply management at the module level. The MMC interfaces with supervising system in accordance to the Intelligent Platform Management Interface (IPMI) specification. Such monitoring and control equipment is a key factor in ensuring high reliability and services availability of these cards. B. Micro Telecommunication Computing Architecture The μtca standard defines requirements for shelves, that can house AMC modules and provide them with the direct connectivity to the backplane. The specification does not enforce any form-factor of the shelf, but instead focuses on providing AMC modules with a sustainable mechanical and thermal environment with access to communication fabric, power supply and management hardware [2]. The backplane fabric organization is not defined in the specification, and may be adjusted to specific system requirements. It may provide AMC modules with direct connectivity between them or with some form of switched fabric. Several different standards and topologies usually coexist on a single backplane to maximize the shelf's flexibility. The backplane is obliged to provide signals required by the MMC and to supply payload and management power. The dedicated shelf management hardware is integrated into a MicroTCA Carrier Hub (MCH) and has an AMC form factor. Its primary function is to control the activation state of the plugged in AMC modules, the links they currently utilize, the power modules load and the operation of cooling units. Apart from that, the MCH usually provides an Ethernet switch or PCI Express root complex. The shelf may offer setup for one or two carrier hubs. Use of two MCH modules improves the system reliability in two ways: by introducing redundancy of the management hardware and by enabling use of redundant switched fabric topologies. In such a setup every MCH is responsible for operating on only a part of all the backplane communication links. The μtca-based systems are at the same time reliable and cost-effective. In consequence, they are gradually gaining popularity in the industrial and scientific control systems. The μtca specification was developed with focus mainly on telecommunication systems, but it provides a firm base, on which more specialized standards may easily evolve. C. AMC, μrtm and μtca Shelf for Physics The MTCA.4 specification, which is now being under development, aims to address the issues which render the μtca standard a suboptimal solution for HEP. One of the most important improvements over the μtca is the introduction of the Micro Rear Transition Module (μrtm). Such an extension board can be connected to any double-width AMC module, if only it is equipped with necessary connectors located in Zone 3. Application of the μrtm module effectively doubles the available PCB area and face plate space, see Fig. 2. Specification suggests using the Advanced Differential Fabric (ADF) connectors, allowing for up to 54 differential pairs to be passed between boards along with a 4-wire JTAG port and an I2C management interface. With proper support from the AMC module, the μrtm can fully benefit from the hot-plug mechanism and provide basic diagnostic functionality. Figure 2. AMC module interfacing the μrtm The second important modification is the definition of a bus consisting of eight bidirectional signals in the Multipoint Low Voltage Differential Signaling (M-LVDS) standard terminated on the backplane. The bus was implemented on existing AMC ports with numbers from 17 to 20. The defined architecture enables some global scope signals (e.g. timing events, interlocks, additional clocks) to be generated not by the MCH, but by any standard AMC module, such as a timing card. Every module may implement an adequate receiver, transmitter or both to take advantage of the multipoint communication. The usage of the M-LVDS bus lines depends only on the application requirements and is not limited by the specification.

3 IV.μTCA-BASED CONTROLLER FOR LINEAR ACCELERATORS A. Outline of Cavity Field Control System The operation principle of linear accelerators requires control of the EM field phase and amplitude along the accelerating cavity at the event of bunch arrival and during its travel inside the cavity. To achieve a satisfactory level of control, several signals from each cavity of the accelerating module are acquired (ref. Fig. 3). The system should maintain the field vectors according to the settings provided by the operator and perform appropriate corrections based on the information received from other beam diagnostics equipment. modules. Then it should calculate the feasible correction using e.g.: PID, MIMO or feed-forward algorithms. The involved complex numbers computations are required to complete in the time scale of about 1 μs. Finally the results are passed to VM. The second mode (case B on Fig. 3) requires several μtc boards to cooperate. The module is supposed to collect data from beam diagnostics devices, such as beam arrival time monitors. These signals have to be processed and the results should be provided to the cavity field controller. In both cases, the μtc board has to deliver computing power for signal processing needs. It should also provide a number of Low Latency Links (LLL) both on the front panel and at the backplane. Moreover, since the control algorithms will be improved with time, the device shall support in-system firmware upgrade, using some fast serial link. Both systems should be easily scalable. There has to be a possibility to increase the system capacity in some simple manner. This can be done either, by installing new modules into existing shelf or by connecting two shelves in a master-slave relationship. C. Comparison with SimCon-DSP The μtc board combined with appropriate data acquisition modules is very well suited to be used in place of the Simulator-Controller (SimCon) board. The SimCon board was designed according to Versa Module Eurocard (VME) standard. The board is equipped with eight ADCs, four DACs, Xilinx Virtex II Pro FPGA and TigerSharc DSP [7]. Although the DSP circuit is the same, the μtc offers greater processing power due to the use of a more advanced FPGA circuit Figure 3. A simplified block diagram of accelerator control system Probe signals that come from the cavity are of the frequency at which the accelerator operates, that is often in the gigahertz range. Before any digital processing the Data Acquisition Modules (DAQ) need to first down-convert them to some intermediate frequency. The resulting signals are then sampled, digitized and their phases and amplitudes are extracted. This preprocessed information is eventually provided to the controller [4][5]. The controller should collect information on phase and amplitude from all the cavities and compare it with the current setpoint. The resulting error term is further provided to regulation algorithms, that estimate the correction that should be applied to the EM field to achieve the desired acceleration parameters. The μtc provides fast FPGA and DSP circuits for implementation of the regulation algorithms. The calculated offsets are sent to the Vector Modulator (VM) module which modifies the I and Q components of a reference signal, which is then amplified and fed to a group of accelerating cavities [6]. B. Board Usage Scenarios The μtc hardware was designed to meet the requirements of the following applications: Regulator of the field stabilizing feedback loop, Fast Beam-Based Feedback (BBF) processor. In the first mode (case A on Fig. 3), the μtc first has to collect information on field parameters from data acquisition Compliance with the AMC standard, instead of the VME, allows to use of the hot-swap mechanism, hence providing much better serviceability. The reliability is improved by implementation of the IPMI software extensions, that provide self-monitoring capability and early warning of e.g. thermal issues. Designing with accordance to drafts of MTCA.4 specification enables implementation of the Vector Modulator as a μrtm module which interfaces directly with the μtc. Use of a smaller form-factor card implicates placing ADC circuits on separate boards. Since the modules may be developed and upgraded independently the system modularity and flexibility is greatly improved. Also, the cost of these processes and time required for their completion are consequently reduced. D. Hardware Implementation The FPGA circuit is the heart of the system. Although the DSP processor significantly contributes to the μtc processing power, it is connected only to the FPGA (ref. Fig. 4), and hence the latter is much more important. The programmable gate array provides the system with three major resources: 16 high-speed serial transceivers, Hardware support for DSP applications (contains slices composed of multiplier, adder and accumulator), Soft processor and other required logic. The FPGA implements all the serial links, using embedded GTP transceivers. The support for high-performance serial protocols is further extended by means of a built-in PCIe endpoint block and four Gigabit Ethernet compatible MACs.

4 E. High-Speed Serial Connectivity The FPGA circuit contains 8 dual high-speed serial GTP transceivers, which enable establishing of up to sixteen bidirectional communication channels of throughput reaching 3.75 Gbps [8]. Due to support of two modes of operation, the controller is obliged to provide even more link options and some of them have to be multiplexed. The exact topology of the links and their multiplexers is shown in the Fig. 6 Figure 4. Block diagram of the μtca-based Controller The μtc core is supported by numerous memories. The FLASH memories are provided for storing firmware, bit streams and other constant data. The Quad-Data Rate (QDR) memory enables realization of efficient queues with separate read and write ports that can be operated simultaneously. Slower Double-Data Rate (DDR) memory may be used by an embedded microprocessor application. The system is supplied by five efficient DC/DC converters and several linear low-drop voltage regulators. All the power supplies are controlled by the management hardware. Figure 6. The μtc high-speed and low latency links μtc offers the following data transmission channels: PCI Express x2 on AMC connector for accessing the controller settings; this link may be configured to x4, Gigabit Ethernet on the front panel and at the AMC connector, for general purpose connectivity, Two low latency links for the μrtm module, Six, up to ten, low latency links at the AMC connector, Up to six low latency links at the front panel in form of SFP transceiver sockets. F. Clock Distribution Scheme High-speed interfaces require stable low-jitter clock signals. The μtc board contains an extensive set of generators, synthesizers, multiplexers and fan-out buffers. Their topology is briefly depicted in Fig. 7. Figure 5. Printed Circuit Board layer stack-up of the μtc The board supports a number of BGA circuits and contains high-speed differential serial links. All the vias are filled and hence may be located in the component's pads. To ensure proper clearances between tracks, the μtc PCB was designed with 5 internal signal layers. Routing signals on multiple layers and widespread application of the differential signaling reduces cross-talk. Moreover, to further avoid potential signal integrity issues, the signal layers, with one exception, are separated with at least one ground plane. The board contains two supply layers and external layers for components connection. Together they form a 1.64 mm thick stack-up shown in the Figure 5. Signal separation with ground planes improves the noise performance, but at the same time causes a serious problem. To achieve the 100 Ω differential impedance, the traces at the internal layers are required to have a width of only mm (3 mils). The configurable clock generator contains a PLL and a set of dividers. Due to its high flexibility it is used to provide clock signals for many important system components e.g.: FPGA, DSP, GbE PHY. The jitter of this clock may be too large for the reliable highest-speed transmissions, but is satisfactory as a general purpose clock for various peripherals. The dedicated telecommunication clock may be sourced from 125 MHz precise integrated oscillator which is the preferred frequency for implementing the Gigabit Ethernet links [8]. The dedicated clock for PCI Express is supplied from AMC connector and passed through jitter attenuator based on a PLL. Such a solution enables using the spread spectrum mode, reducing the potential electromagnetic interferences related to the operation of this serial bus.

5 H. System Management Hardware The μtc board implements the Module Management Controller (MMC) required by MTCA.4 and AMC specifications. In fact, it extends its capabilities in various aspects. The simplified block diagram of the μtc management circuity is shown in the Fig. 9. The Intelligent Platform Management Interface (IPMI) is handled by an 8-bit AVR microcontroller, which is the heart of this subsystem. The MMC firmware is described in a separate article [9]. Figure 7. Main clocks of μtc The board offers several other general purpose clocks. One is fed through SMA connector, conditioned and provided for FPGA and μrtm module in an LVDS standard. Another one is selectable between local generator, μrtm clock and one of AMC M-LVDS clocks. Two more clock signals, that are sourced by μrtm, are passed directly to the FPGA of the μtc. G. Firmware Upgrade Support Almost all the non-volatile memories can be accessed using the JTAG interface. This solution provides a way of loading the firmware for the first time and simplifies crash recovery, if one should happen. Nevertheless, since the μtc is designed for operation in the accelerator tunnel, after the installation it will be impossible for operator to access it for manual reprogramming. Even if possible, this upgrade scheme would be extremely inefficient. Consequently, the μtc is required to provide an in-system firmware upgrade feature. The in-system firmware upgrade feature is implemented in the μtc with assistance of the MMC, see Fig. 8. The FPGA may be boot loaded using the dedicated Platform FLASH memories or general purpose SPI FLASH memories. The selection is made accordingly to the state of the 'mode bits', which are provided by the management hardware. In case of booting from SPI FLASH, the selection of the memory that Figure 9. μtc Module Management Controller and its environment The microcontroller monitors the most critical signals, such as overheating warnings, using its own GPIO lines. All the hardware required for the hot-plug mechanism also interfaces directly with the MCU. This method is fairly fail safe, but the number of available I/O lines is significantly limited. Consequently, several management functions are provided using accompanying circuits that are better suited for some specific operations. The microcontroller functionality is extended with the following dedicated sensors and actuators: Power management circuits connected to I2C bus monitor the supply voltages and overall current consumption of the controller, the μrtm power supplies are keyed and their load is also measured, The CPLD circuit provides a simple serial interface for controlling the FPGA configuration process, highspeed serial interfaces multiplexers state, JTAG chain configuration and presence of several clocks, The temperature sensors are monitoring temperatures in several PCB areas and directly at the FPGA die. The MMC of the AMC module is also required to provide the management functionality for the supported μrtm modules. The μrtm may be identified by reading its serial EEPROM memory, located at a known I2C address. If the module is not recognized, it shall not have the payload power enabled. In the opposite case, the module may be activated and its links and clocks enabled. Figure 8. FPGA and DSP configuration options will be used is also done by the MMC. After booting from the memory that contains the boot loader bit stream, the FPGA may load any SPI memory with firmware received over the PCI Express or Gigabit Ethernet. I. μrtm Interface The MTCA.4 specification suggests connecting the μrtm module using two ADF connectors. Although many choices of sockets are possible, the use of the proposed ones improves interchangeability between vendors and increases chances for

6 the design to be reused. There are two sizes of the ADF connectors, the larger one offers space for 30 differential pairs. The MTCA.4 specification draft reserves 6 pairs for power and management purposes, hence the connectors may provide up to 54 pairs for application-defined differential connectivity. The controller implements the following LVDS and LVPECL links to the Vector Modulator realized on the μrtm module: 2 pairs of high-speed bidirectional low latency links, 20 pairs connected directly to the FPGA for implementation of custom communication interfaces (e.g. 16-bit wide bus with four control signals), 6 clock signals exchanged with the μrtm. V. MEASUREMENTS AND TESTS RESULTS The μtc hardware was tested in the laboratory at the Department of Microelectronics and Computer Science and in the Deutsches Elektronen-Synchrotron research facility. The board was housed in two different MTCA.4 prototype shelves. Some of the measurements were also performed outside the crate, with use of the modified MMC firmware. A. Power Supplies The μtc has five DC/DC converters providing six supply voltages, all of them are regulated with accuracy better than 3 %. Four further supply voltages are provided by low-drop regulators fed from 1.8 V provided by DC/DC converter. The resulting voltages have accuracy better than 2 %. B. High-speed Serial Links All the high-speed serial connectivity was tested in the laboratory using a prototype μtca for Physics shelf. The PCI Express transmissions were performed with the assistance of MCH and CPU modules. The Gigabit Ethernet interfaces were tested with an ICMP echo application running on a MicroBlaze platform. The μrtm links were tested using loopback test fixture. The Low Latency Links, both on the front panel and backplane, were tested in communication between a pair of μtc boards achieving BER of about bit/s. C. Non-volatile Memories All the SPI FLASH memories are working correctly. The SPI memories can be used for configuring the FPGA. This process takes about 15 s. The FPGA circuit is able to program any of these memories using a developed PCIe boot loader. The bit stream for the FPGA circuit is larger than the capacity of the largest Platform FLASH available at the time of entering the design, hence two pieces of them were used. The FPGA can also boot from any of them, using the compressed bit stream. D. Random Access Memories The μtc contains DDR2 and QDR II / QDR II+ memories. The DDR2 memory is used as a general purpose RAM for soft processor that may be implemented in the FPGA. It has been tested with custom application running on a MicroBlaze-based platform. No malfunctions were found. The achieved throughput is: about 26.3 MB/s for sequential write and about 11.8 MB/s for sequential read operations. The QDR II+ memory controller cannot be automatically generated by the FPGA design environment. The dedicated IPcore is now being developed. Suitable performance tests will be performed after its completion. E. Digital Signal Processor The link ports of the DSP are used as a communication channel with the FPGA. A full-duplex double data rate transmission with a 125 MHz clock is currently used. Taking into account that the link port is 4-bit wide, the available throughput is 1 Gbps. However, since the transmission unit is a quad-word (128 bits) and the transmit buffer needs to be reloaded after each quad-word is sent, the real throughput is approximately 960 Mbps. That is because a transmission of a single quad-word takes 16 clock cycles, resulting in 128 ns delay and then the reloading consumes another 5 ns. The transmission latency measured between loading the transmit buffer of the DSP and receiving the first bit in the FPGA is around 130 ns. This makes the full quad-word available about 250 ns after loading the transmit buffer. VI. SUMMARY The μtca-based Controller board is one of only few devices that were designed with accordance to drafts of the MTCA.4 specification. The μtc provides considerable processing power, large number of high-throughput serial links and flexible configuration schemes. This, combined with high reliability and serviceability renders μtc an universal platform, well suited for implementation of accelerator control equipment. It is currently being evaluated at DESY in the Low Level Radio Frequency (LLRF) system. ACKNOWLEDGMENT The research leading to these results has received funding from the European Commission under the EuCARD FP7 Research Infrastructures grant agreement no The authors are scholarship holders of project entitled "Innovative education..." supported by European Social Fund. REFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9] PICMG AMC.0 R2.0 Advanced Mezzanine Card Base Specification, PICMG, November 15, 2006 PICMG MTCA.0 R1.0 Micro Telecommunications Computing Architecture Base Specification, PICMG, July 6, 2006 Design of the digital RF control system for the TESLA test facility, S. N. Simrock, I. Altmann, K. Rehlich, T. Schilcher, EPAC 96, Spain LLRF Control System Upgrade at FLASH, V. Ayvazyan et al., PCaPAC 2010, Saskatoon, Canada Low Level RF system for Jefferson Lab cryomodule test facility, T. Plawski, T. Allison, J. Delayen, C. Hovater, Tom Powers, PAC 2003 Achieving Phase and Amplitude Stability in Pulsed Superconducting Cavities, S. N. Simrock, PAC 2001, Chicago SIMCON, ver Initial Results for 8-Channel Algorithm, W. Gierusiewicz, W. Koprek, W. Jalmuzna, K. T. Pozniak, R. S. Romaniuk, ELHEP Group, IES, WUT, Warsaw, Poland Virtex-5 FPGA RocketIO GTP Transceiver User Guide, June, 2009 Module Management Controller for MicroTCA-based Controller Board, P. Perek, A. Mielczarek, P. Prędki, D. Makowski, A. Napieralski, MIXDES 2011