The Use of LabVIEW FPGA in Accelerator Instrumentation. Willem Blokland Research Accelerator Division Spallation Neutron Source
Introduction Spallation Neutron Source at Oak Ridge National Laboratory: $1.4B facility using a 1 GeV proton beam to generate pulsed neutrons Built by a collaboration of five national laboratories Neutron scattering to study materials 2
Introduction The Beam Instrumentation Group at the Spallation Neutron Source is responsible for measuring various attributes of the particle beam, such as losses, electrical charge, and position. Many instruments can and are implemented with a general OS (XPe). However, some measurements must be done consistenly and quickly (16ms) to protect the machine or may require very fast reaction times (10 µs) We began using FPGA on NI Hardware as well as LabVIEW FPGA for various instruments: Beam Accounting systems that use FPGA and LabVIEW but was directly programmed with the Xilinx tools (fast charge calculator and differential monitor) A Beam Loss Monitor that was programmed in LabVIEW FPGA on crio A new version of the Beam Accounting system, using FlexRIO and LabVIEW FPGA 3
Current Current Current Current Spallation Neutron Source Beam Pulse At SNS: The beam pulse repeats at 60 Hz Beam is chopped at ~1 MHz RF is 402.5 Mhz and 805 MHz 16.6 ms 945 ns Chopper system makes gaps mini-pulse Accumulated in the Ring 1 ms macro-pulse 2.5 ns 1ms 4
Accelerator Beam Instrumentation Features 400+ PCs 16+ VME systems 5
Accelerator Beam Instrumentation Features A beam instrument s function can be one of the following: Measurements to a studier: Is my accelerator setup better or worse? Only needed during studies Interactive (1s) Measurements to an operator: Is the accelerator operating within specifications? 24/7, Needed during production Interactive (1s) and/or automatically monitored Measurements for beam accounting: What are the statistics? 24/7, Needed during production Automatic (<16.6ms) Measurements to the abort system: Is something going wrong? 24/7, Needed during production Automatic either with alarm or direct hardware connection to abort system (e.g. abort beam in less than 20µs) 6
Example of Beam Instruments SNS Wire Scanner (PC, LabVIEW) Steps a wire through the beam Tied to a max beam rep rate of 1Hz to protect wire Only used when performing a study Single failures allowed Beam Position (PC, LabVIEW) Measures the position of the beam pulse Only needed at 1Hz (beam rep rate is 60Hz) Digitizes at 40Mhz Single failures allowed Beam Loss Monitors (VME, VxWorks) Measures beam losses to protect machine Software must run at 60Hz Analog hardware to abort beam within 20 µs No failures allowed (exceptions can be made) Fork BPM Plates Neutron detectors 7
Accelerator Beam Instrumentation Types Always Uptime Hardware Circuits, FPGA, DSP,.. Consequences Damage Mostly VxWorks, LVRT, PLC, Real-Time Non RT standalone Machine Downtime Now and then Win, Mac, Linux, µs ms s Non RT interactive Time lost Response time 8
How Do We Implement An Instrument? We are interested in the fastest and cheapest way to implement a reliable instrument Small group can t afford to know everything Budget can be used to buy time from contractor with needed expertise Work with other groups within division Have to work with different parts from different vendors Real-time OS, Compilers, FPGA Board, Digitizer, Slotzero controller, Backplane, Amplifiers Maintenance Solutions If deployment is large enough and time is available, then learning curve is not as significant and inhouse hardware can be developed Buy turn-key system Buy off-the-shelf parts and integrate to complete Considerations Cost Available personnel Development time Reliability Performance Obsolescence Maintenance 9
Use of FPGA in Accelerator Instrumentation These are unique instruments, one or two copies of each at most. We have used LabVIEW FPGA for various small projects, such as: Laser position feedback loop Pulse delay generator Accelerator Instruments: Pulse-by-Pulse Beam Charge Monitor (PBCM) Beam pulse Calculate the charge of a pulse and transmit digital value over optical fiber within 2-3 ms to Target Hall Instruments Differential Beam Current Monitor (DBCM) Subtract the signals of two current transformers and determine if the difference is too large (5, 50, 500 µs sliding windows). Abort beam within 5 µs. For these instruments LabVIEW FPGA was not available with a fast enough digitizer. We choose to use COTS parts (NI PCI-5122EX), but involve a contractor (Maverick Systems) to write the Xilinx FPGA code. 10
PBCM Setup PXI crate with 5122EX To target instruments PC with EPICS and PCI Timing card Toroid in tunnel 11
PBCM Singal Processing The FPGA on PXI version of digitizer calculates independently from LabVIEW and OS the charge and beam pulse edge. Processing of each channel includes: Baseline correction, FIR/IIR filter or no filter or test pattern, Charge integration, and Detection of beam pulse edge. Fiber-optic Transmission includes: Transmission of the charge and begin of beam pulse over two digital outputs, and Electrical to optical conversion (Optical converter box by R. Riedel) to transmit value to the Target Group for correlation with neutron measurements. 12
PBCM Use For Commissioning To target instruments 13
DBCM Setup Differential Beam Current Monitor Red trace is difference Chopped Beam Beam with 5% beam loss (zoomed) PXI with EPICS and 5122EX card Sliding window traces 14
Flow Chart Of DBCM Signal Processing The FPGA on PXI version of digitizer calculates independently from LabVIEW and OS the difference between two current monitors Processing of each channel includes: Baseline correction, FIR/IIR filter or no filter or test pattern, Charge integration (5, 50, 50 µs) Abort includes: Digital Out to Machine Protection System 15
Experiences with 5122EX (PBCM CBCM each) Contractor 9-12 months (not full time) Labor cost ~$85k Learning Curve Taken care of by contractor Hardware $17k Specification 2 weeks (within group: no cost) Integration to Control System 2-3 weeks (within group: no cost) Upgrades Any change would require outside help as expertise is not within group -> high cost and time overhead 16
Beam Loss Monitor One unique instrument for use in a test facility. Similar to a standard Beam Loss Monitor, but resources to modify existing systems were scarce. crio was identified as viable alternative. RF Test Facility Beam Loss Monitor Specifications Sample 12 channels at 100kHz Baseline correction, calibration, and integration Alarm in 20usec when integrated beam loss is above threshold Control and read back HV power supplies voltages Send test pulse to test Photo Multiplier Tubes Feature VME crio Sample rate 100kS/s 100kS/s Resolution 24 16 Alarm Analog FPGA Analog Channels 16 12-16 Hardware Cost* $29k $12k-13k * CPU, Crate, Power supply, ADC, DAC, DIO 17
Beam Loss Monitor crio 18
Beam Loss Monitor Diagram Impressive: Adding test feature required minimal code and development time 19
Beam Loss Monitor crio BLM IOC CPU with EPICS BLM crio BLM Amplifiers HV Power supply RF Cavity Video System PXI crate 20
Beam Loss Monitor VME Rack Abort System Modules VME Controller, Digitizer, Digital IO, Custom Timing Card Custom Analog Card HV modules Power Supply Analog Front-end Analog front-end 21
Experiences with crio-based BLM Contractor 3 months part-time (to get started) Labor cost ~$7.5k, also covers LabVIEW RT setup Learning Curve Sample-by-sample processing versus standard Setting up LabVIEW FPGA Hardware $12k. Specification 1 week (within group: no cost) Integration 1 week (within group: no cost) Upgrades Easily done for small changes ~1 day 22
Comparison BLM VME programming versus LabVIEW FPGA & RT Sample-by-sample processing fitted requirements very well and got rid of issues such as Sequential processing bottlenecks: Waveform acquisition takes 2 ms, then the transfer to slot zero, then processing. This must fit within 16.6msec. Balancing act between digitizing speed, backplane data transfers and CPU processing. FPGA can process data sample-by-sample without data transfers. Separate hardware board to perform Abort functions. An analog board with integrator and analog threshold is used to implement an abort within 20 µs. FPGA can set alarm in each 10 µs cycle. crio Hardware integration takes care of low-level drivers Our project requirements matched very well with crio capabilities and no FPGA hardware knowledge was required. VME approach is standard in accelerators Custom timing decoding card and software is available Many groups (Controls) have experienced people Floating point math Multi-year project (multiple labs) 23
FlexRIO FlexRIO is now available and the Differential Current Monitor can now be implemented with LabVIEW FPGA. The goal is to upgrade the DCM (change trigger behavior and simplify interface) and improve maintenance but stay with a 100Mhz single cycle loop (aka 1000x faster). The implementer is a student in the Physics Bachelors Program with some LabVIEW background. NI provided with a proof of principle example program bringing us 90% to completion (Of course, the last 10% is at least 90% of the work) NI Support during development Integration Windows Courtesy of Mariano Padilla 24
Experiences Using FlexRIO at 100MHz Student 2.5 Man-Months in 4 months, about 0.5 MM to complete Labor cost <$10k Learning Curve (~60%) Point-by-point processing versus standard FlexRIO details Hardware $14k. Specification 1 week (within group: no cost but time) Integration Estimated at 3 weeks (within group: no cost but time) Upgrades Easily done for small changes (~1 day) 25
FlexRIO Experience Experiences with 100MHz single cycle loop: Detailed knowledge of LabVIEW FPGA implementation is required: Number of pipeline delays Fixed point math and exact conversion details Detailed knowledge of FlexRIO hardware is required: Being able to estimate most likely utilization of FPGA resources Knowing how DDR memory is interfaced: how many channels/timing No Xilinx expert required Iterate many times to complete project 26
FlexRIO Experience When performance requirements fit well below the maximum performance of what LabVIEW FPGA and hardware supports, you can expect fast development with a good hiding of the underlying technology. When challenging the performance, development becomes slower and more complex. BUT It is an integrated development environment by one vendor Can cover at least up to 100Mhz 27
Comparison Xilinx crio FlexRIO Processing Speed 100 MHz 100 khz 100 MHz Project Length 12 Months 3 Months 6 Months (Learning) Expertise Labor/Hardware Cost ratio VHDL FPGA DSP Hardware FPGA DSP FPGA DSP Hardware 5:1 1:1 1:1 (Student) 28
Future LabVIEW FPGA Based Diagnostics Instrumentation Systems with > 1MS/s, reaction within 10 µs, rep rate of 60 Hz: Pulse-by-Pulse BCM Systems with 100kS/s 1MS/s, reaction within 100 µs, rep rate of 60 Hz: BLM Charge/Power Measurements Long Term: Wire Scanners Harp Image Analysis Damper (1GHz) 29
Conclusion LabVIEW FPGA has enabled us to cover a wider range of instrumentation developed and supported within our group LabVIEW FPGA FlexRIO LabVIEW FPGA crio Uptime Consequences Always Hardware Circuits, FPGA, DSP,.. Damage Real-Time Mostly VxWorks, LVRT, PLC, Non RT standalone Machine Downtime Now and then Win, Mac, Linux, µs ms s Non RT interactive Time lost Response time 30