LASER INTERFEROMETER GRAVITATIONAL WAVE OBSERVATORY -LIGO-

Transcription

1 LASER INTERFEROMETER GRAVITATIONAL WAVE OBSERVATORY -LIGO- CALIFORNIA INSTITUTE OF TECHNOLOGY MASSACHUSETTS INSTITUTE OF TECHNOLOGY Document Type DCC Number July 7, 2005 AdvLigo CDS Discussion Paper R. Bork Distribution of this draft: This is an internal working note of the LIGO Laboratory California Institute of Technology Massachusetts Institute of Technology LIGO Project MS LIGO Project MS 20B-145 Pasadena, CA Cambridge, MA Phone (626) Phone (617) Fax (626) Fax (617) www:

2 1 Purpose The purpose of this paper is to present some thoughts on a possible controls infrastructure for AdvLigo. It is presented solely as a basis for discussion. 2 Overview The remaining sections of this document are organized as follows: Analog Front End: This section describes the proposed analog to digital interfaces of the CDS system. These are separate units which reside in the same rack enclosures as the remaining analog devices which connect to the field sensors and drivers. Processing layer: This section describes the CDS computers and their connections to the analog front ends. Network layer: This section describes the interconnection of CDS computers. There are presently LIGO fast systems, synchronized at 16KHz and 2KHz, and LIGO slow systems, 10Hz or less EPICS systems. Most systems are some combination of fast and slow, with separate slow auxiliary crates. The proposal here is that future AdvLigo systems be either fast or slow, not a mix. For example, LSC would have only fast signals, no auxiliary crate with slow signals. As an example of this, the present HEPI system has no slow or auxiliary signals. To avoid the design and implementation of a second Ligo Bus to handle strictly slow systems, such as the PSL, slow systems would use the exact same design as the fast system proposed here. The only difference may be in the uplink between the analog front ends and the CDS processors.

3 3 Analog Front End The desire in AdvLigo is to separate the control processors from the ADC/DAC devices. This means the removal of VME systems presently used in LIGO. It is also desirable for CDS to produce its own ADC/DAC modules. The reasons for this have been pointed out in other documents, with the primary driver being that low noise units required by Ligo have not been readily available COTS units. The following subsections describe some ideas for replacement of the current VME front ends and example layouts. 3.1 Front End Bus Presently, many analog front end modules are laid out in 6U Eurocard format, with fast signals off the front panels to VME modules and slow signals off the back edge connector to cross connect blocks. The first proposal here is to combine the analog modules with ADC/DAC modules into a single Eurocard cage, as shown in the following figure Housing In this design, the cage is expanded to allow cards to be inserted front and rear, with a center backplane. The Application Modules or analog modules, such as whitening, AA, AI, dewhite, etc., are inserted from the rear of the crate into the backplane. The Ligo Bus Modules (LBM) refers to ADC/DAC modules, inserted from the front. The modules would be 6U, with doors front and rear to seal the enclosure. The crate itself may be 9U or higher. This would allow an additional space at the bottom, with custom connection panels, to allow the introduction of cabling to field devices without interfering with the extraction/insertion of cards Backplane The backplane itself only carries power and timing signals, connected to each module in the crate via the P1. This unit would not have an internal power supply, but would rather be powered externally. Timing signals would come in via fiber to a Ligo Timing Receiver module, with signals either across this P1 connector, a separate P0 connector, or, alternatively, ribbon cabled across the front of the Ligo Bus modules.

4 Signals between the analog modules and the LBM would be carried across the P2 (I/O) connector. This is a 64 pin connector which feeds through, but is not bussed, across the backplane. The lines of the P2 would be standardized for all modules. As an example, the top 32 pins may be all analog I/O and the bottom 32 devoted to binary I/O. 3.2 LIGO Bus Modules The LBM are 6U cards with ADC/DAC I/O. These modules interface with the application modules via the 64 pin P2 backplane connector and to a data uplink module via a Front Panel Data Port (FPDP) ADC/DAC with Binary I/O A basic block diagram of an ADC/DAC module is shown in the following figure. This idea is that this become a generic module, which is directly mated to an application module. For example, an 8 ch ADC LBM might interface via the P2 connector to a suspension whitening/aa module. An 8 ch DAC might interface directly to a suspension 8 ch AI module. As described in examples in section 3.3, eight channels of ADC/DAC may be the best density to suit most applications. This should allow plenty of space in a 6U module to implement. Each LBM would include an FPGA. This would be required to initialize the ADC/DAC chips, handle the timing, provide programmable binary I/O and send/receive data via an FPDP interface. The FPGA may also be used to perform some filtering functions on the data as well. In operation, this FPGA performs the necessary initialization and then continuously produces/receives data at the Ligo clock rate as provided by the timing signals. No external setup/startup commands are to be required. I don t believe separate binary I/O modules would need to be developed. There are some instances where an application card, such as a 4 ch dewhitening module, only requires binary I/O, but this can be done differently. This is described in the examples section later. Also, it should be standardized that the analog application modules be either input or output, not a mix. This is already a usual case. If it is necessary to have mixed I/O, then perhaps the application module could be designed as a double wide, with a LBM ADC in front of one slot and a LBM DAC in front of the other.

5 3.2.2 Digital Interface The digital interface for the LBM is FPDP. This is a VITA standard with several advantages: 32 bit data width High speed (to 247Mbytes/sec) Simple protocol which is simple to implement (2 chips) Low latency, which is as important in the Ligo control applications as bandwidth One downside of FPDP is that it is uni-directional ie separate transmit and receive buses must be provided. This could be overcome with some additional logic which switches the bus between receive mode and transmit mode, but is probably not necessary in the Ligo applications Uplink Module The uplink between the Ligo analog front end and CDS processors would be Serial FPDP(SFPDP). SFPDP has a number of advantages for Ligo high speed systems: Up to 247MByte/sec transfer rates. Low latency Simple protocol Bidirectional FPDP to/from SFPDP cores readily available for FPGA. Fiber Optic coupling, with transmission distances to 10km. COTS PCI interfaces available for the processor end (up to 4 ports/pci card). Open standard. The primary disadvantages of SFPDP are: Point to point link. While there may be multiple receivers of data, only one transmitter is allowed. This protocol cannot be used in a switched fabric or share network links with other protocols. Limited number of COTS vendors. For Ligo high speed applications, I believe the low latency is the key factor in going with SFPDP. While something like standard fiber channel would produce similar data rates, I have seen latencies quoted in the hundreds of microseconds, which would be totally unacceptable. A module which does what is shown in this proposed layout is commercially available in a 6U VME format. It only requires power, so could be directly used in this analog front end if power is defined properly on the P1 connector. This may be an option, particularly to start with, so we could concentrate on the ADC/DAC modules for now and design our own SFPDP module in the future. For slow (16Hz EPICS) applications, this could possibly be a fiber optic Ethernet link. This would allow the use of standard hubs and switches. In this fashion, the front end I/O is not drastically different between fast and slow systems and does not require the development of a separate LIGO slow data bus architecture Timing Module A standard Ligo timing module would be provided to synchrously operate these analog front ends. The connection to the timing system would be via fiber optics. Connections to the LBM would be via the backplane (P1 or P0), or through a separate front panel ribbon connector. The connection to LBM would carry a 1PPS synch and clock. It may also require some additional signals to sych up data insertion/extraction to the FPDP bus.

6 3.2.5 Example Crate View The following figure depicts how the front of one of these crates might look with the LBM inserted. ADC modules would have their FPDP connectors on the bottom and DAC modules would have their connections on the top. This is done to separate the transmit and receive FPDP buses coming from the SFPDP module. With this configuration, the only connections out of the analog front end for the LBM would be two SFPDP and one timing fiber optic cable Alternative Layouts There are some alternatives to putting the analog modules and the LBM in the same crate. One is to simply separate the modules into separate crates with interconnecting cables. This, of course adds cables and connectors and more RFI shielding. The LBM could then also be housed in rack mount chassis instead of a Eurocard format. The problem I see with this approach would be in shielding the FPDP cables which would interconnect these chassis. If each chassis had its own SFPDP output, then the number of fibers and connections to the proper processors at the back end could become a problem (primarily cost of receivers at the computer end of $3K/channel).

7 3.3 Example Layouts As a bit of a fit check, I ve tried to apply this basic architecture to a few future/existing systems. These are described is the following subsections. In the figures shown, the top part of each diagram is the front view (LBM) of a crate and the bottom is the back view (application modules) Quad Suspension Controls The following figure depicts what a quad suspension controller crate may look like in this format. It uses the present design of suspension analog modules for layout purposes, with the exception that whitening boards are shown as 8 channels, while they are 5 channel devices now. As can be seen, the layout is such that each application module inserted from the rear is mated with a LBM inserted from the front. To avoid FPDP send/receive cable overlaps, input devices are placed to the left and output devices to the right. In this example, no LDM line up in front of the Universal Dewhite modules, which require binary switching conditions. The suggestion here is that future designs of the AI carry binary I/O over to the dewhite modules along the same ribbon that carries the analog outputs. Also, the ADC/DAC modules are labeled with Binary Outputs (BO), but would be a configurable combination of BI/BO, as necessary for the application module.

8 3.3.2 Hepi Controls The following figure shows a possible HEPI analog crate layout for the corner station controls. HEPI may be an example of a system where it may not be desirable to redesign the present analog electronics into a Eurocard package. So, in this case, the application modules inserted from the rear are simply cross connect modules in that they take the present HEPI connector which goes to a VME module and convert it to the 64 pin connection of the LBM.

9 3.3.3 End Station ADCU and HEPI As a final example, a combined ADCU and HEPI crate is shown for an end station. This, and other ADCUs, could use the same cross connect module as shown for HEPI above to make use of the present 16 channel connector from the existing AA chassis. Or, as shown here, an 8 channel AA module could be developed, with 8 BNC or 4 Limo connections to bring in the signals. This crate would have separate uplinks for data and a common timing module.

10 4 Processing Layer The processing layer takes the inputs via SFPDP, performs the desired control algorithms and outputs control signals back to the analog layer, once again over SFPDP. A layout which matches processors with analog front end crates is shown in the following figure. 4.1 SFPDP Interface There are COTS SFPDP PCI/PMC modules available from at least three vendors. They come in configurations of 1, 2 or 4 SFPDP links per PCI card. Each SFPDP has a transmitter and receiver and therefore could communicate directly with up to 4 analog input devices and 4 analog output devices or 4 mixed control devices( devices here referring to analog front end Eurocard crates). The multi-channel units also come with an FPGA, which can be further programmed to do tasks in addition to SFPDP, such as decimation filtering, whitening filtering, etc. Since it is an FPGA with an SFPDP core, it could be reprogrammed to handle some other protocol if desired in the future.

11 While some sort of data concentrator or switch has been suggested in the past, but may not be a viable solution. For one, SFPDP is not directly switchable and other protocols through commercial switches would have a high latency. Another issue is that the Ligo data must run at a high bandwidth. While the overall average bandwidth is not excessive, in order to provide sufficient processing time, the I/O traffic to/from analog interfaces cannot consume more than 10usec of the 61usec cycle time of a 16KHz control system. 4.2 Computers The CDS computers would reside in the MSR, away from the analog front ends. Perhaps the fastest machines out at the moment are AMD64 based computers. By the time of AdvLigo or even Ligo 1.5, who knows what will be available. Given that an AdvLigo Quad Suspension requires roughly 4 times the processing of a present optic controller, much more CPU power will be required for AdvLigo. The move will continue toward using Linux on all control processors. While we have managed to stay away from requiring a realtime Linux version, that may or may not be the case in the future. RT Linux may buy us some added capabilities, particularly in multiple CPU computers. A multiple CPU unit would be useful in that the realtime front end code could run on one CPU, using SFPDP for communications, while the EPICS which supports that subsystem runs on the other CPU and uses Ethernet communications. This would alleviate the front end to/from EPICS traffic that we presently have on our RFM networks. 4.3 Application Software Once the interface to the analog front ends is set, then a graphical code generator can be further developed. A prototype tool presently exists for VME systems, using the EPICS VDCT tool and a few Perl scripts. The VDCT tool is used to graphically capture the desired control algorithms and then the Perl scripts generate the realtime C code and supporting EPICS databases and sequencers. 4.4 Timing The control process is driven by the availability of data at the SFPDP interface. The code will essentially wait for an interrupt or perform some sort of polling. This is not unlike what is done in VME now. 5 Network Layer The network layer here is defined as that mechanism used to communicate data between the CDS processors. The choices appear to be to stay with the present RFM scheme or use a more traditional network that allows switching of traffic. One switch fabric network is presently being tested as a possible replacement for RFM. Initial tests show data latencies of about 10usec with high data rates (2.5Gbit/sec). The cost would be approximately half of the present RFM system. The primary test of such a system would be to blast the data at the Framebuilders over such a network. With the RFM, the data can come in asynchronously and then be captured by the Framebuilders as a single copy of 1/16 second of data. In a more traditional network scheme, large buffers are not provided therefore the Framebuilders may become busier just moving data in from the network.