System Engineering, Moore s Law and Collaboration: a Perspective from SKA South Africa and CASPER

Transcription

1 System Engineering, Moore s Law and Collaboration: a Perspective from SKA South Africa and CASPER Francois Kapp, Jason Manley SKA SA - MeerKAT francois.kapp@ska.ac.za, tel: Abstract: The Digital Back-end group at SKA South Africa are designing sub-systems for the MeerKAT telescope (and possibly for the Square Kilometre Array) using a System Engineering approach with the double classifier of relying on the continued advance of processing along the Moore s Law trajectory; and working in collaboration with institutions from around the world. This paper describes the adventures and misadventures of designing radio astronomy instruments for the future, using contributions from independent collaborators and open source tools. We explain how we view technology road maps, what we look for in technology readiness and how we align specifications of hardware with incomplete system specifications. We also detail our flexible system architecture design that is at the core of tying together architecture blocks of varying maturity; and key to our view that processing hardware is an operational expense. Introduction Radio Telescopes are traditionally designed in an academic environment, funded by governments and implemented by industry conglomerates. This results in long development times (typically of the order of 10 years), challenging integration problems and schedule overruns with associated increase in cost. In addition, they have typically relied on one or a few experienced champions to achieve a working system. As the system complexity increases, partly due to the ability of signal processing to achieve system characteristics not affordable if built using a brute force method, this approach is not viable anymore. System Engineering offers a promising opportunity to help solve these problems by systematically breaking down the complexity into manageable portions. The challenge in applying System Engineering to Radio Astronomy is that it is a changing field. There are always new insights developing about instrumentation and the scientific goals evolve over timescales that are short compared to the development time. At the same time the available technology is also rapidly evolving. If an instrument could be deployed with the latest possible technology, the capital and operating costs could be substantially reduced. This realisation gave birth to the Collaboration for Astronomy Signal Processing and Electronics Research (CASPER), which emerged from U.C. Berkeley. The digital team at the MeerKAT project chose the CASPER architecture as a basis for

2 development of the correlator sub-system (sometimes known as the Digital Back-End, or DBE) of the MeerKAT telescope. CASPER Goals From the CASPER web site, the goal statement reads: The primary goal of CASPER is to streamline and simplify the design flow of radio astronomy instrumentation by promoting design reuse through the development of platformindependent, open-source hardware and software. Our aim is to couple the real-time streaming performance of application-specific hardware with the design simplicity of general-purpose software. By providing parameterized, platform-independent "gateware" libraries that run on reconfigurable, modular hardware building blocks, we abstract away low-level implementation details and allow astronomers to rapidly design and deploy new instruments. CASPER Philosophy As a direct result of the remarkable and relentless advance of silicon devices an observation first made by Gordon E. Moore in 1965 processing becomes cheaper very rapidly. If the technology procurement can be delayed by one process node, savings in space, power and other metrics can often make the difference between a viable and non-viable system design, or at least reduce capital and operational costs substantially. But in order to take advantage of this, a way must be found to reduce development time. Any reduction in development time can then be used to delay procurement of hardware sometimes known as technology freeze. CASPER proposes a path to reduced development time, by providing ready solutions at multiple layers of system design and implementation. One can draw parallels between the Personal Computer development and CASPER. CASPER would ultimately want to commoditise processing hardware, so that the cost is driven down this is similar to the way PC costs dropped when the market adopted a standard design en masse. CASPER would also like to be able to re-use software across multiple generations of hardware very much like PC industry s long term support for the X86 instruction set. Clearly the entry level into the PC industry and that of Radio Astronomy are orders of magnitude apart, but to say that CASPER creates the PC of Radio Astronomy would not be inappropriate. Firstly, hardware platforms are designed to be generic and extensible within the Radio Astronomy signal processing domain, so that they can be used as building blocks for larger systems. Secondly, on top of this commoditised hardware, comes another innovation at least in the Radio Astronomy world the use of an off-the-shelf Ethernet switch as a backplane. In recent history, Ethernet has become one of the longest living standards in interconnect. Even though data rates have gone up by orders of magnitude, it is still easy to interconnect older hardware with the latest generation. Taking advantage of this means that interconnect is via cable rather than backplane. And finally, none of these innovations would have helped without a software development method that enabled Rapid Application Development as well as re-targeting of hardware platforms.

3 When all these items come together, it starts to make sense to see the signal processing part of a Radio Telescope as part of the operating expenditure, rather than as a massive initial capital outlay. This is described in a [3] by Parsons, et al. CASPER Hardware In order to take full advantage of Moore s Law, the CASPER implementation relies on a small number of hardware platforms, which are updated frequently. In this way the development can be shared among the community, which makes it easier to justify re-design of boards when new processing capacity becomes available. The current generation of hardware is called ROACH (Reconfigurable Open Architecture Computing Hardware). ROACH is based on Xilinx Virtex 5 technology and has seen widespread adoption world-wide. To supplement ROACH, there are a number of peripheral plug-in boards available, like Analogue-to-digital and Digital-to-analogue conversion boards. These boards make use of a CASPER standardised interface. At the time of writing ROACH2 was approaching production readiness. ROACH2 uses a Xilinx Virtex 6 device, but is otherwise based on the same architecture and much of the same component base as ROACH. Plans are in place for ROACH3, which is likely to be the baseline processing board for the MeerKAT Correlator/Beam Former Sub-System. CASPER Software Tools On top of the hardware platforms, there are several layers of software needed to implement various system functions. CASPER provides a signal processing development environment that runs in MATLAB/Simulink. This consists of libraries of blocks that perform typical signal processing functions used in radio astronomy. Examples include efficient, high bandwidth FFT implementations, blocks used to implement Polyphase Filterbanks and Correlator building blocks. To augment the libraries, pre-packaged example designs are available. Both blocks and designs are regularly updated by a large international community. There are also protocols defined for control (KATCP), [5] as well as data interfacing (SPEAD), [4]. For all of these, reference implementations are provided by the community.

4 The CASPER eco-system is illustrated in Figure 1. In all the blocks there are community contributions, from the design and production readiness of ROACH boards, through the implementation and test of the purple signal processing components, test and evaluation of 10GbE hardware for suitability and software reference implementations to run on commodity processing hardware. Figure 1 The CASPER eco-system. MeerKAT The MeerKAT array will consist of 64 receptors with 13.5 m diameter dishes, designed to achieve high sensitivity and imaging dynamic range, while providing an array and functionality to provide for a wide range of science. A selection of key parameters for MeerKAT is given in Error! Reference source not found., while the schedule for implementation is given in Table 2.

5 Table 1: Key MeerKAT Parameters Antennas 64 Offset Gregorian Dish Diameter 13.5m Continuum imaging dynamic range at 1.4 GHz 60 db Line-to-line dynamic range at 1.4 GHz 40 db Mosaicing imaging dynamic range at 14 GHz 27 db Linear polarisation cross coupling across -3 db beam -30dB Sensitivity ( GHz) > 220 m^2/k Sensitivity (8 14.5GHz) > 200 m^2/k Table 2: MeerKAT Phasing Phase Precursor (KAT- Phase 1 7) Phase 2 Number of dishes Receiver bands (GHz) Max processed bandwidth (GHz) (goal of 4.0) Max baseline (km) Min baseline (m) As can be seen from the aggressive schedule, this is a departure from traditional radio astronomy implementations and a different approach is required to achieve two main goals: 1. Build a team with the right experience and capabilities. 2. Design, Implement and Verify the performance of the telescope. The DBE team adopted CASPER as a means achieve these goals. We started with a team with strong electronic system design experience, but almost no radio astronomy experience. CASPER helped us to build up experience by joining a large international collaboration with many experts. In interaction with the collaboration we have been able to learn from their experience in Implementation and Verification of radio astronomy instruments. The MeerKAT Digital Back-End From the early key specifications, and with the right experience, it is easy to design a preliminary architecture that would satisfy all the known requirements. An example of such architecture is shown in Figure 2. This architecture is used as the basis for design of lower level components, particularly the ROACH series of Reconfigurable Processing platforms. In this architecture, an FX correlator is used and processing is partitioned between F and X Engines. The F-Engines perform the frequency channelization, while the X-Engines perform the cross multiplication. More on correlator architecture and the associated computational cost is available in [7].

6 Figure 2 Generic CASPER Architecture Once the building blocks are in place, they become a constraint in design, but the architecture can still be freely adapted. An example of such an adaptation is shown in Figure 3. This architecture demonstrates two concepts firstly that the architecture can be easily adjusted to optimised for a different variables (in this case number of switch ports); and secondly that generations of hardware can be mixed (the ibob boards are an older generation). The numbers of switch ports are reduced by sending data from the F-engines directly to the X-engines before entering the switch. Data is looped back through the switch if it is intended for the transmitting X- Engine, keeping the design modular.

7 Figure 3 CASPER F-Engine loopback concept The status at the time of writing for the MeerKAT DBE is that the allocated sub-system requirements have been received from the system level. Requirements analysis is in progress and the planning for the acquisition is near complete. As described above, a preliminary architecture has already been designed and will represent a concept to be evaluated against the maturing requirements. This architecture is shown in Figure 4.

8 Figure 4 MeerKAT Preliminary DBE Architecture This architecture reflects the need for a flexible instrument, with high availability. It sees the introduction of a new architecture component the D-Engine, which performs the function of digitising the signals at the antennae, applying a time stamp and digitally transmitting the signals to a central signal processor, where beam forming and correlation is done. The central signal processor is built around a large Ethernet switch, which allows the system to be highly configurable. This also allows the introduction of hot-swap spare processing capacity that can be configured to replace the functions of any hardware that malfunctions. Heterogeneous processing nodes can be used. For logistic support, the initial production is done with a number of spares (to be determined by reliability analysis of the components). These spares will support the system during the initial phases, but after a period, components that malfunction will be replaced by next generation compatible hardware. This will allow an upgrade in function and a reduction of power consumption. Linking CASPER and MeerKAT The challenge of using the CASPER approach with a formal project like MeerKAT, is that the attractions of flexibility and community contributions of an open source collaboration like CASPER are risky in the more structured and disciplined view required in a large project. The question of how to link these two together must be addressed from multiple points of view. Firstly, the MeerKAT engineering team had to be assembled in the correct way to ensure that the members have the correct combination of skills. For MeerKAT, we did not have the luxury of a large pool of engineers with radio astronomy experience, but we could select a team with a varied background and expose them to international experts by way of seconding them to institutions abroad. Students were sent to UC Berkeley to gain experience and all staff were encouraged to attend CASPER events in order to learn.

9 Secondly, the team must be mandated to get involved with CASPER and indeed to become a leading contributor to CASPER in order to ensure that the direction of CASPER is always compatible with the MeerKAT requirements. This takes resources, but it is important to take the return on investment into account. The contribution to CASPER is a trade, where we invest electronic engineering skills and receive radio astronomy experience and skills in return. Of course, this investment vests in the team members, and would be lost if excessive staff turnover occurred. Another aspect is process. The design cycle of MeerKAT is not aligned with the release cycle of either FPGA vendors or CASPER. We elaborate on each of these in the paragraphs below. Team Composition Parallel teams continuously operate in the domains of hardware development, software and tool flow development and application development. Although team members may perform multiple roles and switch between domains, the development cycles are essentially decoupled. Strong periods of interaction occur around key design reviews. These reviews are typically the very early reviews a block diagram for a ROACH board is an example. During such a review, the hardware team defend a design based on best hardware practice, while the signal processing experts try to extract maximum possible performance. Analysis based on previous generation hardware is used to project future performance. A compromise is reached where the board complexity is low enough not to impact manufacturing yield and the performance is high enough not to necessitate unrealistic numbers of boards to be required in a system. Occasionally, the best option is to wait next generation processing hardware can enable what is impractical today within a short time. Moore s law is still on track today. The hardware team responsibility is to supply a fully tested and industrialised processing platform. This is done by establishing a production baseline with one or more preferred assembly partners. In the MeerKAT project there are typically two dedicated hardware design engineers one focussed on large digital designs, the other on mixed signal design. They start with a next generation design as soon as the current generation is production ready. In practice, miscellaneous work interrupts the flow there are always smaller projects to complete between the bigger steps. The hardware team are assisted by a dedicated test engineer, who is responsible for qualification and production testing. The team also has a dedicated gateware team. Gateware is a term coined at UC Berkeley to describe the aspect of the system design that is based in FPGA (or ASIC s), but at a lower level of abstraction than the signal processing design. This is the area where hardware description languages are used. This sub-team consists of two members. The radio frequency interference environment requirements are extreme emissions must be orders of magnitude lower than industry standards. This leads to the need to design custom enclosures that are very well shielded. In order to design these enclosures, we employ a mechanical engineer. An indication of the continuous nature of software development is that there are three dedicated software engineers. They share their time between application software and hardware support software. The application team consists of three members. Two system engineers take responsibility for two of the sub-systems that the DBE take responsibility for, while the third sub-system duty is shared among them. An analyst is employed to perform detailed mathematical analysis.

10 Somewhat outside the System Engineering process, but strongly coupled to it, is a Project Engineer, primarily concerned with project scheduling and control, and a technician to assist with maintaining a high technology laboratory, as well as installations and field testing. The team composition is summarised in Figure 5, which also shows Figure 5 DBE Team Composition Process Two major risks faced when committing to hardware early, are missing requirements or challenging requirements. If missing requirements are uncovered late, the design is fixed and a re-design could be needed. By focussing early attention on the areas where there may be missing requirements that impact the architecture, the risk is reduced to the level where building block implementation can commence prior to finalisation of the higher level specifications. This is the method used to design the basic building blocks for the MeerKAT DBE, otherwise known as ROACH boards. The conceptual design for a new generation of ROACH board is a process involving experts from many backgrounds particularly signal processing. The Digital Back-End requirements are derived from system level requirements and architecture design as is normal in the System Engineering discipline. However, this process takes time to complete and reach a final conclusion. Typically there are a small number of requirements that are challenging from one or more perspectives, for instance: Requirements that are difficult to validate Requirements that are difficult to verify Requirements that are outside the current state of the art, possibly in need of special test facilities Many of these challenging requirements are often not architecture driving requirements and merely require attention to implementation detail. To the extent that this is true, they allow de-coupling of the development of the processing building blocks from the development of the architecture with little or no penalty in the economy of the design. This is the ideal case for collaborative development and allows most freedom. For the cases where challenging requirements are of the architecture driving kind, early effort must be put into the understanding of these, so that delays can be inserted in the low level developments if risk mandates it. An example of this is the delay in development of ROACH3 while the early requirements analysis of MeerKAT is completed. Another option in such a case would have been to design for ROACH2 as a risk mitigating strategy. Both these strategies remain available; and moreover, both are easily quantified to reasonable accuracy in terms of cost and schedule.

11 The existence of a generation of ROACH is seen as an opportunity that can be traded off against any other naturally occurring opportunities. This usually leads to a design based on ROACH, because the shared development cost makes it the most viable solution. On rare occasions it may be taken as a constraint this is true where legacy hardware is re-purposed for a new design. Both views are now easily integrated with a normal system engineering flow. Opportunities More often, the availability of ROACH is the catalyst to implement a system to address an emerging technical challenge. This was the case in the design of a specialised Radio Frequency Interference (RFI) analysis tool known as RATTY (ReAl Time Transient analyser). This instrument was developed in rapid time to assist in the measurement of transient RFI, as produced for instance by electric fences. A number of papers have been published based on the work done with this instrument. The CASPER eco-system enabled this an opportunity that would otherwise have been lost. Challenges This design flow is not without challenges. While the signal processing and hardware bottlenecks are largely eliminated from the system view, software still presents a problem. This is somewhat mitigated by having compatible platforms in successive generations, but good software engineering principles are, as always, still required. Probably the biggest challenge to an approach like this is in the statutory obstacles to participating in Open Source projects in general. While CASPER was an existing collaboration and our participation in it could only happen in compliance with existing licensing conditions, it is not clear that the Intellectual Property Rights from Publicly Financed Research and Development Act of 2008 would allow entering into such collaborations in a timely manner. Appreciation for non-financial return on investment is not clearly expressed in the act, and care should be taken to follow the prescribed procedures of the act in the case of future collaboration projects. Conclusion The experience of the MeerKAT DBE team has shown that there is tremendous benefit to be had from Open Source collaboration, even when it requires close coupling to formal engineering process not usually associated with such flexible arrangements. The return on the investment in Open Source is often in the form of access to expertise, something that can usually not be easily quantified in financial terms. References Retrieved on Moore, G. E. Cramming more components onto integrated circuits, Electronics, Volume 38, Number 8, April 19, Parsons, et al SPEAD 6. KATCP 7. Wethimer, et al. The Collaboration for Astronomy Signal Processing and Electronics Research. Poster, URSI General Assembly 2011.

12 8. D Addario, L. ATA Memo 24, Correlators: General Design Considerations March Intellectual Property Rights from Publicly Financed Research and Development Act of Biography Francois Kapp is the Sub-System Manager for the Digital Back End of the MeerKAT radio telescope. The Digital Back End includes the Digitiser, Correlator/Beam Former and Time and Frequency Reference Sub-Systems. He joined the SKA South Africa team in February 2006 and became Sub-System Manager in Prior to joining SKA South Africa, he spent 5 years with a start-up developing capacitive touch sensing technology and 5 years designing Electronic Warfare Systems. Copyright 2012 by SKA South Africa, Francois Kapp and Jason Manley. Published and used by INCOSE SA with permission.