FIBER OPTIC NETWORK TECHNOLOGY FOR DISTRIBUTED LONG BASELINE RADIO TELESCOPES

Experimental Astronomy (2004) 17: 213 220 C Springer 2005 FIBER OPTIC NETWORK TECHNOLOGY FOR DISTRIBUTED LONG BASELINE RADIO TELESCOPES D.H.P. MAAT and G.W. KANT ASTRON, P.O. Box 2, 7990 AA Dwingeloo, The Netherlands (e-mail: maat@astron.nl) (Received 2 August 2004; accepted 28 February 2005) Abstract. The construction costs of distributed radio telescopes are to a great extent determined by the deployment costs of the fiber optic data transport network that is needed to transport the received information to the data processor(s). As such, the baseline and data rates that are feasible for a specified amount of money are determined by the status of the technology and deployment costs of the communication network. In this paper the present day data transport status is described and, using a costing model, the most attractive data transport technologies are determined, taking the LOFAR telescope (ASTRON, 2005) as an example. In the outlook, the near-term data transport technology developments are described. Keywords: CWDM, data transport network, distributed radio telescope, Ethernet, GbE, network design, network topology, radio telescope, WDM, wide area network, 10 GbE 1. Introduction To meet the resolution and sensitivity demands that are required in present day radio astronomy, distributed radio telescopes are needed with long baselines and large numbers of stations with many antennas, each producing a large amount of data. For obtaining the required high quality imaging, various antenna distributions can be used, ranging from a random antenna distribution to non-uniform highly structured antenna arrangements. Which of these antenna distributions is chosen depends on construction related issues like, e.g., costs and geographical constraints. In addition, many options for the way the data processing is performed are available: both a completely centralized data processing and a fully distributed processing of the data (i.e., all data processing is performed at the stations) are among the possibilities. The actual situation will be in between both options. Both the antenna arrangement and the way the data processing is handled have a strong influence on the data transport system in the radio telescope. For example, in the case where centralized data processing is applied in combination with a random antenna distribution, the amount of data that needs to be transported is much larger than when an antenna distribution having an increasing antenna concentration towards the central processor is used. If in this latter case part of the data that is produced at the antenna stations is also processed at these stations, even less data

214 D.H.P. MAAT AND G.W. KANT needs to be transported within the telescope. The configuration of the telescope also affects the way the data is handled. When centralized data processing is applied, the data traffic has a unidirectional, point-to-point characteristic, while in the case of a distributed correlator telescope, data routing is required. Which type of data handling is preferred for a distributed radio telescope strongly depends on the state of the available communication technologies during the time the telescope is designed and constructed. 2. Present day network technology for distributed radio telescopes For the transportation of radio telescope data several types of networking technology are available, of which SONET/SDH (Bellcore, 1999) and Ethernet (IEEE SA, 2002) are the most important. The SONET/SDH and Ethernet network technology types differ strongly. The technology for SONET/SDH is specially designed for telecommunication networks, in which the various nodes should be capable of exchanging information synchronously. A large part of the technology in SONET/SDH networks is used for obtaining a very high availability level. This percentage of time a SONET/SDH network is available for data transport is in the order of 99.9999%. The SONET/SDH network topology consists of interconnected rings with bidirectional traffic that form a meshed network. This ring topology in combination with fast re-routing technology strongly reduces the risk that, e.g., a fiber-cut blocks the information exchange between two or more nodes. To establish connections between the different rings, and for connecting the subscribers to these rings, routing equipments like cross connects and add-drop nodes are used. The synchronization, high availability and routing features in SONET/SDH networks require relatively complex technology and fiber topologies, resulting in relatively high deployment costs. The SONET/SDH technology is well suited for transportation of large amounts of information (up to Tb/s) over long distances (>1000 km). The communication equipment in SONET/SDH is currently assembled with the use of building blocks having data rates up to 10 Gb s 1, which will be extended towards 40 Gb s 1 in the coming years. In the past years the use of Ethernet in wide area networks (WAN s) has increased strongly. Unlike SONET/SDH, optical Ethernet is specially designed for data communication in which a synchronous data transport is not required. Several network complexity levels are available for the current Ethernet communication networks. The simplest (standard) version consists solely of point-to-point connections for which relatively simple technology is sufficient. In more complex optical Ethernet systems with a large number of subscribers the information exchange between the various nodes is performed with the use of routers. In standard Ethernet only very basic availability functionality is present. An increased availability level is obtained by employing Resilient Packet Ring (RPR) technology (RPR Working Group, 2005) in combination with an optical Ethernet

FIBER OPTIC NETWORK TECHNOLOGY 215 ring network. In almost all cases WAN Ethernet technology is less complex and less costly than SONET/SDH equipment. Which type of technology is to be applied in a distributed radio telescope depends on the requirements for the telescope and the way it is constructed. In the LOFAR telescope, which is used as an example in this paper, optical Ethernet is employed. Ethernet is chosen because of its low cost, easy to handle, properties. The time synchronization of the data is obtained by time-tagging the data packets that are sent from the antenna stations to the processor. The routing equipment that is available for optical Ethernet is not needed in the LOFAR-WAN since only point-to-point connections from the antenna locations towards the data processor are employed. The availability of point-to-point connections in a distributed telescope WAN can be improved by creating fiber rings in the network topology in combination with RPR technology. Since the construction of these rings requires the deployment of additional fiber tracks, the costs of improving the availability level of the telescope WAN are high, especially for long haul point-to-point connections. Since the very high availability levels of telecommunication networks are not required for distributed radio telescopes, it is likely that the use of relatively simple availabilityimproving technologies is sufficient for a telescope WAN. 3. Optical Ethernet technology The current mainstream optical Ethernet core technology is 1 Gb s 1 (OC-24) Ethernet (1 GbE), which employs a transmitter/receiver combination at both sides of a fiber pair. The transmission distance of the 1 GbE link depends on the type of laser/transmitter and the type of fiber that are used. For COTS Ethernet technology several transmitter/receiver component multi-source agreements (MSA s) exist for which a number of transmission distances are commercially available. Of all Ethernet tranceiver MSA s the SFP(1 GbE) (van Doorn, 2005) and XFP(10 GbE) (XFP MSA Group, 2004) MSA s are the latest and most advanced. In this paper SFP/XFP based technology is addressed. For short-range distances (<300 m) low cost source lasers (e.g., VCSEL s) and multimode fibers are used. The longest distance 1 GbE links ( 100 km) employ high power DFB source lasers and sensitive receivers in combination with single mode fiber. Where data rates larger than 1 Gb s 1 are required along a link, a number of options are available. The most straightforward way is to combine a number of parallel fibers along the track, which is denoted as space division multiplexing (SDM). A more sophisticated way of increasing the data rate is to raise the number of wavelength channels in the fiber. By assigning a 1 GbE data stream to each wavelength (wavelength division multiplexing or WDM), a single fiber can be used for the transportation of a multiple amount of 1 GbE channels. Another single fiber technique is time division multiplexing (TDM), which uses the aggregation of a number of low data rate streams into a single high data rate stream. For present day

216 D.H.P. MAAT AND G.W. KANT Ethernet this means that a number of (electrical) 1 GbE streams are combined into a single (optical) 10 Gb s 1 (OC-192) Ethernet stream (10 GbE) with the use of an aggregation switch. 3.1. PRESENT DAY WDM ETHERNET For 1 GbE small WDM systems are currently available that use a relatively large wavelength channel separation: coarse WDM. In these CWDM systems a maximum of eight wavelength channels can be employed with a maximum transmission distance of about eighty kilometers. The price of an eight-channel 80 km CWDM module is about $28000 (US dollars are used throughout this paper). 3.2. PRESENT DAY 10 GbE TECHNOLOGY 10 Gb s 1 Ethernet equipment is currently available with a (relatively short) maximum transmission distance of 10 km. This equipment is relatively costly: $5000 for an optical module in combination with $9000 for the accompanying aggregation switch. For extending the 1 GbE and 10 GbE transmission distances, existing optical amplification equipment like EDFA s (erbium doped fiber amplifier) or SOA s (semiconductor optical amplifier) can be used. However, since standardization is not available for optical amplification in Ethernet links, system research is necessary when this type of technology is to be applied. The only way to increase the transmission distance within the current Ethernet standards is to combine a commercially available Ethernet receiver and transmitter and to use it as a regenerator. This regeneration solution is, especially in WDM and 10 GbE systems, a relatively costly way of extending the transmission distance. 4. Multiplexing technologies in Ethernet links: A cost comparison By applying multiplexing technologies like TDM and (C)WDM big bundles of fibers can be replaced by just a few fiber pairs, in which way a reduction of the deployment costs can be obtained. Since TDM and WDM equipment is costly, the use of multiplexing technologies will not always lead to a cost reduction. To determine which of the three technology options (SDM, WDM or TDM) will provide the lowest costs for a specific communication link, a model has been developed which is used to predict the deployment costs of a communication link or network. Apart from the costs of the optical communication equipment, the fiber cable and the trenching (existing runs are not used), this model also takes into account the costs of all other in-door equipment like patch cords and distribution frames. Also, the cost consequences of the required engineering, fiber testing, licensing and splicing are inserted in the model. The equipment pricing that is used in this

FIBER OPTIC NETWORK TECHNOLOGY 217 Figure 1. Deployment costs comparison; (a) SDM vs. (C)WDM; (b): SDM vs. TDM (10 GbE), both as functions of link data rate and distance. tool is determined with the use of generally available information (internet) or via direct requests for list prices. All labor-involved costs only apply to the ASTRON location in the Netherlands and may be different at other places around the world. The pricing that is used in the model was updated up to the summer of 2004. Since the costs of Ethernet equipment changes rapidly, the situation after 2004 may differ from what is presented in this paper. In Figure 1 the calculated relative deployment costs of a point-to-point link are depicted for the SDM, (C)WDM or TDM (10 GbE) technologies as functions of the data rate and the transmission distance. Figure 1a shows that for transmission distances up to about 10 km the application of SDM is less costly, while for larger distances and data rates exceeding 10 Gb s 1 the most attractive solution is provided by (C)WDM technology. A comparable situation can be seen on the right side in Figure 1b which shows that TDM (10 GbE) becomes economically attractive for distances larger than 3 km. 5. Costs of the data transport network in distributed radio telescopes The costs related to an increase of the antenna station data rate or an increase of the telescope extent can also be determined with the use of the earlier mentioned cost model. In these calculations a distributed radio telescope topology is required. In the calculations presented in this paper a generalized topology of the LOFAR telescope is used (Bregman et al., 2002), in which the following WAN features can be distinguished: Antenna stations are distributed along a number of curved arms along which the stations are positioned, using an exponentially increasing distance from the center (0.4 km, 1.0 km, 1.8 km, 3 km,...). All data is processed at the center of the telescope, resulting in a unidirectional data transport along the arms towards the center.

218 D.H.P. MAAT AND G.W. KANT Figure 2. The calculated relative deployment costs of the data transport network along a LOFAR arm as a function of the length of the arm for both 1 GbE SDM and 1 GbE CWDM. In this calculation a station data rate of 10 Gb s 1 is used. For both SDM and CWDM the communication equipment costs and the costs for the deployment of the fiber are given. The data will be transported with point-to-point connections: ring and/or mesh topologies are not used such that costly routing is avoided. In the calculations, the antenna stations are used as link nodes at which signal regeneration or multiplexing can take place. CWDM or TDM techniques are only applied in between the various nodes in case a cost reduction with respect to SDM is obtained. In Figure 2 the calculation results for SDM and CWDM are given. Results for TDM (10 GbE) are omitted since these results differ only slightly from the SDM results. Figure 2 shows that for a station data rate of 10 GbE most of the WAN costs are related to the deployment of the fibers. For arm lengths exceeding 50 km the fiber related costs depend approximately linearly on the length of the arm. The increased costs per kilometer at arm lengths smaller than 10 km is caused by the strongly increasing amount of stations per kilometer towards the data processor, which results in increased costs, e.g., fiber management equipment and splicing. The equipment costs per kilometer arm length get smaller for increasing arm lengths. This cost reduction per kilometer is caused by the more efficient use of communication equipment in the longer tracks, which holds for both the transmitter/receiver and for the WDM equipment. This latter observation is supported by the calculation results depicted in Figure 2: at an arm length of 17 km the CWDM/SDM (total) cost ratio is 0.95 while at 200 km this ratio is reduced to 0.85. For longer arm lengths this deployment cost ratio remains approximately the same. In the evaluated WAN topology the use of CWDM instead of SDM is only favorable when long arms lengths are used (>10 km). Assuming a linear cost vs. arm length relation, a length doubling from, e.g., 200 km to 400 km results for SDM

FIBER OPTIC NETWORK TECHNOLOGY 219 Figure 3. The calculated relative WAN deployment costs as a function of the station data rate using an arm length of 200 km. The total costs, the communication equipment costs and the costs for the deployment of the fiber are given for SDM, CWDM and TDM (10 GbE). in (almost) a cost doubling (1.9), while for CWDM this length increase results in a cost increase by a factor of 1.6. The amount of data that needs to be transported from each station to the data processor strongly influences the WAN deployment costs. Since relative costly trench digging is not involved when data rates are considered, the cost impact of a station data rate increase is not as big as the impact of an arm lengthening. In Figure 3 the calculation results of the WAN deployment costs vs. station data rate are depicted. It shows that the deployment costs grow linear with a station data rate increase. The relative short transmission distance (<10 km) of the current 10 GbE equipment makes 10 GbE unsuited for long range links and necessitates closely spaced regenerators. As a result the use of 10 GbE does not provide a cost reduction compared to the use of SDM. In case CWDM equipment is deployed substantial cost savings are obtained. The larger the data rate the more profitable the use of CWDM becomes: for a 10 Gb s 1 station data rate the CWDM/SDM (total) cost ratio is 0.85 while for a station data rate of 100 Gb s 1 this ratio has reduced to 0.69. 6. Outlook In the coming years further optical Ethernet technology development will take place. For 10 GbE communication it is likely that long distance ( 80 km) optical communication equipment will become available. In addition, new Ethernet related WDM technology will become available. The goal in this development will be to introduce Optical Ethernet WDM systems with more wavelength channels ( 100)

220 D.H.P. MAAT AND G.W. KANT than in the current CWDM ( 8) systems. The combination of the new WDM and 10 GbE technology leads to a strong improvement of the current Ethernet multiplexing techniques, resulting in further deployment cost reductions for high bit rate communication networks. 7. Conclusions In this paper the deployment costs of a distributed radio telescope WAN have been investigated. In this investigation, a generalized LOFAR-WAN topology was used as an example data transport network. Optical Ethernet was determined to be the most suitable data transport technology for this network. Point-to-point Ethernet link deployment cost calculations show that the application of multiplexing techniques in high bit rate (>10 Gb s 1 ), long distance (>3 km) communication links will lead to a reduction of the deployment costs for this link. Most of the network related costs for expanding the baseline of a distributed radio telescope concern the deployment of fibers. Deployment cost calculations for the generalized LOFAR-WAN topology show that the WAN expansion costs can be reduced by about 15% by applying CWDM equipment. The costs related to an increase of the station data rate can also be reduced by using CWDM technology: the larger the data rate the bigger the cost reduction, ranging from 15% for a station data rate of 10 Gb s 1 to 31% for 100 Gb s 1 per station. The application of currently available 10 GbE technology does not provide big cost savings due to its relative short transmission distance. In the coming years improvements are expected in this area. References ASTRON: 2005, http://www.lofar.org Bellcore: 1999, GR-253-CORE SONET Transport Systems: Common Generic Criteria. Bregman, J. D., Kant, G. W. and Ou, H.: 2002, Multi-terabit routing in the LOFAR signal and data transport networks, Proceedings XXIV URSI GA. IEEE SA: 2002, http://standards.ieee.org/getieee802/802.3.html RPR Working Group: 2005, http://www.ieee802.org/17/ van Doorn, S.: 2005, http://www.schelto.com/sfp/ XFP MSA Group: 2004, http://www.xfpmsa.org/