Experiences and measurements in operational PTP synchronized mobile networks

Transcription

1 Experiences and measurements in operational PTP synchronized mobile networks Antti Pietiläinen Contributors: Georg Hein, Lasse Oka, Petter Isaksen, Joachim Eckstein, Albert Anreiter, Hannu Kallio, and Miguel Lopez Garcia 1 Nokia Siemens Networks 2012

2 Short history of PTP, IEEE 1588 Precision time protocol frequency synchronization in telecom IEEE 1588 committee started to develop version 2 in Telecom companies joined in for having more freedom - than was available in IETF NTP group - to create an optimal synchronization protocol for telecom purposes. As the last technical input in early 2007, informative annex A.9 was added. It suggests which features of the standard should be used when unicast messaging, without on-path support, is used. Almost all PTP slaves in the field at the moment are of this type. IEEE 1588 v2 was published in ITU Q13/SG15 started to develop PTP telecom profile for frequency synchronization without on-path support in The profile, G was finished in the summer The main achievement was to fix more parameters than in Annex A.9. On the other hand, G was made noninteroperable with Annex A.9. Since then, Q13 has defined packet timing metrics for frequency. G.8260 and G were consented in December 2011 but 2 Nokia Siemens Networks 2012

3 PTP synchronized network examples: European network A Old ATM based UMTS base stations supplemented with IP based base stations PTP synchronization deployed since MHz/ (1) 2 Mbit/s GPS (2) IP/ MW Grandmaster 2 MHz/2 Mbit/s sync IP 3G bts Packet Router E1 ATM STM-1 STM-1 STM-1 Legacy ATM 3G bts ATM/PDH MW Switch RNC Mixed ATM/ IP 3 Nokia Siemens Networks 2012

4 Error [ppb] European network A Longest microwave radio chain 6 hops. Total number of loaded links: 16. The number of PTP synchronized base stations in this network is currently about Grandmaster GE GE n x STM1 GE GE GE Mbit/s Switch SDH SDH 30 Mbit/s Router RNC Error estimate of the worst of 18 measured base stations: 4 Nokia Siemens Networks 2012 GE GE FE FE

5 North American network Longest microwave radio chain recorded by the authors that passes PTP timing, 10 radio hops. Operational since summer Number of PTP base stations: 450 Really has been working great the whole solution!! Radio network controllers GE GE 400 M 400 M 400 M Grandmaster 100 M 400 M 400 M 400 M Fiber or copper Microwave 100 M 100 M 50 M 5 Nokia Siemens Networks 2012

6 European network B Basic architecture: Optical network with up to three radio hops to tail sites. RF sharing between GSM, UMTS, and LTE requires tight synchronization between all three radio signals. Therefore, one of the technologies need to be the synchronization master in a site. LTE units were chosen to act as the PTP slaves. BSC GSM 2G 3G LTE PTP slave Base station controller Global system for mobile communications LTE RNC SGW UMTS About 1000 IEEE-1588 synchronized LTE base stations in the start. Non-co-sited UMTS and GSM base stations are being migrated separately into PTP. Current total number of active PTP slaves ~ (rough guess). A challenge is that the network was originally built for much lower bit rates but now it is being upgraded as the LTE traffic ramps up. Optical network Long term evolution Radio network controller SAE (system architecture evolution) gateway Universal mobile telecommunications system PTP Grandmaster 2G GSM 3G UMTS LTE BSC RNC SGW 6 Nokia Siemens Networks 2012

7 European network B Problems can be elementary: Synchronization planner had first designated synchronization packets into best effort service class. Additionally, there was no plan how to allocate different base stations to different masters. After a review, higher priority class was assigned for PTP and base stations were designated to different masters area-wise. In this way it was also easier to control that the load of different masters became evenly distributed and hop counts remained shorter. 7 Nokia Siemens Networks 2012

8 Key elements in building successful PTP deployments The slave design was based on somewhat pessimistic expectation of packet delay variation and the algorithm was designed to have long integration period. The initial fast aging of oscillators and large temperature variations were taken into account. Trust the crystal, but not too much! Delay jumps were taken into account from day one of the design. Apart from the above, the algorithm was kept simple to avoid problems that a too clever algorithm might produce when it faces the unexpected. Metrics were developed and tolerance masks were specified so that it was possible to qualify packet networks for packet timing and also monitor, if required, how the safety margins change while the traffic increases. 8 Nokia Siemens Networks 2012

9 Network limits for PDV and G ITU has defined Hypothetical reference model 1 (HRM-1) in G consisting of 10 one-gigabit and higher fiber optic links. HRM-2 has been also defined, see below. DSLAM DSL modem HRM-2a Packet master clock 10 GE GE Packet network N = 5 OLT ONU HRM-2b Packet slave clock Floor delay packet population Packet node FDPP (e.g. ernet was selected switch, IP router, as the MPLS metric router) for HRM-1. Especially the network limit but also 10 Gbit/s the metric fiber optical itself link have been considered conservative. HRM-2 metric and limits are for 1 Gbit/s further fiber study optical in link G HRM-2 cases exhibit more noisy delay floors than HRM-1. Because FDPP does not Microwave link make a difference between easy-to-filter noise and the more challenging characteristics of packet delay variation (PDV), NSN utilizes MAFE metric to establish HRM-2 PDV network limits. 9 Nokia Siemens Networks 2012 HRM-2c

10 MAFE MAFE PDV metric for defining PDV network limits and network design rules MAFE and MATIE (maximum average frequency error & time interval error) were developed in 2008 as PDV metrics. It appeared that MATIE formula is exactly the same as ZTIE proposed by G. Zampetti almost 20 years earlier. MAFE and MATIE are included in the new version of G MAFE is a convenient metric for base stations since the target is frequency stability. 16-ppb output requirement. Delay jump, temperature, and aging margin: 4 ppb. 100 ppm 1E-4 10 ppm 1E-5 1 ppm 1E-6 1E-7 10 ppb 1E-8 1 ppb 1E-9 1-% fastest packet preselection before calculating MAFE tau [s] 10 Nokia Siemens Networks ppm 550 µs 250 ppb 250 µs 12 ppb 120 µs 12 ppb 1.2 ms [s] Timing packet stream should have at least the same priority as the real-time traffic and receive expedited forwarding QoS. Maximum number of hops: 20. Maximum number microwave hops: 10. In this case the total number of hops should be less than 15. The average load of the links along the path should not be persistently above ~50 % if the path is long.

11 Delay [ms] Delay [ms] Delay measurements and analysis GPON last mile, Asia Pacific A small percentage of packets have delays of 10 s of ms, even up to 210 ms but the average delay is 4.05 ms and the median delay is 1.15 ms. Lost PTP packets: 3 %! Nokia Siemens Networks 2012 Time [s] 1.12 Delay min/ave/max: 1.1/4.05/208.7 ms Minimum delay 1-%, 200-s average Minimum delay - 1-%, 200-s average magnified E.g. 16 pps average of 32 fastest packets in 200-s 1.11 window Clocks use more or less this data Time [s]

12 MAFE [relative] Delay measurements and analysis Asia Pacific, GPON last mile GPON is an HRM-2 case. The particular path has a 27- fold margin to NSN HRM-2 limit but also a 6-fold margin to G HRM-1 limit Floor delay packet population (FDPP) 1 %, 200 s 23.5 µs, limit 150 µs 1/6 of HRM-1 limit Fastest packet 1E-5 1E-6 1E-7 1E-8 1E-9 1E-10 1E-11 MAFE 1 %, 200 s NSN HRM-2 mask [s] Delay jump removed 1/27 of NSN HRM-2 limit 12 Nokia Siemens Networks 2012 Time [s]

13 Delay [ms] Microwave radio Africa Radio bandwidths decrease from 150 Mbit/s to 30 Mbit/s toward the edges. The measured path had 6 MWR hops. At the time of measurement only circuit emulation traffic to co-siting 2G base stations. Delay min/ave/max: 1.84/1.88/3.19 ms Minimum delay - 1 %, 200-s average The circuit emulation traffic interfered first with the timing packets. The interference was eliminated by switching on dither in the masters. Dither creates intentional jitter to the timing packets while maintaining the accuracy of the timestamps Nokia Siemens Networks 2012 Time [s]

14 MAFE [relative] Microwave radio Africa Microwave radio is an HRM-2 case. The path had a 24-fold margin to NSN HRM-2 limit but also an 8.4-fold margin to G HRM-1 limit Floor delay packet population (FDPP) 1 %, 200 s 1E-5 1E-6 1E-7 1E-8 1E-9 1E-10 1E-11 MAFE 1 %, 200 s NSN HRM-2 mask 1/24 of NSN HRM-2 limit [s] Fastest packet 17.8 µs, limit 150 µs 1/8 of HRM-1 limit 14 Nokia Siemens Networks 2012

15 Remarks on experiences gained from PTP synchronized base stations Measurements for evaluating network performance, if done, are carried out before traffic is ramped up. Ample margin observed. When the network is up and running, communications regarding synchronization performance stops and no complaints are received. This hints that the NSN HRM-2 limit is adequate. Remarks on attempts to satisfy G.823 synchronization mask everywhere Frequency synchronized base stations require 16-ppb stability, which allows 16- µs drift every 20 minutes. Many vendors have committed to SEC-mask, 5.3-µs drift per day, typically leading to excessive reliance in the local oscillator. After the algorithm loses track, erratic operation follows. A third party product, tested with a delay pattern that should result into about 3-ppb maximum error, remained quite stable for a long while but suddenly produced 28-ppb frequency error for 6000 seconds (170 µs drift). 15 Nokia Siemens Networks 2012

16 Remarks on PDV metrics G HRM-1 limit based on floor delay packet population (FDPP) and NSN HRM-2 mask based on maximum average frequency error (MAFE) require approximately the same time constant, 8000 s, for the slave clock. However, for example in the GPON case, the NSN HRM-2 mask would allow 4 ½ times more noise than FDPP HRM-1 limit, which is due to the conservative nature of FDPP metric, see explanation below The ceiling of 1-% Floor delay packet population. Resembles closely the data used by the clock algorithm. Delay Fastest packet Worst-case delay pattern with same amplitude as the signal used by the clock Filtering ability: FDPP does not make a difference between measured and worst-case waveform Scattered floor: Packet slaves are not affected by this PDV but, nevertheless, consumes FDPP margin in full. 16 Nokia Siemens Networks 2012

17 PTP Time synchronization G , IEEE 1588 Telecom profile for time synchronization Multicast ernet (as in SyncE) has been chosen as the default mode. Allowing an optional IP mode will be decided upon in a later meeting. IP unicast and multicast have been proposed. ITU model, 20 hops. ITU Q13 has decided that equipment must have capability of hybrid operation with SyncE. However, PTP time transport should not fail if SyncE portion fails. All ports use for PTP the MAC address: C E S BC 8 BC 7 BC 6 BC 5 BC 4 BC 3 BC 2 BC 1 M Multicast address but no multicast forwarding!!! 17 Nokia Siemens Networks 2012

18 PTP Time synchronization. Partial on-path support, not to be standardized by ITU for time being The slower than expected spreading of Synchronous ernet indicates that building full on-path support for time synchronization might take long. Partial on-path support scheme might come to speed up the development. Utilizes IEEE-1588 Annex A.9 or ITU Telecom frequency profile messaging. Unicast IP. NSN initial rule: Max 10 hops, max three hops without PTP support. The time constant of the clocks should be clearly smaller than in frequency mode * * * * * * S BC 3 BC 2 BC 1 M 18 Nokia Siemens Networks 2012 * Single IP address per BC for PTP is also possible

19 Conclusions PTP frequency synchronization with carefully designed robust algorithm has proved to be very reliable even in challenging networks. Attempting to fulfill tight phase masks like G.823 synchronization or SEC masks in other than high-quality optical networks results into erratic performance. Tens of thousands of PTP synchronized base stations have operated very well for up to 2 ½ years and so far without any alarming incidents. PTP has been widely adopted in cellular networks. NSN alone has delivered PTP-synchronization solution to 85 networks. The current situation with G network limits is unsatisfactory: Conservative metric and even more conservative limits have been chosen. PTP time synchronization is coming next. In the end, full-on path support will be spread widely. In the meantime, partial on path support may have a life span of several years. 19 Nokia Siemens Networks 2012