Mega Event Testing Methodologies and benefits explained with examples from FIFA 2018 Florian Leeder
SIGOS Entire Solutions for Business Optimization Technical Perspective in Business Assurance Active Testing & Monitoring in heterogenous networks Active Testing & Monitoring across countries End User Perspective to Assure Your Business Service Management with Quality of Service End-To-End Service Management with Customer Experience Service Management with App Experience in enterprise and telco Commercial Business Assurance Revenue Assurance with Fraud Management Revenue Assurance with Billing & Tariff Verification Expertise & Know How to Assure Your Business Success Utilization of your tools with SIGOS senior experts Page 2
SIGOS Testing & Monitoring for Business Optimization Roaming and Interconnect World s largest roaming test system GSMA 4G reference system #1 roaming rollout and services Interconnect Fraud Detection #1 fraud detection solution used Most innovative & certified solution Fastest fraud detection available End-to-End QoS & QoE Testing SITE: world leading active test system Testing & monitoring QoS and QoE All technologies, all services App Experience Testing & Monitoring Apps Largest global device bank Used by over 600 enterprises and mobile network operators Billing & Tariff Verification Most comprehensive active RA Any stage, any tariff, any promo National and roaming RA OPERATIONS REVENUE Page 3
Mega events FIFA, UEFA, CAF, CONMEBOL, AFC 1,7 Mio International Visitors 32 Nations 4 Weeks Winter Olympics Summer Olympics Paralympics 5-7 Mio Visitors (20% foreigners) 200 Nations 3 Weeks Commonwealth Games 1 Mio Visitors 71 Nations 2 Weeks World Expo 10-20 Mio Visitors 150 Nations 4-6 Months Page 4
FIFA Russia 32 participating countries, 1.7 million fans, tourists, VIPs, media * 85 Million Roaming Business (50 AVG) Page 5 *source: http://tass.com/sport/997490
Special Offers https://www.t-mobile.com/coverage/roaming https://www.albawaba.com/business/pr/vodafone-egypt-negotiatesrussian-operators-offers-egyptian-fans-world-cup-1056286 https://alertify.eu/vodafone-spain-gives-away-free-1gb-data-on-spain-match-days/ Page 6
Who makes money? Both cakes value 85 Million! Megafon Beeline MTS Tele2 Roaming Partners Page 7
Host Strategy Operation Commercial Agreements Market Share Roaming Footprint Technology Page 8
Host Strategy Commercial Agreements Roaming Footprint Roaming Implementation Service FIFA 2006 South Africa: 444 new Roaming Agreements Page 9
Host Strategy Inbound Market Share Distribution Market Share A B C Preferred from OTA - - - MNO C - MNO B - - - Page 10
Host Strategy Operation Technology 24 x 7 Monitoring Private SIMMUX connected to GlobalRoamer for Roaming Partner s SIMs Intelligent reporting and alarming NOC Dashboard Management (SSI/SQI) Page 11
Visitor Strategy Steering Commercial Offers Best Roaming Partner(s) Operation & Customer Care Page 12
Visitor Strategy Best Roaming Partner(s) Pre-Event Roaming audit using GlobalRoamer to understand performance Undertake technical initiatives to have best commercial partner also best quality partner Potentially re-negotiate IoT and traffic volumes Page 13
Visitor Strategy User Experience Verification Steering SoR Platform Verification Forbidden Network Monitoring Location Update Success Not OK Page 14
Visitor Strategy Operation & Customer Care 24 x 7 Monitoring Intelligent reporting and alarming NOC Dashboard Management (SSI/SQI) Customer Care API Integration of Customer Care and GlobalRoamer SIGOS - GlobalRoamer (1) Request Test Execution (3) Send Test Results (2) Test execution Verification from Customer Care Page 15
Time planning Roaming Footrpint Commercial Agreements Technology Market Share Preparation 1 year before Implementation 6 months before Monitoring during event Best Roaming Partner(s) Commercial Offers Steering Customer Care Operation Page 16
Managed Service Our managed service offerings World Cup period: 15 June 15 July 2018 Jan 18 May 18 June 18 July 18 Implementation Phase Monitoring Phase Connectivity Package Quality Package Steering Package Page 17
Managed Service Packages Connectivity Package Simple connectivity check for the basic service in each local network with low roaming costs: Check Location Update PDP Context Activation Non Answered calls MO and MT in Roaming HTTP Browsing session (one URL) Min. 6 test cycles per day Automatic daily reports Connectivity Package KPIs Page 18
Managed Service Packages Connectivity Package Simple connectivity check for the basic service in each local network with low roaming costs: Check Location Update PDP Context Activation Non Answered calls MO and MT in Roaming HTTP Browsing session (one URL) Min. 6 test cycles per day Automatic daily reports Welcome SMS Quality Package Quality Checking Package and KPIs monitoring of QoS in each local network: Accessibility to Visited networks Voice Quality MO and MT in Roaming HTTP Download or data capacity SMS end to end performance MO and MT Welcome SMS verification 6 test cycles per day Automatic daily reports Different level of consulting available Page 19
Managed Service Packages Connectivity Package Simple connectivity check for the basic service in each local network with low roaming costs: Check Location Update PDP Context Activation Non Answered calls MO and MT in Roaming HTTP Browsing session (one URL) Min. 6 test cycles per day Automatic daily reports Steering of Roaming Quality Package User experience for SS7 Steering User experience verification in different scenarios Check the reject cause 12 test cycles per day per visited network Verify the SMS OTA content received from your customers SMS OTA content verification Test cycles according to OTA Platform behavior Checking and monitoring of QoS in each local network: Accessibility to Visited networks Voice Quality MO and MT in Roaming HTTP Download or data capacity SMS end to end performance MO and MT Welcome SMS verification 6 test cycles per day Automatic daily reports Different level of consulting available Page 20
Testing in Russia FIFA is important: 4 x more tests in Russia compared to January BUT: 75% of all tests in 2G/3G and 25% in LTE BUT: Legacy test cases most used ones # Test Case Percentage How much revenue is generated with SMS still? 1 SMS_MO_MT 24% 2 HTTP_ps 21% 3 VoiceQuality 22% 4 Speedtest_CLI 7% 5 BC_Voice_MO_MT 6% 6 Ping_ps 5% 7 ICQ_Basic 4% 8 HTTP_Download 4% 9 HTTP_Browser 3% 10 Activate_PDP_Context 3% Page 21
France vs Peru France vs Peru in Yekaterinburg on 21st June 2018 at 17:00 (CEST) Page 22
0 4 8 12 15 16 17 18 20 0 4 8 12 13 15 16 17 18 19 20 4 13 16 17 18 20 1 5 9 14 15 16 17 18 19 21 Call Setup Duration [seconds] Roamer to Home Voice Performance Roamer to Home Network - Hourly trend on 21st June 2018 120 70 100 60 80 50 60 40 30 40 20 20 10 0 0 ru beeline ru megafon ru motiv ru mts Visited Roaming Partner in Russia - Location Yekaterinburg Average of Call Setup Success Rate Average of Call Setup Alert Duration Page 23
0 4 8 12 15 16 17 18 20 0 4 8 12 13 15 16 17 18 19 20 4 13 16 17 18 20 1 5 9 14 15 16 17 18 19 21 Call Setup Duration [seconds] Roamer to Home Voice Performance Roamer to Home Network - Hourly trend on 21st June 2018 70 4 60 3,5 50 3 40 30 2,5 2 1,5 20 1 10 0,5 0 ru beeline ru megafon ru motiv ru mts Visited Roaming Partner in Russia - Location Yekaterinburg 0 Average of Call Setup Alert Duration Average of MOS_Roamer Listening Average of MOS_Home Listening Page 24
CLI Issues Calls from Roamers visiting MTS Russia to Home Network CLI issues were observed for calls from roamers visiting MTS Russia to home subscribers. The received CLI was of Israel. In some cases a blank CLI was displayed. Page 25
0,00 4,00 8,00 12,00 15,00 16,00 17,00 18,00 20,00 0,00 4,00 8,00 12,00 13,00 15,00 16,00 17,00 18,00 19,00 20,00 1,00 4,00 5,00 13,00 16,00 17,00 18,00 19,00 20,00 1,00 5,00 9,00 14,00 15,00 16,00 17,00 18,00 19,00 21,00 SMS end to end duration [seconds] Home to Roamer SMS Performance SMS Home to Roamer Network - Hourly trend on 21st June 2018 140 120 100 80 60 40 20 0 ru beeline ru megafon ru motiv ru mts Visited Roaming Partner in Russia Page 26
0,00 4,00 8,00 12,00 15,00 16,00 17,00 18,00 20,00 0,00 4,00 8,00 9,00 12,00 13,00 15,00 16,00 17,00 18,00 19,00 20,00 13,00 16,00 17,00 18,00 19,00 20,00 1,00 5,00 9,00 14,00 15,00 16,00 17,00 18,00 19,00 21,00 End to End SMS Duration [Seconds] Roamer to Home SMS Performance 160 SMS Roamer to Home Network - Hourly trend on 21st June 2018 140 120 100 80 60 40 20 0 ru beeline ru megafon ru motiv ru mts Visited Roaming Partner in Russia Page 27
Round Trip Time [seconds] Packet loss [%] Data Ping Scenario 5 Data Ping Scenario - Hourly trend on 21st June 2018 35 4,5 4 3,5 30 25 3 2,5 2 20 15 1,5 1 0,5 10 5 0 0 4 8 12 15 16 17 18 19 20 0 4 8 12 15 16 17 18 19 20 5 13 16 17 18 19 21 1 5 9 13 15 16 17 18 19 21 ru beeline ru megafon ru motiv ru mts Visited Roaming Partner in Russia 0 Average of Rount Trip Time Average of Packetloss Page 28
What s next? UEFA EURO 2020: 12 countries, 24 participants Tokyo 2020 Olympic Summer Games (5G launch) World Expo 2020 Dubai Beijing 2022 Olympic Winter Games FIFA Worldcup Qatar 2022: 32 participants Others? Talk with us! Page 29
Thank you www.sigos.com info@sigos.com Page 30