open eir Training Day Guide to UG, Fault Handling & FTTH

Transcription

1 open eir Training Day Guide to UG, Fault Handling & FTTH

2 Introduction Sinead O Gorman Head of Customer Service

3 Introduction - Welcome The open eir customer care team continually looks for ways to improve our performance. The overall objective is always clear in everyone s mind, to streamline and optimize the delivery of faults, regardless of which operator is serving the end customer.

4 Introduction open eir (oecc) Team Structure

5 01 Fault Handling Joe Phelan & Gerry Loughney

6 FAO Repair Organisation Field- Centre Model, with centralised management of the repair workload from the Works Management Centre in Sligo. Field service repair resources = 868 with up to double this resource in winter storms. Structure and organisation: 1250 exchanges. 33 teams areas, geographic areas containing 10 to 55 exchanges. 13 nationally mobile teams (hit squad) of technicians. 5 cabling teams for infrastructure repair.

7 FAO Repair Organisation Technicians cover ~50km radius to faults, less in the large urban centres. Default in to a subgroup (primary area) of ~10 exchanges. Jobs loaded on to technician s mobile device from 5am. Technicians go directly from home to their first job. Hit squad Technicians are mobile and can deploy to anywhere in the country at 24 hours notice. On going support from our Contractors for civils work, poling & cabling work Capacity planning function integrated into work management: Resources deployed to match workload on a pre-planned basis Dynamic re-allocation of resources in response to workload variations

8 Products repaired PSTN - Basic telephone service ISDN - Digital line usually used by companies as part of a integrated phone system ADSL - Normal broadband anything from 2 megs to 24 megs LLU - Local loop unbundling where operators use their own core equip in the exchange LS - Products are split between operators (Voice with one BB with another) FTTC - Fibre broadband served from a Cabinet near the premises EVDSL - Fibre broadband served directly from the exch (line not fed through a cab) FTTH - Fibre to the Home high speed broadband delivered over fibre to the customer s premises

9 Exchange Exchange This is where we connect all our customer s to switching equipment. Each individual customer s cable pair is connected to exchange equipment (LI and ADSL or evdsl ports.) All customer services are routed through our exchange and core networks

10 Cabinet Distribution point FTTC cab Cabinet Roadside Cabinets are used to distribute cable pair s to individual customer premises The recent roll out of superfast broadband (NGA) is provided using the FTTC cab situated alongside. Powered by electrical feed. DSL Splitters FTTC cabinets are fibre powered and contain NGA ports to connect to our copper network

11 Centralised work management summary During storm recovery periods when backlogs are high, daily reports are issued showing average SLA days per county in the storm impacted areas During normal workload periods we clear 84% + of faults within 2 working days The remaining 16% generally fall into the non-standard fault category (e.g. survey, cabling, poling, DUG, etc) Daily service plans (based on team fault volumes, age profile, resource availability, etc) are used as a tool to optimise our SLA performance Jeopardy management of faults approaching their SLA expiry times Jeopardy management of customer appointments Escalation management Field resource utilisation: Pre-planned deployments Reactive re-allocation of resources when required to manage local peaks Deployment of support / incremental resources Overtime allocation and control

12 Standard / non-standard faults Standard faults - most fault reports are dispatched directly to our repair crews around the country and are completed by them within SLA Non-standard faults - some faults cannot be completed directly by our repair crews and require additional network intervention. Examples of non-standard Faults are: Poles required Cable replacement required Civils underground duct and boxes DUG lead replacement Faults that require survey, works orders, specific customer interaction or investigation, etc, are assigned to the local field manager

13 Non Standard - Overview

14 Non standard - Poles

15 Non standard - Cabling

16 Non standard Civils/UG

17 Works Order Types CUD underground civils work, e.g. duct and boxes (will require a LA licence in practically all cases) CUC / MLU underground cabling/jointing (may require a LA licence in some cases if blockages in the duct line are encountered) CPO poling (may require a LA licence in some cases if the pole is on public property) CCO overhead cabling (generally does not require a LA licence) Faults that are dependent on works orders will have a note pasted into them from the works order system the time stamp on these notes will be 2am This note will include an indication as to whether the fault is awaiting a LA licence at that time Information on how to interpret the acronyms in the works order note is included in the glossary of terms Escalations can be raised on faults associated with works orders however the dependency on the completion of the works order will remain

18 Examples of W/O notes on faults..

19 Logging a fault report Testing & Screening / Troubleshooting with End User Electrical test completed CPE isolation Synch test completed if BB fault Interpret the troubleshooting results Determine if the fault is in the open eir network If yes, proceed to log the fault report and confirm that access will be available if required and customer contact details are correct

20 Adding value to a fault report If indications are that access will be needed to the End User s premises - book a customer appointment on the fault If the End User lives in a gated development or an apartment complex the customer must ensure that access to the development / complex is available for the repair technician when he calls When booking customer appointments in UG make sure that the End User is aware of the booked appointment date / time Check that the End User liaison number is up-to-date and accurate and that the End User can answer an incoming call on that number during working hours: 9am 5pm If a fault report is cleared 003 with a note indicating that the End User s premises cannot be found the End User s eir code should be included in the fault notes when it is sent back to open eir

21 Glossary of Terms Term Access Network O/H U/G Chamber / JB / manhole / Joint Box Poly W/O (or Works Order) KNN / KNIS Civils Rurtel Explanation The network from the local exchange to the customer s premises Overhead cable on a pole route Underground cable or plant All terms to describe an underground box (may be in the footway or carriageway) that contains open eir cables & joints Polythene insulated cable - may be overhead or underground A job-pack to carry out a network repair job. May involve cable and/or poles and/or underground plant replacement Contractor that carries out work for open eir Repair work on underground duct or chambers Analogue radio based system for the provision of telephone service FWA FCS Fixed Wireless Access: radio based system for the provision of telephone service Fixed Cellular Service: telephony service provided at a specific location using the Meteor mobile network

22 Glossary of Terms Term MDF Cab or Cabinet Explanation Main Distribution Frame (the cross connection frame in every exchange where the exchange equipment (LI s) are cross-connected to the cables serving customer s premises. The cross connection point where main cables from the exchange are jumpered to smaller distribution cables serving DP s DP CD NTU For BU outside WM S/C DIS DW LI Distribution point (the point to which individual customer drop-leads are connected to exchange / cabinet cables Wall mounted distribution point (external or internal) Network Termination Unit (eircom point/socket) For Business Unit outside of Works Management (fault not appropriate for dispatch to field access network technicians) Short Circuit (Fault condition on line where both legs of the copper pair are in contact) Line Disconnected (line broken at some point between exchange and customer premises) Drop Wire (wire between pole and customer premises) Line Interface (the termination that a customer s line is connected to at the exchange MDF for dial tone)

23 Glossary of Terms W/O related Term (Works Order Related) CoCo Contr COS ETA FC FFPU LA LIC APP LIC GRA NONF A NONF D NONF L NONF M NONF R NONF S Explanation County Council contractor crew on site estimated time of arrival forecast or re-forecast fit for purpose (works order completed) local authority licence applied for (+ date) licence granted awaiting third party dependent order private wayleave awaiting customer awaiting re-plan awaiting stores

24 Glossary of Terms W/O related Term (Works Order Related) NONF T NONF W NONF X NRA OPFF REFC TII Explanation awaiting DFE approval licence application (LA) awaiting cancellation National Roads Authority (see TII below) operation freeflow re-forecast Transport Infrastructure Ireland (formed when the National Roads Authority and the Railway Procurement Agency merged) WO or w/o LA Licence REL_ASIG REL_NONF works order Local Authority licence or permit to carry out works on underground duct or chambers SAP statuses on Works Orders. Effectively means that the works order is assigned to a resource for attention. Works order is non-fluid and cannot progress at present. Usually awaiting LA Licence. Note in FHS should clarify

25 02 Fault Process and Routines

26 What is a Fault Report? A Fault Report is where an End Customer has an issue with a Product or Service and contacts their Operator the Operator s Reception area must determine if the customer s issue is a valid open eir Network Fault. All product fault reports can be made via our Unified Gateway. The Service Assurance Team within our Wholesale customer contact centre provide a fault reception and end to end fault management facility and act as a single point of contact throughout this process. For interconnect faults please contact our OLO desk at our Network Management Centre in Citywest, at Olodesk@openeir.ie

27 Benefits of improved Analysis at First Point of Contact The End Customer experience of how they are treated by their Service Provider is paramount in securing customer satisfaction & customer retention Fault management is about treating every customer s Fault report with respect and consistency in understanding what their issues are and dealing effectively with them to have them resolved Effective screening and lower non faults will improve customer satisfaction Benefits to the Operators are: Improved fault identification and technical support Operators can focus efforts on allocating resources where they are genuinely required Improved fault clearance time Benefits to open eir are: Ability to focus efforts on genuine faults

28 What is a Decision Tree? The Decision Tree is a set of Routines that the Customer Facing Centre should follow in order to analyse the issue that the End Customer has It looks at how an agent should deal with a call when presented with Line Test Results including when there is no test result or when the line is testing ok. Open eir developed this by identifying and documenting the Best in Class practice for dealing with Narrowband and Broadband issues.

29 Front End - Decision Tree Line Test Conditions DC Voltage AC Voltage DIS Earth Short Circuit No Test Testing OK Routines Outage Dangerous Plant ELF ISDN CPE Repeat Carrier FCS Testing: The objective of the Testing aspect of Decision Tree Implementation is to ensure that Customer Facing Centres understand how to interpret Test Results and know what to do when a line is testing faulty, testing ok or where no test is available Routines: The objective of the Routines aspect of the Decision Tree implementation is to ensure that CFCs know how to process a report when dealing with certain specific elements of a report

30 Overview Reception PSTN Process

31 Report Codes Wholesale Operators use the Unified Gateway to report a fault. It is the responsibility of all Operators to ensure that only valid Network Faults are reported to open eir Field Operations. Report Codes Report Codes are selected by the Agent taking the Call from the End Customer and must reflect the issue being reported. The Report Codes are selected via UG or Web Services. The Report Codes should be supported by clear and concise Comments based on the analysis at the first point of contact.

32 High Level Flow

33 Outages Scheduled Outages Change Management Planned event Notification to relevant group e.g. upgrade of exchange equipment Unscheduled Outages Unexpected event Incident Management e.g. 100 pair cable dug up

34 CPE Isolation Routine The termination point at the customer end of the open eir network is the NTU (Network Terminating Unit) Faults in the network up to and including the NTU is the responsibility of open eir. Faults in the internal wiring after the NTU are the responsibility of the customer. Faults in the CPE equipment are the responsibility of the customer unless the equipment is rented or under warranty. Useful Tips re CPE Customer should be at the premises Check for recent CPE changes Check for recent construction work at the premises Confirm the fault status at the NTU Confirm that alarm is wired correctly Advise Customer of Possible Charge if no fault found

35 CPE Isolation Step by Step Process Step 1 Is customer at fault location? If No ask customer to ring back from fault location If Yes - ask customer for mobile number and call back Step 2 - Is problem with incoming calls? If Yes Check ringer switch on phone is correctly set If fault is cleared - inform customer and clear fault report Step 3 Is CPE rented? If Yes and CPE is faulty inform customer to change CPE or contact CPE supplier If No Continue Step 4 Ask customer to unplug phones & equipment (e.g. Sky Box, Alarm, other CPE etc) Retest and compare with previous test Step 5 Is fault cleared? If Yes inform customer and clear fault report If No - Ask customer to plug phone into NTU & resume analysis Step 6 Is fault cleared? If Yes inform customer and clear fault report Otherwise resume analysis

36 Repeat Fault - Fault that is reported for a second time within 28 days Step 1 - Reception Agent checks the Fault Report History looking at the previous Clear and Report Codes including Comments and establish any related issues. Does Repeat History invalidate current Report? If Yes Reject Fault Report If No - Continue Step 2 Was previous Clear 003/002? If No - Log Fault if appropriate with supporting Notes If Yes Continue, paying careful attention to the note from the technician on the previous fault Step 3 Is Line Test indicating Fault in the Open Eir network at the customer s premises? If Yes Create an Appointment If No Contact or Network DIS Log Fault using appropriate Report Codes with Supporting Notes

37 Repeat Routine - <28 days

38 Line Testing 03

39 Testing Short Circuit DIS Earth AC Voltage No Test Result DC Voltage Testing OK

40 Testing In this section we are looking at Routines that should be followed by the Agent when presented with certain line test conditions: DC Refers to Direct Current an exchange generates 50volts DC on a PSTN Circuit AC Refers to Alternating Current generally this is 220volts AC generated by Power Supply Note always a Health and Safety issue and requires a Warm Handover (Threshold 10 volts AC) Short Circuit is when the two wires of the same circuit are touching Earth is where one or both legs of the Copper Pair are in contact with Earth DIS A Dis is where one or both legs of a circuit have been disconnected. The Copper Pair Test will display the above Test Conditions there are Routines below in relation to each Test Result Note: that the Line could be Testing OK or No Test Result may be presented in both cases there are Routines to be followed.

41 Testing Most faults are caused by the deterioration of the Insulation around the copper wire this can lead to Earths, Short Circuits, DC Contacts and even AC contacts and eventually a DIS condition. We can have Outages where more than one customer is affected by a common single fault e.g. faulty exchange cards, cable damaged by builders or contractors - some outages may be Scheduled and others Unscheduled such as Storm Damage or where there has been an accident. One other aspect of testing that must be taken in to consideration is Capacitance Balance Readings which look in particular at the A to B element of the Test. Note: Line Test interpretation is a key enabler in prevention of wasted Truck Rolls

42 Line Test Characteristics

43 Short Circuit Short Circuit

44 Short Circuit UG Screen Short Circuit Short Circuit - Insulation value <.17M (170K)

45 Useful Tips on Short Circuits Some exchanges can switch off Dial Tone when presented with a Short Circuit you may hear the terms Line Lockout or ABL (Auto Blocked Lines) (Note this can be caused by a Phone Off Hook or by some customer premises equipment as well as a Line Fault Condition) If the customer is not calling from their premises then the Agent should request that they report the Fault from their premises When the Agent is presented with a S/C the agent should request the customer to disconnect all CPE and re-test

46 DC Voltage DC Voltage

47 DC Voltage UG Screen DC voltage DC voltage - Value > 5V

48 DC voltage > 5 volts - Step by Step Process Compare Line test result with line Signature Step 1 If it is better or same Then If Yes Check for evidence of electrical deterioration If No Follow CPE Isolation Process If this resolves the fault then issue Clear ticket Step 2 Is this a repeat Fault? If Yes Follow Repeat fault process If No Log fault using appropriate Report Code/s Plus supporting notes

49 DC Voltage

50 04 NGA Fibre to the Cab (FTTC)

51 FTTC Next Generation Access (NGA) Overview FTTC NGA ( Next Generation Access) refers to fibre based networks that offer Broadband Speeds up to 100Mbs. These speeds will support Next Generation products

52 Product Variants POTS based & Stand Alone Broadband Fibre to the Cabinet with POTS (PSTN) FTTC + POTS Based (PB FTTC) Fibre to the Cabinet Stand Alone FTTC Stand Alone (SA FTTC) Fibre to the Home with POTS FTTH + POTS Based (PB FTTH) Fibre to the Home Stand Alone FTTH Stand Alone (SA FTTH) Exchange Launched VDSL evdsl Standalone ADSL SA ADSL

53 Demarcation Points RAP (Regulated Access Product) NON RAP PSTN Network PSTN Switch Copper Cable Roadside Copper Cab DPE Only Splitter Phone NGN Network Exchange Fibre Cable DSLAM Roadside FTTC Cab NTU/ Splitter Modem/PC FTTC Customer Internet Provider Network 53

54 VDSL Decision Tree.

55 Testing on FTTC MELT Test (Copper Pair) Sync Test Sync History Performance SELT Test CPE Checks

56 Metallic Line Test Faults on Copper Pair DC Refers to Direct Current an exchange generates 50volts DC on a PSTN Circuit AC Refers to Alternating Current generally this is 220volts AC generated by Power Supply Note always a Health and Safety issue and requires a Warm Handover (Threshold 10 volts AC) Short Circuit is when the two wires of the same circuit are touching Earth is where one or both legs of the Copper Pair are in contact with Earth DIS A Dis is where one or both legs of a circuit have been disconnected.

57 SYNC Test The following information is retrieved: Port status Actual bit rates upstream and downstream Attainable bit rates upstream and downstream Attenuation Noise Margins Last Synch time Last loss of Synch Showtime (seconds) This information gives the user a snapshot of the state of the line at the point that the test is done.

58 Sync Test used for FTTC, FTTH and SADSL

59 Operator Screen on UG synch check Attenuation: :signal loss over copper pair lower better Profile Code Downstream_Upstream_Type i.e. Provisioned bitrate_bitrate_ratetype Attainable bitrate: : maximum current bitrate capability of the local loop may never be reached in practice Noise Margin: :difference in strength between the VDSL signal and the background noise on the line. Higher better Actual Bitrate: current bitrate at which modem is synched Last Synch time: date and time of last synch

60 SYNC Test History

61 Melt Test

62 Performance Test

63 Performance Test

64 Number of initialisations (Synchronisation): Number of re-synchs that the VDSL modem has completed with the network DSLA Performance Test This test interrogates the DSLAM and retrieves the performance counters for the NGA VDSL2 port. These measure line performance characteristics over 15 minute and 24 hour intervals. Typical line performance parameters are: Corrected Blocks: The number of data blocks which had errors corrected in the period (not seen by end-user) Uncorrected Blocks: The number of data blocks which had errors not corrected in the period Errored seconds: Number of seconds containing at least one error Severely errored seconds: Number of seconds with errors above a certain threshold (typically 10-3) CRC errors : Errors in Cyclic Redundancy Check Checksum in digital bitstream

65 Performance Test

66 Test Performance for NGA (TPN) This data comes from 15 minute and 24 hour error counters in the DSLAM Low numbers are better Results for Current 15 min period, Previous 15 min period Current 24 hour period Previous 24 Hour Period

67 Test Performance for NGA (TPN) (2) End-user Impact Nil Low Parameter Corrected blocks in Current 15 Min Uncorrected Blocks Current 15 Min Errored Second Severely Errored Second Code Violation CRC Errors Initialisation count Explanation Number Data blocks which have had errors corrected in them in the present 15 min period Number Data blocks which have had errors not corrected in them in the present 15 min period Number of Seconds containing at least 1 error Number of Seconds with errors above a certain threshold Errors within a Data Block Error in CRC Checksum in digital bitstream Number of resynchs High Failed Initialisation Count Number of resynch attempts where the modem failed to come into showtime first time around

68 05 Fibre to the home (FTTH)

69 Overview of Design Completely new network.

70 FTTH Fault diagnosis The service provider will carry out the following: Establish that they are dealing with an FTTH service Correlate the fault against network faults or incidents which open eir has raised as an external trouble ticket on UG via the Outage Alerts Tab Complete Level 1 Fault Management and proves the fault out of its network and CPE Level 1 Fault Management: Level 1 Fault Management consists of high level diagnosis that takes place between the service provider and the customer It is the process whereby the service provider determines that the fault lies with one of the following: The open eir network The service provider network CPE

71 In-house installation In-house installation Modem NTU ONT

72 Troubleshooting Fibre network faults One test required synchronisation (sync) test If ONT is active and synched and RX power level is >= -22dB there is no fault in the FTTH access network Based on our current product offering additional information is presented in the synch test result that can assist the Operator during in-home troubleshooting with their customer: Line profile Last up time and last down time Last down cause MAC address ONT serial number

73 Fibre fault causes LOSI

74 ONT LEDs Part 1 LAN Power

75 Overview Reception Process

76 Overview FTTH decision Tree

77 ONT Routine

78 ONT-Modem Connections Routine

79 ONT-NTU Connections Routine

80 Power Routine

81 Synch test results Current profile: Profile cannot be changed, Package can only be upgraded or downgraded: MAC address: From modem confirms modem is connected : Receive power level: divide by 100 to get value in dbm Port status Last Down Cause: reason broadband dropped the last time it was offline ONT serial number : Power level threshold is the same for all packages The result shows that the ONT is up and a signal is being received The result indicates that there are no issues on the network side A fault should not be logged without troubleshooting of the CPE

82 Fault validation Front End & UG Validation steps to be taken by agent before logging a fault: Query fault with end user and status of equipment in customer s premise Reset ONT Synch Check Synch History Check ONT port profile Fault Validation rules on UG Fault rejected if operator has not attempted the TSB test in the last hour when logging a FTTH fault. All other report codes a synch test is performed to check the ONT status and the RX power level if ONT is UP and RX power is better than -25dBm UG will reject the fault. If the ONT is down and the last down cause is dying gasp (mains loss) Report code 386(ONT not responding) -> fault is accepted Any other report code used -> fault is rejected

83 Fault Reporting Form - FNH It is imperative that the correct report code is used when logging a fault and that all questions are correctly answered to ensure an efficient resolution of the issue

84 Fault Reporting Form - FNH It is imperative that the correct report code is used when logging a fault and that all questions are correctly answered to ensure an efficient resolution of the issue

85 Value Add for Call Reception Centres Eircode add to fault notes especially for rural unstructured addresses if last clear was 003 stating premise could not be found Confirm customer liaison number is valid and that calls will be answered during normal working hours In the case of Short Circuit faults and on ADSL/VDSL faults access to premises will probably be required confirm access arrangements with customer and consider booking an appointment In gated communities or apartment blocks access to the DP or cable distribution will be required agree access arrangements with customer when logging fault rep

86 Wrap Up. Evaluation survey will be sent out shortly All Training Material is listed on the open eir website Thank You!!!

87 Thank You