AARNet Network Operations. Network Operations. Customer Alerts & Maintenance NOC. Questnet 2009 Mike Groeneweg. Copyright AARNet Pty Ltd

Size: px

Start display at page:

Download "AARNet Network Operations. Network Operations. Customer Alerts & Maintenance NOC. Questnet 2009 Mike Groeneweg. Copyright AARNet Pty Ltd"

Julian Higgins
5 years ago
Views:

1 AARNet Network Operations Customer Alerts & Maintenance Network Operations Questnet Mike Groeneweg NOC 2 1

2 Network Operations NOC 24 hour phone service (1300 APL NOC or ) is monitored during business hours noc@aarnet.edu.au NOT your_favourite_engineer@aarnet.edu.au engineer@aarnet edu au On-call engineer is from a pool of Network Engineers & Systems Administrators from AARNet Operations. 3 NOC IP Routing CPE Operations & Maintenance Peering Backbone Circuit Monitoring Optical Circuit Co-ordination Traffic Billing, Capacity Management & Planning New service development & deployments Maintenance scheduling & notifications Emergency fault notifications 4 2

3 Network Engineers 5 20 Pops, 150+ routers 6 3

NOC E-mail We get a lot of mail, sometimes things get missed.

2327 October 4554 April 4205 November 4682 May 2318 December 2391 June 1941

4 NOC We get a lot of mail, sometimes things get missed July 4197 January 3038 August 3471 February 2826 September 4130 March 2327 October 4554 April 4205 November 4682 May 2318 December 2391 June 1941 TOTAL* Includes all Alarms, Notifications, Peering, SPAM & Customer e mails 7 NOC 8 4

5 Maintenance 9 Customer Maintenance Events The NOC would always like to be told about your upcoming maintenance event. But how? Currently an to noc@aarnet.edu.au with a subject of Upcoming maintenance xx/yy at hh:mm But what timezone is this? How long? Which CPE in particular, and is it the whole rack/room/building/site/state offline? 10 5

6 Customer Maintenance Events The NOC then will do the following: Create a ticket (NOCTTS-xxxx) Set a reminder in the Ops Diary Inform other NOCs Inform other affected parties (if any) Send back an acknowledgment 11 Maintenance Event Notifications SITE-L mailing list or aarnet-ops-contacts@customer.edu.au 120 Site-L messages in (1/7/08-30/6/09) 100 topics Is the best method? Multi-site customers & multiple engineers well co-ordinated? 12 6

Site-L inconsistency Messages are manually generated which leads to different styles & accuracy SUBJECT: Outage AARNet Hawaii to Seattle 10G SXTransport (North) circuit SCOPE: SUBJECT: 10G backbone

7 Site-L inconsistency Messages are manually generated which leads to different styles & accuracy SUBJECT: Outage AARNet Hawaii to Seattle 10G SXTransport (North) circuit SCOPE: SUBJECT: 10G backbone Site Power circuit Works CORE SCOPE: NODES: HNL A BB1, AARNet SEA A BB1 Perth B Pop (PER B, CSIRO ARRC, Kensington) DESCRIPTION: CORE SUBJECT: NODES: A 10G PER B BB1 OUTAGE circuit SB2 between SYDNEY BRISBANE Hawaii and Seattle is currently down. AARNet, START CORE SNOC TIME: NODES: and PLNI Saturday syd a bb1 are currently 27 bne a bb1 June working 06:00 to diagnose AWST (22:00 the UTC 26/6) END TIME: START TIME: problem. Saturday 1:00 The am, circuit 27 June 21/04/09 has been 16:00 down AWST since (08:00 3:25AM UTC on27/6) 18/06/ DESCRIPTION: END TIME: Scheduled 06:00am, Site 21/04/09 Power works requiring complete mains power AFFECTED: shutdown. DESCRIPTION: None, Repairs Traffic to is the being carriers rerouted network via SXTransport ( next gen (South). ). TICKET AFFECTED: NO: AFFECTED: NOCTTS 2616 None. Generator and UPS power to PoP. CONTACT: TICKET NO: NOC NOCTTS 2524 The APL gig NOC link (1300 between ), Brisbane noc@aarnet.edu.au and Sydney will be down for 5 COUNTER: CONTACT: hours. IP Message traffic NOC #1 will 1300 re route APL NOC over (1300 another ), circuit. noc@aarnet.edu.au SDH circuits on this run, COUNTER: L2 curcuits that Message span #1 Sydney to Brisbane will be down for the period. There should be no outage of ip Traffic will re route over SB1. TICKET NO: aarnet NOCTTS Maintenance Event Portal? CPE portal improvement an event list? Any customer engineer can see what s happening, if they missed the original notification or reminder

8 Optical Circuits 15 Optical Circuits Most circuits on single leg. But some circuits on 2 sections and a few go the long way around. 16 8

9 Optical Circuits - Notifications Operations works with Infrastructure Group to Identify Optical Circuits that will be affected. Currently a manual process. Automation will happen. AARNet is deploying Connect Master to track the devices & fibres but we re still a way off yet ConnectMaster 18 9

10 Fault Notifications Emergency Maintenance 19 Customer Contacts Who to ring, and when? Are they on leave, sick, no longer associated with the organisation? Up to date contact t information, is sometimes, not very up to date. doesn t work, if you ve got a network emergency

Customer Contact Portal? Would a customer contact portal, per organisation be useful? E-mail? SMS? Phone call? Twitter? What is the preference? And when to ring?

As it wasn't monitored, the datacentre support team didn't notice the loss of the chilled water.

11 Customer Contact Portal? Would a customer contact portal, per organisation be useful? ? SMS? Phone call? Twitter? What is the preference? And when to ring? 21 After-hours? To call or not to call? the brown-out tripped the chilled water system, cutting chilled water to the hospital campus. As it wasn't monitored, the datacentre support team didn't notice the loss of the chilled water. A datacentre t employee came on scene to check everything was running, but being happy that there wasn't anything wrong, he left

12 Questions? 12

Service Level Agreement

Service Level Agreement General This document is a (SLA) setting out the indicative levels of services to be provided to you by Luminet. This document must be read in conjunction with our standard Terms and Conditions of Service