Enterprise Communication Network Cisco ACE Load Balancer

Transcription

1 Cisco ACE Load Balancer Late Monday afternoon, April 17, 2017, the Cisco ACE Load Balancer went offline. This outage affected eschoolplus, Teacher Access Center and etextbooks. It was a team effort to resolve this issue. Oscar Villar, John Crumbley and a variety of team members were able to create a temporary work around to access resources internal to our network. First, Rich Crisock was able to write a software load balancer. Then on Tuesday, April 18, 2017, Lance Melancon was able to spin up a VMWare server to house the software load balancer. Next, Andrew Tran was then able to point etextbooks, eschoolplus, and Teacher Access Center to the new server and the students and staff were able to access resources internally. The above mentioned temporary solution provided access to resources only if a user was on our network. Staff and students were not able to access these resources from outside of our network. Therefore, the team devised a short term solution (referenced above), and intermediate solution, and a long term solution. Posted by: Andrew Tran, Senior Network Analyst

2 Cisco ACE Load Balancer In an attempt to provide a more comprehensive, intermediate solution, Andrew Tran was able to locate a module replacement for the Cisco Ace Load Balancer. Kim Bowlin was able to procure the replacement module and have it shipped over night. Wednesday morning, April 19, 2017, Andrew Tran installed the replacement ACE module and configured it. Andrew then restored eschoolplus, Teacher Access Center and etextbooks back to the Cisco ACE Load Balancer. Everything was restored to the original configuration and users were able to access resources both inside and outside of our network. A long term replacement strategy will be addressed as the team continues to migrate to the new virtualized environment. Posted by: Andrew Tran, Senior Network Analyst

3 On Monday morning, April 17, 2017, at 6:30am, there was a loud bang, smoke, and a strange smell on the third floor of ISC/W. Susan Elias, Customer Care Center (CCC) Analyst, called 911 and within minutes there were five fire trucks and several emergency vehicles on the scene. All ISC/W staff employees were asked to evacuate the building and relocate to ISC. Scottie Stevens, Paula Ross, Janette Martinez, and Jennifer Miller rerouted the CCC calls from ISC/W to ISC. The CCC team was temporarily located in the Technology conference room of ISC until ISC/W received the all clear notification.

4 Monday evening, April 17, 2017, CFISD hosted the annual Horizons Student Showcase in the Berry Center arena. Pictured left is Becky Cook, Director of Curriculum and Technology. Becky is working with Ray Leung, Paula Ross, plus Ken Wasko and Scott Lester of Layer 3 Communications. The team was onsite to support the inaugural use of wireless in the Berry Center arena.

5 Tuesday morning, April 18, 2017, Andrew Tran, Oscar Villar, Greg Rhodes, and Paula Ross conducted a meeting with team members from both Fortinet and M&S Technologies, with Ken Wasko from Layer 3 presiding. The collective teams conduct a status conference call each week to plan next steps for firewall installations. Below are the action items from Tuesday s meeting: M&S - Gather information needed for diagrams to be finalized Oscar/Andrew - review diagrams sent to CFISD from M&S M&S - reach out to Fortinet regarding the FortiBalancers to discuss best practices, etc. (per Ira suggestion) Oscar/Andrew continue to look for decision on the HA or Clustering preference, understanding additional talks with Anthony will be required following him reaching out for a FortiBalancer SME prior to coming to a final decision Ken Send meeting invite for Oscar/Andrew/Anthony for policy sync tentatively 10am on Friday, April 21 st

6 Oscar Villar, Andrew Tran, and Anthony from M&S Technologies met at 10am on Friday, April 21 st. The team worked on policy sync between the firewall at Data Center 3 and Data Center 4.

7 Wednesday, April 19, 2017, CFISD, Layer 3 Communications, and Aruba HPE conducted the weekly wireless meeting. At the highest level the agenda included the following items: Onsite final walkthrough status Outdoor APs Optimization settings for "sticky clients" complete Facilities IP Addressing sent to district Pridgeon Stadium significant discussion regarding flag pole installation and options Natatorium

8 The wireless team recently began receiving complaints regarding spotty WiFi coverage at a variety of campuses. This was perplexing for Ray Leung due to the fact that both he and Glen Carpenter were monitoring WiFi connectivity daily. In addition, Ray knew that final walk-throughs had been performed, sign offs were complete, and these campuses were in a production capacity. Therefore, the team was puzzled as to so what could be causing connection loss. Ray and Glen began tracing the cabling for the wireless access points (APs). The two discovered that when the cables were terminated, there is a piece of plastic between the wires that should have been removed when terminated. If not removed, this plastic could cause intermittent connectivity issues. Pictured to the left is Ray Leung reterminating a CAT6A cable for a wireless AP. Once the plastic was removed, the AP regained connectivity. Ray Leung and Paula Ross had a conference call with Presidio and MCA on Wednesday, April 19, 2017 to discuss the issue and determine a plan for resolution.

9 Kudos to Tony Rogers on two accounts: First of all, for taking the time to document a process for the department on how to avoid creating a network loop. Secondly, for attending the department s Service Technician meeting and sharing this information with the service technicians It takes time and effort to document and share this information, but it will save more time in the long run if we can avoid creating a network loop.

10 Posted by Michael Raspet, Network Analyst/NIC April 24, 2017 Radio Tower Project Status Update Recently, the ECN team learned about a new project called the Radio Tower Project. Michael Raspet visited the new radio tower installation sites at Cook MS, Cypress Creek HS, Cypress Woods HS, and Barker Cypress transportation center. These new towers will provide emergency radio communications for district employees in the Operations and Maintenance Departments. The towers are equipped with natural gas generators which can provide emergency power to operate and cool the electronics. The towers will be operational early this summer and need network connectivity.

11 On Friday,, Henry Deleon from Layer 3 Communications removed a Juniper switch that was temporarily installed in our network. Layer 3 Communications is a strategic partner of CFISD for multiple reasons. One example of this partnership is the installation of this Juniper switch. In November 2014, during the early design phase of our bond related projects, Layer 3 recognized that CFISD needed additional hardware resources. Since it was early in the design phase, CFISD was not in a position to determine the necessary hardware and procure the resources. As a temporary solution, Layer 3 Communications offered to loan CFISD a Juniper switch. This switch has been temporarily installed in a production capacity for over two years in the CFISD network. Henry Deleon Since the fall of 2014, CFISD has been able to procure and install a significant portion of our core network services. As a result, the Juniper switch is no longer needed. Thank you Layer 3 Communications for being a strategic partner to CFISD!