Virtualizing SCADA. Improving Control System Reliability with proven IT Technology. Speakers: Jason Hamlin Carter Farley

Transcription

1 Virtualizing SCADA Improving Control System Reliability with proven IT Technology Standards Certification Education & Training Publishing Conferences & Exhibits Speakers: Jason Hamlin Carter Farley 2014 ISA Water / Wastewater and Automatic Controls Symposium August 5-7, 2014 Orlando, Florida, USA

2 Presenters Jason Hamlin & Carter Farley Responsible for the care and feeding of process automation & control systems at the City of Lynchburg Regional Wastewater Treatment Plant 13 years experience in industrial electrical and process controls Jason Hamlin Carter Farley Director of Engineering for InstruLogic Corporation Responsible for program management and engineering oversight 18 year s multi-discipline experience in designing, planning, and executing projects. Emphasis on water/wastewater control and SCADA projects Aug 5-7, 2014 Orlando, Florida, USA 2

3 Presentation Outline What is Virtualization? What s a hypervisor Type 1 & 2 Advantages / Challenges What s right for you What do you need? Case Study Lynchburg, VA Regional WWTP Takeaway Why did we virtualize our SCADA Project hardware & software Screenshots (2) Benchmark tests Screenshots (2) Project costs Aug 5-7, 2014 Orlando, Florida, USA 3

4 What is Virtualization? Workingman s definition: The creation of imaginary yet usable computer hardware within a host of real world computer hardware. Practically speaking, virtualization is a tool that adds powerful flexibility to servers beyond their normal physical limitations. Not (usually) above the physical server s real world specifications Not limited by operating system Usually only limited by the stoutness of the real world hardware IT Alchemy, = 1 or conversely 1 = 4 (or more) Aug 5-7, 2014 Orlando, Florida, USA 4

5 What s a Hypervisor? Meet the Hypervisor Software which manages your Virtual Machine (VM) Resides on the host machine Serves as the liaison between guest machines and physical hardware The Two Main Types of Hypervisor Type 1 VM VM VM Hypervisor Type 2 VM VM VM Hypervisor Operating System Aug 5-7, 2014 Orlando, Florida, USA 5

6 Which type are you? Type 1 Hypervisor Hypervisor resides on the physical hardware of the host machine No host machine operating system Only job is to manage the guest machine operating systems Most any guest operating system (OS) can be used Common examples: Hyper-V, Xen, VMware ESXi Type 2 Hypervisor Host machine runs a standard operating system (OS) Hypervisor runs as a program on the host OS VMs are then run on the hypervisor at as a third level OS Examples: KVM* (Linux), VMware Workstation, Virtual Server (old Hyper-V) Aug 5-7, 2014 Orlando, Florida, USA 6

7 Advantages / Challenges Advantages Hardware Consolidation - Prevent server sprawl! Centralized administration Efficient use of hardware Redundancy, Redundancy! Testbedding/development Disaster recovery Fast! Counter to hardware obsolescence Easy migration Challenges Licensing Licensing Licensing All or nothing availability Administration = complexity Competent Employees Dedication to maintenance Cost of host hardware Vulnerability to viruses? Aug 5-7, 2014 Orlando, Florida, USA 7

8 What do YOU need? What is best for typical plant SCADA Depends on a few main factors Support of higher-ups Knowledge and dedication of staff / administration / higher-ups Relationship with your IT department Complexity of SCADA system (databases, backups, multi-clients) Depth of pockets Get the right system for your system or you ll fight with it more than it will help you Does your setup match the rest of your system? Are you running CVU with 5 screens? You may not be ready for virtualization Are there other areas of your system that would yield bigger benefits for the cost of virtualization? What s the opportunity cost? Aug 5-7, 2014 Orlando, Florida, USA 8

9 Case Study Lynchburg, VA City of Lynchburg Regional Wastewater Plant 13 MGD daily flow 60 MGD peak flow 70,000 Residents $7M Operating Budget Aug 5-7, 2014 Orlando, Florida, USA 9

10 Why did we virtualize our SCADA? Catastrophic data loss Traditional system of one application per server Factory hardware failure caused complete data loss of primary SCADA server Backup solution hadn t been tested and verified 18 hour recovery effort to get SCADA back online Some plant electronic historic data permanently gone How to prevent this again? All the above concerns answered via server virtualization Aug 5-7, 2014 Orlando, Florida, USA 10

11 Project Hardware and Software 2 New Servers (with 3 year next-day vendor support) Dual 6-core processors (24 logical processors) 6 SAS drives, 146GB, 15k rpm, RAID GB Total Hardware RAID controller, battery backed up 32GB ECC RAM 6 gigabit NICs Out-Of-Band management Hot-swappable, dual fans and power supplies Hypervisor (with vendor support contract) Backup / Replication Software (with vendor support) Never enough Ethernet patch cables Coffee Aug 5-7, 2014 Orlando, Florida, USA 11

12 Hardware - screenshot 1 Manage everything in a single pane of glass Aug 5-7, 2014 Orlando, Florida, USA 12

13 Hardware - screenshot 2 On-the-fly resource editing Guest hardware upgrades w/o downtime! Aug 5-7, 2014 Orlando, Florida, USA 13

14 Benchmark Tests Old system 3 Servers and a laptop RAID failure took 18 hours to recover to SCADA is running Final recovery took about 2 months Some plant historical data is gone no recovery possible Onsite/Offsite backup mishandled and didn t work when needed New System 2 Servers, a tablet, and a smartphone Primary Host failure recovery to SCADA is running in 7 minutes 24 hours of tested and verified hourly restore points 3 months of weekly backups Hourly restore points stored onsite in recovery Host Weekly backups stored offsite in two network storage locations Aug 5-7, 2014 Orlando, Florida, USA 14

15 Benchmark screenshot 1 Hourly backups are tested and verified with weekly reports! Aug 5-7, 2014 Orlando, Florida, USA 15

16 Benchmark screenshot 2 Failover test recovered and ready in 2.5 minutes! Spun up on a virtual network not live control network! Patches & upgrade tests w/o out harm to live system Aug 5-7, 2014 Orlando, Florida, USA 16

17 Project Costs Description Cost Hardware $11,000 Software $9,000 Labor * $? Total $20,000 * Project was done in-house, so one month of my salary Aug 5-7, 2014 Orlando, Florida, USA 17

18 Takeaway Virtualization is not an unobtainable It can be practical on a municipal budget Disaster recovery times can far exceed physical server methods Sandbox testing of patches and upgrades is unparalleled compared to traditional methods Ease of migration lessens time and cost of future hardware upgrades Adjustment of system resources on-the-fly is not available with traditional server methods IT people LOVE when control people play in their field Aug 5-7, 2014 Orlando, Florida, USA 18

19 Thanks for Listening Questions? Aug 5-7, 2014 Orlando, Florida, USA 19