2014 Software Global Client Conference

Transcription

1

2 WW HMI SCADA-10 Best practices for distributed SCADA Stan DeVries Senior Director Solutions Architecture

3 What is Distributed SCADA? It s much more than a distributed architecture (SCADA always has this) It is distributed work, and the location of the work changes The following are examples: Solutions Center Operations Center A Operations Center B Field Operations Operations Center A Regional Operations Center Field Operations Operations Center B Operations Center A Regional Operations Center Operations Center B Field Operations

4 Factors Impacting Distributed SCADA Political Boundaries Jurisdictional Boundaries Geographical Boundaries Contractor Boundaries (Multiple SIs) Functional Boundaries Distributed SCADA can t be designed as a larger version of a small system

5 Distributed SCADA Requirements 1. Trustworthy in spite of the complexity and size, the information and functions can be trusted 2. Always On very high availability 3. Enables distributed work allows supervisors, applications and operators to frequently adapt to situations 4. Maintainable the system can be maintained without experts 5. Infrastructure time synchronization, access, shared databases etc.

6 1. Trustworthy Old SCADA proverb: garbage in, garbage out Distributed SCADA should detect the following failures: Bad data Broken network connections/failed network switches Failed servers/failed operating systems Failed applications Loss of datacenter/operations center/control room Cybersecurity failures

7 Trustworthy: Steps 1 & 2 of 5 Client Application Server Device Integration Store-forward Connection management Automatic data validation Control System

8 Trustworthy: Step 3 of 5 Client Application Server Device Integration Historical Gap On Failover Redundant Engines (active-passive) Redundant Device Integration (active-passive) Control System

9 Trustworthy: Step 4 of 5 Client Client Switching (active-passive)!! Redundant s (active-active) Alarms & Events Application Server! Redundant Engines (active-passive) Device Integration Redundant Device Integration (active-passive) Control System

10 Trustworthy: Step 5 of 5 Client Client Switching Redundant s Reliable Access Redundant Infrastructure Application Server Device Integration Redundant Engines Redundant Device Integration Reliable Collection Control System

11 2. Always On The following mechanisms are best practice: Redundant networks Redundant servers Redundant applications On-line backup and disaster recovery De-militarized zone (DMZ) for databases and historians Modular application design that minimizes unavailability when upgrades or repairs are made

12 Availability Definitions Fault tolerance continues to function during one (or more) failures Some users expect perfection (not feasible) High availability loss of function is short-term, and resumption is automatic Disaster recovery partially automated restoration of the function using automated reinstallation of the software Business continuity a set of plans, procedures and tools to allow operation to continue or resume despite serious incidents Resilience tolerance of serious incidents Recovery restore operations Contingency planning and training for resilience and recovery

13 Availability Comparison SIL 2 Availability % Downtime per year Downtime per month Downtime per week 90% 36.5 days 72 hours 16.8 hours 95% 18.3 days 36 hours 8.4 hours 98% 7.3 days 14.4 hours 3.4 hours 99% 3.7 days 7.2 hours 1.7 hours 99.5% 1.8 days 3.6 hours 50 minutes 99.8% 17.5 hours 86 minutes 20 minutes 99.9% 8.8 hours 44 minutes 10 minutes 99.95% 4.4 hours 22 minutes 5 minutes 99.99% 53 minutes 4 minutes 1 minute % 5 minutes 26 seconds 6 seconds How does this relate to SIL? it depends on safe and dangerous failures

14 Fault Tolerant Server Options Hardware Stratus FT Dual physical servers with lock-step processing and failover firmware, requires annual fee Software Stratus everrun Dual virtual machines with lock-step processing and failover software Requires larger servers (CPU cores and RAM) Note: other methods are not fault tolerant Clustering, live migration (VMware, Microsoft), network load balance (NLB) NLB can be used for InTouch and Workflow

15 Geographic Fault Tolerance the Basics When the network path is outside of a building, then the fault tolerance is geographic (i.e. not local) FT Replication Network requirements for bandwidth and latency are the same: 10 Gigabytes/second bandwidth 4 millisecond latency (maximum) Lower bandwidths and higher latencies can use high availability, which is not pure fault tolerance; some data can be lost Software, Hardware and Environment Failures must be handled separately

16 Normal Operation Control Room Redundant Availability Networks Control Room Availability Management Network Workflow InTouch Rack Servers FT Rack Servers HA Redundant PLC Networks PLC s

17 Failure of a PLC network connection Control Room Redundant Availability Networks Control Room Availability Management Network Workflow InTouch Rack Servers FT Rack Servers HA Redundant PLC Networks PLC s

18 Failure of a Server Control Room Redundant Availability Networks Control Room Availability Management Network Workflow InTouch Rack Servers FT Rack Servers HA Redundant PLC Networks PLC s

19 Failure of an Application Object Server Control Room Redundant Availability Networks Control Room Availability Management Network Workflow InTouch Rack Servers FT Rack Servers HA Redundant PLC Networks PLC s

20 Failure of one of the VM hosts or servers Control Room Redundant Availability Networks Control Room Availability Management Network Workflow InTouch Rack Servers Rack Servers HA Redundant PLC Networks PLC s

21 Failure of one of the Datacenters Control Room Redundant Availability Networks Control Room Availability Management Network Workflow InTouch Rack Servers HA Redundant PLC Networks PLC s

22 Break in all of the Geographic Networks Control Room Redundant Availability Networks Control Room Availability Management Network Workflow InTouch Workflow InTouch Rack Servers Rack Servers HA Redundant PLC Networks Storage Array Network (SAN) PLC s

23 3. Enables Distributed Work The location of operator control and the span of control changes frequently For safety, effectiveness and trustworthiness, distributed SCADA should use a mechanism that adapts location and span of control

24 SCADA Functions Supervisory control has been extended to include procedure automation One function of procedure automation is linking multiple subsystems (PLCs or RTUs) Automatic, scheduled reading and writing between subsystems Span of control means that alarms and ability to command the equipment moves between consoles or between control rooms

25 Adaptive Roles Information/ Activities Key Needs/Aspects Transformational Across State Collaboration Across Spectrum of Interfaces Machine Process Plant Site Enterprise Scale of Solution (Increasing Scope of Responsibility of Operator) Ease of Use Process Visualization Basic Alarming Short Term History Device Connectivity Larger Tag/Graphic Counts Graphic Standards Extensible Vis (Client Controls) Basic Reporting Basic Alarm Management Server Consolidation Standard Equipment Model Thin Client Centralized Fault Tolerance Situational Awareness Collaboration Cumulative Roaming Knowledge Working Multi-Plant HMI/Common Control Room Advanced Alarm Management Large Engineering Teams Common Standards Distributed s Collaboration Monitoring Operational Centers PIMS (Information Management) Distributed Sites/ Multi Site Distributed History Natural Collaboration Pro- Active Escalation Field / Task Scheduling Inspections Alarms Ack Alarms Mgmt Lock out Control Maintenance Setup Utilization Review Actions vs Roles

26 Transferring Control Between Locations Consoles have relatively flexible configurations for view, alarm and control Scope moves to the operator as conditions change In this example, Operation areas A & B uses control room or Desk 1 If Operation A needs focus, then an operator uses Desk 2 to control area B Shared/overlapping control is used to handle transitions between desks and operations (such as between Desks 1 & 2 and operations A and B ) Relaxed conditions allow all operations to be controlled from one desk A B C D Control does not permanently move to another location

27 Adaptive Span of Control

28 Alarming Alarms must be routed for desired and undesired situations Alarm management has 2 levels Basic: grouping, shelving, routing etc. Advanced: pattern recognition probable cause(s), probable consequence(s) Alarm processing must have redundancy Dual providers to multiple clients What should be done when a designated client fails? How does the system know? Prevent false negatives, false positives and duplicates Enable the compliance to standards (EEMUA etc.) Probable consequence Probable cause

29 Site () Regional Central 2.5K 5K Multiple Operations Centers 50 total remote Galaxy connections (10+40) C1 60K total I/O (50K+10K) HMI HMI HMI Galaxy R1 20K total I/O (10K+10K) R2 R3 R10 S1 10K total I/O S2 S3 S4 What are the bandwidths, mirroring times and failover times? S37 13 total remote Galaxy connections (9+4) S38 S39 S40

30 Deployment Example Test Environment Development Clients Runtime Configurators Storage Development Run Time * * Data Center Test Environment contained in a single VM running on the Physical Production Rack Server. The full software stack for both Development and Run Time is consolidated onto one virtual machine coupled with another whole instance of the VM running on the secondary physical redundant partner rack server for High Availability Production Environment Development Clients Runtime Configurators Storage Development* Development Development* Run Time * Production High Availability Partner Server Runtime Configurators Storage Deploy Disaster Recovery Update Replicate Plant Refinery Station Facility Run Time * Production Environment running multiple VMs running on the Physical Production Rack Server coupled with another Physical Production High Availability Partner Rack Server. The full software stack is distributed across multiple virtual machines to balance load and also to distinguish between development and run time. Backup Replicate

31 4. Maintainable Distributed SCADA is more complex than small SCADA It is easier to make configuration mistakes It is easier to overload a system: just because you can, doesn t mean that you should Best practices: Modular design, including Multi-Galaxy Maximize the use of automation and templates Good governance of changes and templates Just right amount of encapsulation Install and maintain subsystems for support and training

32 5. Infrastructure Distributed SCADA requires time synchronization The architecture should plan for the following: Update and antivirus applications Shared SQL Server database Demilitarized Zone (DMZ), often for tier 2 historians and SQL Server System/application health monitoring, e.g. Microsoft SCOM Cry out function using or fax (necessary to maintain availability and reliability)

33 2014 Schneider Electric. All Rights Reserved. All trademarks are owned by Schneider Electric Industries SAS or its affiliated companies or their respective owners. 33