Specification For SCADA and PLC Architecture. Pr9833

Transcription

1 Specification For SCADA and PLC Architecture Pr9833

2 Documents Details This document is only valid on the day it was printed. Version Review Revision Reviewed by Approved by Date approved Revision type /11/2010 Draft for Review /11/2010 Submission for Review /06/2011 Issued for Interim Use /07/2011 General Revisions 0.5 1/03/2012 Addition of Siemens PLC Architecture 0.5 General Revision 1.0 T.Stewart A.Schoenmaker 6/04/2016 Finalisation of review and uploading to Written Direction 2.0 J. Duncan 26/10/2016 Technical review for next project Written Direction Control Document Sponsor Document Author (Owner) Subject Matter Expert References Infrastructure Technical Standards Committee Capital Delivery Manager, SIS Control Systems Engineer, Network Operations Refer to Section 3 of this document Page ii

3 Contents 1. Purpose Scope References General Applicable Legislation and Regulation Codes of Practice (ratified by Legislation) Codes of Practice (not ratified by Legislation) International and Australian Standards Definitions/Abbreviations General Requirements System Overview Schneider PLC Architecture Siemens PLC Architecture Equipment Selection Equipment Supply Equipment Naming Typical System Layout Devices and Connectivity Schneider PLC Architecture PLCs Siemens PLC Architecture PLCs PLC I/O Instrumentation Profinet Devices Profibus Devices Communications Protocols Computer Systems SCADA Platform Computer Configuration and Domain Access SCADA Servers SCADA Client/Engineering Workstations Inch Racks Page iii

4 8.6. Modifications to System Architecture Software Licensing Requirements Networks IP Addresses Backbone Configuration Network Switches Wireless Communication Optical Fibre Copper Ethernet Profibus Network Security Firewall Router Remote Access Network Segregation List of Tables Table 1 - Equipment Supply Responsibility... 9 Table 2 Software Licensing Requirements Table 3 Standard Network Equipment Table 4 Standard Equipment Schneider PLC Architecture List of Figures Figure 1- Document Hierarchy... 5 List of Annexes Appendix 1 - Treatment Plant PLC & SCADA Control System Typical Architecture Drawings Appendix 2 - Industrial Control Systems Standard Equipment List Page iv

5 1. Purpose has produced a set of specifications for the supply, development and implementation of SCADA and PLC systems for its Sewage Treatment Plants (STPs) The purpose of these specifications is to provide the document users with clear directions of s requirements, and to encourage standardisation The specifications have been developed to align with s Water Supply and Sewerage Reticulation Telemetry SCADA system where appropriate and with due regard for the international ASM Consortium Guidelines Effective Operator Display Design. The figure below shows the structure of this document set. Figure 1- Document Hierarchy SYSTEM ARCHITECTURE SPECIFICATION DEVICE TYPE SPECIFICATIONS (SCHNEIDER OR SIEMENS) SCADA STANDARDS SPECIFICATION HISTORIAN AND REPORTING SPECIFICATION IMPLEMENTATION SPECIFICATION 2. Scope This Specification defines the intended architecture of the control systems that is to be implemented. This includes the type of PLC, SCADA and other devices that are to be used The required network architecture is detailed at all layers, from the corporate network connection to the field devices The required security measures of the system that are to be implemented are listed. The equipment naming scheme that is required to be used is detailed and indication is provided for who will be responsible for the supply, configuration and installation of the various categories of equipment. 3. References 3.1. General Document Number Pr9833 Title SCADA and PLC Architecture (formerly UWDMDR-D-TS-0001) this document UWDMDR-D-TS-0002 Treatment Plant PLC & SCADA Specifications Page 5 of 24

6 Device Type Specification (Schneider PLC) Pr9834 Pr9846 Pr9845 Pr9844 Pr9835 Pr8843 SCADA Standard Specification (formerly UWDMDR-D-TS-0003) SCADA and PLC Specifications Historian and Reporting Specification (formerly UWDMDR-D-TS-0004) SCADA and PLC Implementation (formerly UWDMDR-D-TS-0005) SCADA Device Type (Siemens PLC) (formerly UWDMDR-D-TS-0006) Specification For Electrical Installations at Treatment Plants (formerly UWDSTD-E-TS-0001) Major Projects Specification for Drawing, Document and Equipment Tag Numbering 3.2. Applicable Legislation and Regulation Not applicable Codes of Practice (ratified by Legislation) Not applicable Codes of Practice (not ratified by Legislation) Not applicable International and Australian Standards Not applicable. 4. Definitions/Abbreviations Term ACL DMZ FOBOT ICS Integrator I/O KVM LAN Modbus Modbus TCP NIC PLC Definition Access Control Lists Demilitarised Zone Fibre Optic Break Out Tray Industrial Control System Person/entity responsible for the supply, installation and commissioning of the control system Input/Output Keyboard, Video, Mouse switch Local Area Network Serial communications protocol published by Modicon in 1979 for use with its programmable logic controllers Modbus variant used for communications over TCP/IP networks Network Interface Card Programmable Logic Controller An Industrial computer used for automation of electromechanical processes Page 6 of 24

7 Term RSTP SCADA SNMP SMS SOE Unity Pro UW VPN VSD WAN Definition Rapid Spanning Tree Protocol Supervisory Control And Data Acquisition Simple Network Management Protocol Short Message Service Standard Operating Environment Schneider Electric s Software Package for programming Unity PLCs Virtual Private Network Variable Speed Drive Wide Area Network 5. General Requirements 5.1. System Overview The system shall consist of a ClearSCADA implementation with either a Schneider UnityPro PLC architecture or a Siemens PLC architecture. The selected architecture for a plant will be detailed in the relevant project specification All sites shall have two separate Industrial Control System (ICS) specific Ethernet networks, designated as the SCADA LAN and the Site Control LAN. The SCADA LAN shall provide connectivity between the SCADA servers and SCADA Client/Engineering workstations within the site. The SCADA LAN shall also provide a connection to the SCADA WAN by a router/firewall The Site Control LAN shall provide connectivity between the SCADA Servers, Engineering Workstation and PLCs, as well as between the PLCs and Remote I/O Racks and other Modbus TCP Ethernet devices as required The Site Control LAN shall be configured as a redundant ring, and shall have no connectivity external to the treatment plant Schneider PLC Architecture Generally the Schneider PLC Architecture consists of a Schneider Modicon Quantum Hot-Standby PLC system programmed via Unity Pro software with remote I/O and other control/monitoring devices utilising Modbus TCP communications Modbus TCP devices may be connected in sub-rings where possible and otherwise shall be connected directly to a managed switch on the Site Control LAN Ethernet Ring. Each Serial Modbus device shall be connected to the Site Control LAN via its own serial to Ethernet Bridge. Devices without Modbus capabilities may be connected via inputs and outputs on motor control devices. Other devices that do not have Modbus capabilities shall be connected via Ethernet IO modules If the use of separate package plant PLCs has been specifically approved by for the project being undertaken, the package Plant PLCs shall be Unity Pro programmable processors, with data transfer between package plant PLCs and the main plant PLC via Modbus TCP on the Site Control LAN. Page 7 of 24

8 The typical system layout is depicted in the following drawings included in Appendix 2: UWDSTD-E-SD-0000 Treatment Plant PLC and SCADA Control System Architecture Overview Diagram; UWDSTD-E-SD-0001 Treatment Plant PLC and SCADA Control System Architecture Information and SCADA Layer Overview Diagram; UWDSTD-E-SD-0002 Treatment Plant PLC and SCADA Control System Architecture Control and Device Layer Overview Diagram (Schneider) The typical system layouts depicted in Appendix 2 shall be used as the basis of design for the entire industrial control system networks. All network design, including any amendments to the typical system layouts, requires approval by Siemens PLC Architecture Generally the Siemens PLC architecture consists of an S7-400H Hot-Standby PLC system programmed via Step7 software, with remote I/O and other control/monitoring devices utilising Profinet or Profibus communications The SCADA LAN shall be Ethernet, the Site Control LAN shall be Profinet and an additional Profibus DP device layer network may also be implemented All control and information collection from the field shall be performed by the main plant PLC over the Site Control LAN or via the Profibus DP device network. This includes Profinet remote I/O and Profibus I/O hardwired digital and analogue signals as well as direct connection to Profinet/Profibus enabled devices such as VSDs and Soft Starters Devices supporting Profibus PA may be connected to the Profibus DP device network via a Profibus PA/DP coupler Where a device does not have an appropriate Profinet/Profibus communications option, a Profinet/Profibus Interface or Bridge shall be used If the use of separate package plant PLCs has been specifically approved by for the project being undertaken, the connection to each package plant PLC shall be via Profinet on the Site Control LAN. The package plant PLC shall be selected from the Siemens S7-300 range. The main Plant PLC shall be the master of all communications to and from the package plant PLC, with both the reads and writes configured in the main plant PLC The typical system layout is depicted in the following drawings included in Appendix 2: UWDSTD-E-SD-0000 Treatment Plant PLC and SCADA Control System Architecture Overview Diagram; UWDSTD-E-SD-0001 Treatment Plant PLC and SCADA Control System Architecture Information and SCADA Layer Overview Diagram; UWDSTD-E-SD-0002 Treatment Plant PLC and SCADA Control System Architecture Control and Device Layer Overview Diagram (Siemens) The typical system layouts depicted in Appendix 2 shall be used as the basis of design for the entire industrial control system networks. All network design, including any amendments to the typical system layouts, requires approval by. Page 8 of 24

9 5.2. Equipment Selection Document No: Pr All supplied equipment shall be as per the Standard Equipment List, refer to Appendix 3 for details. If the supply of alternative equipment would provide cost, technical or other advantages for, the details of the proposed alternative equipment shall be provided to for consideration Equipment Supply In general, the supply, configuration and installation of all of the SCADA LAN connected equipment (excluding the configuration within ClearSCADA) is the responsibility of The supply, configuration and installation of all of the Site Control LAN connected equipment is the responsibility of the integrator The Site Computers shall be supplied and configured with their SOE by. will provide the platform for the integrator, i.e. SOE with all applications installed and base configuration, templates, libraries, etc. for the integrator to work within that structure. Refer section 8 Computer Systems for further details on computers and software The table below provides some specific examples. Equipment Type All SCADA LAN Network Equipment, including: Router Firewall Remote Access Hardware SCADA LAN Switches SCADA LAN Fibre Patch Leads Site Computers including paging modems Site Computer Software Applications and Licences SCADA LAN Computer Ethernet Patch Leads SCADA LAN Network Equipment Interconnecting Patch Leads Table 1 - Equipment Supply Responsibility Entity Responsible for Equipment Supply and Configuration Refer Table 2 Software Licensing Requirements Entity Responsible for Equipment Installation Refer Table 2 Software Licensing Requirements 19 Inch Racks for servers UPS for server & SCADA LAN 19 inch racks UPS for PLCs, other control equipment, workstation Integrator Integrator Page 9 of 24

10 Equipment Type Entity Responsible for Equipment Supply and Configuration computers and other equipment Fibre and FOBOTs Integrator Integrator Site Control LAN Networking Integrator Integrator Equipment including Ethernet Switches Patch Leads PLC Equipment Integrator Integrator Electrical Equipment Integrator Integrator 5.3. Equipment Naming Entity Responsible for Equipment Installation All equipment shall be clearly labelled with its Plant Equipment Number. Each equipment number shall conform to Pr8843 Major Projects Specification for Drawing, Document and Equipment Tag Numbering which contains examples of equipment name implementations. All equipment numbers are to be approved by prior to implementation. 6. Typical System Layout A typical high level overview diagram is shown in UWDSTD-E-SD-0000 Treatment Plant PLC and SCADA Control System Architecture Overview Diagram included in Appendix Rack mounted redundant ClearSCADA servers shall be used for the SCADA implementation Each of the redundant servers shall be connected to a dedicated 3G or PSTN modem for alarm paging via text message (SMS) The secondary SCADA server shall also host Microsoft Reporting Services, to provide required reporting functionality. Microsoft Reporting Services shall report on data from a separate SQL Server Data Warehouse database which shall reside on the secondary SCADA server The Data Warehouse database shall obtain data from the ClearSCADA database at a regular interval Each SCADA workstation shall have a full ClearSCADA ViewX Client licence and installation. There shall be a minimum of two workstations with at least one located in the main control room, each with a minimum of two monitors. There may be other SCADA workstations located around the plant as required. Workstations located outside of the main control room shall have a minimum of one monitor. One of the SCADA workstations shall double in function as an Engineering Workstation and shall have Unity Pro or Siemens Step7 and any other relevant configuration and monitoring software, such as instrument, VSD and Ethernet Switch configuration and monitoring software, installed with valid licences. Refer Section 8 Computer Systems for more details. Page 10 of 24

11 6.0.7 The Site Control LAN and Device LAN (where applicable) shall be configured as a redundant ring with no redundant paths of the ring allowed to share a physical path, e.g. being located in the same conduit Any sub-rings shall be connected in a fault-tolerant manner such that any single switch or connection failure shall not result in communications loss to any device on the sub-ring. A diagram of typical device connectivity on the site control LAN is shown in UWDSTD-E-SD-0002 Treatment Plant PLC and SCADA Control System Architecture Control and Device Layer Overview Diagram and UWDSTD-E-SD Treatment Plant PLC and SCADA Control System Architecture Control and Device Layer Overview Diagram (Siemens PLC). Additionally, failure or powering down of multiple devices on the ring shall not affect communications to any other device All SCADA computers and associated devices such as printers shall be connected to the SCADA LAN. Devices that are required to access both LANs, including the SCADA servers and Engineering Workstation, shall have two Network Interface Cards (NICs), one connected to each of the LANs. Each NIC shall have addresses in completely separate subnets and shall not be bridged The SCADA LAN shall be connected to the SCADA Domain and the Corporate Domain via a WAN link. shall supply and configure the firewall/router/vpn hardware associated with this link. Any remote access to the system shall be via this link and controlled by. All computers shall be part of the SCADA domain and shall be configured by with s Standard Operating Environment (SOE) Any access from the Corporate Domain shall be limited by a firewall that shall be configured by. This will only allow Web Reports and ClearSCADA WebX to be accessed from the UW Corporate Domain. All other traffic to and from the UW Corporate Domain shall be blocked A Diagram of typical connections to the site SCADA LAN is shown in UWDSTD-E-SD-0001 Treatment Plant PLC and SCADA Control System Architecture Information and SCADA Layer Overview Diagram included in Appendix Devices and Connectivity 7.1. Schneider PLC Architecture PLCs The main plant PLC shall be a Schneider Unity Quantum Hot Standby system. The Primary and Standby PLC racks shall be physically separated in separate buildings where practicable At a minimum the PLC racks shall be separated into separate physical compartments of the switchboard or MCC. The Primary and Standby PLC shall have direct interconnections as required by the Quantum Hot Standby system, with the fibre optic network designed accordingly. Page 11 of 24

12 All control and monitoring shall be performed by the main plant PLC over an Ethernet network using Modbus TCP. This includes Ethernet I/O for hardwired digital and analog signals and directly to Modbus TCP enabled devices such as VSDs, Soft Starters and motor protection relays. Where a device does not have an appropriate Modbus TCP communications option, an Ethernet to Serial Modbus Bridge shall be used with one bridge per device Each Hot Standby processor rack shall have at least two Ethernet Network Modules, both connected to the Site Control LAN. The first Ethernet Network Module shall be used exclusively for SCADA and Unity Pro traffic. The second Ethernet Network Module shall be used exclusively for connection to Modbus TCP devices (I/O Scanning). The Hot Standby system will handle the automatic changeover of IP addresses in the event of a processor primary-standby changeover The power supplies of the two Hot Standby processors shall be supplied independently from each other. As a minimum, each processor shall be supplied from an individual 240Vac UPS supplied circuit breaker and shall conform to the requirements in Pr9835 Specification For Electrical Installations at Treatment Plants (Design and Installation) If the use of separate package plant PLCs has been specifically approved by for the project being undertaken, the package Plant PLCs shall be Unity Pro programmable processors. They may be either Quantum PLCs or M340 PLCs. Data transfer between package plant PLCs and the main plant PLC shall be via Modbus TCP on the Site Control LAN The main plant PLC shall be the master of all communications to package plant PLCs and will control both reads from and writes to the package plant PLC. The main plant PLC shall write a watchdog so that the package plant PLC shall be aware of a loss of communications with the main plant PLC. The SCADA shall communicate directly with the package plant PLC for monitoring and control where practicable PLC I/O All I/O attached to the main plant PLCs shall be Modbus TCP Ethernet remote I/O utilising Modicon Advantys STB modules. A loss of communication between the Ethernet I/O and PLC for a defined time shall result in outputs failing to a pre-defined safe state. A changeover from primary to standby PLC or change in status of Ethernet redundant ring shall not result in any process interruption. Thus, the configured communications timeout in all I/O shall be long enough to allow bumpless transfer of all redundant systems, and short enough to ensure safe plant shutdown in the event of a complete communications loss Ethernet I/O shall be connected such that the loss of any single node or multiple nodes does not affect the communications to other nodes. Ethernet connections shall not be daisy-chained through multiple Ethernet I/O nodes with integral switches unless the connections are able to form a redundant ring Each redundant ring of Ethernet I/O devices in a location shall have a maximum of six devices in the ring. Where Ethernet I/O devices are not connected in a redundant ring, they are to be spread across the two Ethernet switches in that location as far as practicable Due consideration shall be made to allocating I/O in a fault tolerant manner. For example, the I/O for multiple identical items of equipment (such as two pumps in a well) shall be split across multiple Remote I/O racks or failing that multiple cards Page 12 of 24

13 within a single rack as applicable. The I/O for directly related equipment (such as a flowmeter controlling a valve) shall be located within a single rack as far as practicable Instrumentation All analog instrumentation shall be connected to Ethernet remote I/O via a 4-20mA current loop connection. Any secondary values required, including flow pulses, shall be connected via Ethernet remote I/O Any relevant instrumentation configuration software shall be provided on the site engineering workstation. Complete up to date parameter lists shall be shall be stored on the site engineering workstation Modbus Devices Where a device has a Modbus communications option, this shall be utilised for monitoring and control. Serial Modbus and Modbus TCP connections are acceptable with a preference for Modbus TCP where available. Serial Modbus devices must be connected via a bridge as detailed in section Ethernet to Serial Bridges The Modbus devices shall be connected such that any single device or connection failure shall not result in loss of communications to any other devices. Thus, if Modbus devices have an integral Ethernet switch, devices shall only daisy-chained together if the chain forms part of a redundant ring Each redundant ring of devices in a location shall have a maximum of six devices in the ring Where Modbus devices are not connected in a redundant ring, they are to be spread across the two Ethernet switches in that location as far as practicable Additionally, devices shall be powered such that if two devices are powered off, other devices on the ring between the two powered down cells do not lose communications Where a device has integral inputs and outputs these may be used for hard-wired connections directly related to the device. An example of this is an Auto/Manual/Off selector switch connected directly to digital inputs integral to a VSD Any relevant device configuration software shall be provided on the site engineering workstation. Complete up to date parameter lists for all devices shall be shall be stored on the site engineering workstation Ethernet to Serial Bridges Ethernet to Serial Bridges shall be used for connections to serial Modbus devices. A separate dedicated Ethernet to Serial Bridge is to be used for each serial Modbus device. There shall be no multi-drop serial Modbus networks Communications Protocols Communications shall be via native Ethernet connections where available. For devices that do not provide appropriate Ethernet communications options, communications shall be via one of the other communications protocols below. Use of any other communications protocols must be authorised by The preferred standard communication protocols in order of preference include: Modbus TCP; Page 13 of 24

14 Modbus RTU (via Ethernet to Serial Bridge); SNMP. o RS485; o RS232; 7.2. Siemens PLC Architecture PLCs The main plant PLC shall be a Siemens S7-400H Hot-Standby system. One extra interface/extension Profinet module is to be included with each PLC. The expansion module Profinet port shall be used for data transfer between PLC and SCADA and for PLC programming The CPU Profinet port shall be used for data transfer between PLC and field devices. In case there is a device which cannot be connected to Profinet network, Profibus ports can be used for data transfer between PLC and field devices If the use of separate package plant PLCs has been specifically approved by for the project being undertaken, the package plant PLC(s) shall be Siemens S7 300 PLCs. Data transfer between package plant PLC(s) and the main plant PLC shall be via Profinet network The main plant PLC shall be the master of all communications to package plant PLC(s) and will control both reads from and writes to the package plant PLC. The main plant PLC shall write a watchdog so that the package plant PLC(s) shall be aware of a loss of communications with the main plant PLC The power supplies of the main plant and package plant PLCs shall be supplied independently from each other. As a minimum, each processor shall be supplied from an individual 240Vac UPS supplied circuit breaker and shall conform to the requirements in Pr9835 Specification For Electrical Installation at Treatment Plants (Design and Installation) PLC I/O All I/O attached to the main plant PLC shall be Profinet or Profibus DP or Modbus/TCP connected Remote I/O. A loss of communication between the Remote I/O and PLC for a defined time shall result in outputs failing to a pre-defined safe state Remote I/O shall be connected such that the loss of any single node or multiple notes does not affect the communications to other nodes. Remote I/O connections shall not be daisy-chained through multiple Remote I/O nodes with integral switches unless the connections are able to form a redundant ring Instrumentation All analogue instrumentation shall be connected to Profinet remote I/O via a 4-20mA current loop connection. Any secondary values required, including flow pulses, shall be connected via Profinet remote I/O. In case connection to Profinet I/O is not possible, Profibus I/O should be used instead. Page 14 of 24

15 Any relevant instrumentation configuration software shall be provided on the site engineering workstation. Complete up to date parameter lists shall be stored on the site engineering workstation Profinet Devices Where a device has a Profinet communications option, this shall be utilised for monitoring and control. The Profinet devices shall be connected such that any single device or connection failure shall not result in loss of communications to any other devices. Thus, if Profinet devices have an integral Ethernet switch, devices shall only be daisy-chained together if the chain forms part of a redundant ring Additionally, devices shall be powered such that if two devices are powered off, other devices on the ring between the two powered down cells do not lose communications Where a device has integral inputs and outputs these may be used for hard-wired connections directly related to the device. An example of this is a Local/Remote selector switch connected directly to digital inputs integral to a VSD Any relevant device configuration software shall be provided on the site engineering workstation. Complete up to date parameter lists for all devices shall be shall be stored on the site engineering workstation Profibus Devices Where a device does not have Profinet communications option and has Profibus communications option instead, this shall be utilised for monitoring and control The Profibus devices shall be connected such that any single device or multiple device or connection failure shall not result in loss of communications to any other devices Profibus devices shall be powered such that if two devices are powered off, other devices on the ring between the two powered down cells do not lose communications Where a device has integral inputs and outputs these may be used for hard-wired connections directly related to the device. An example of this is a Local/Remote selector switch connected directly to digital inputs integral to a VSD Any relevant device configuration software shall be provided on the site engineering workstation. Complete up to date parameter lists for all devices shall be shall be stored on the site engineering workstation Communications Protocols Communications shall be via native Profinet connections where available. For devices that do not provide appropriate Profinet Ethernet communications options, communications shall be via Profibus DP or may also use Modbus TCP. Use of any other communications protocols must be authorised by. Page 15 of 24

16 8. Computer Systems 8.1. SCADA Platform Document No: Pr The SCADA system for each site shall have two ClearSCADA servers configured as a Hot-Standby Pair. Each ClearSCADA server shall run on its own physical server. The version of ClearSCADA to be used shall be specified by Microsoft SQL Server and Microsoft Reporting Services shall be installed on the Secondary SCADA Server. This SQL Server shall act as a Data Warehouse database that will extract, process and store data from the ClearSCADA database on a periodic basis. Microsoft Reporting Services will be used to create reports from this data stored in the Data Warehouse Database. Reports are able to be viewed on any site PC or WAN connected PC via a web interface Refer to Appendix UWDSTD-E-SD-0003 Treatment Plant PLC and SCADA Control System Architecture - SCADA Information Transfer Block Diagram for a block diagram of the ClearSCADA system communication links Computer Configuration and Domain Access All SCADA related PCs shall be part of s SCADA domain, which is centrally controlled and administered by All SCADA PC logins shall be via the SCADA Domain. Each user requiring access to a SCADA computer is required to complete and submit a SCADA User Request Form. Upon successful processing of this form, the user will then be provided with a username and password for use on the SCADA domain. Sharing of a person s login details is strictly forbidden as per s IT Access Policy Each computer shall be supplied by and configured by with the SCADA Standard Operating Environment (SOE). The SOE includes the Windows operating system, anti-virus software, remote control software and other configuration software to function on the SCADA domain The integrator shall then be responsible for the further configuration of any other required applications and installation of the computers, in consultation with The SCADA LAN and Control LAN shall not be connected to the Internet or any other external network at any time SCADA Servers The SCADA servers shall act as servers of ClearSCADA and the secondary SCADA server shall also function as a DataWarehouse and reporting server ClearSCADA software shall be installed on each server by, and configured with s standard template and object library, ready for configuration by the Integrator to suit the project requirements On the secondary SCADA server, an SQL database server and Microsoft Reporting Services shall also be installed by as per the requirements of the Historian and Reporting Specification. Page 16 of 24

17 8.3.4 Backup facilities shall be provided by to enable manual and automatic backup of all ClearSCADA configuration and historical data, in addition to full system backups The SCADA servers shall be connected to both the SCADA LAN and the Site Control LAN. The servers shall each be connected to a different managed switch on the Site Control LAN with each managed switch being part of a redundant ring Each SCADA Sever shall have a ClearSCADA server licence installed of the required point count and features including the specified number of WebX licences Each SCADA server shall be connected to a 3G and/or PSTN modem for alarm paging via an SMS service. Alarms selected by to be paged out shall be configured with Critical severity within ClearSCADA, with ClearSCADA configured to redirect all critical alarms via text message to nominated mobile numbers The SCADA servers shall be located in 19 inch racks with requirements detailed in section Inch Racks The power supply to the servers shall be configured such that a single UPS failure shall not result in a loss of power supply to the both the primary and secondary server as detailed in section Inch Racks SCADA Client/Engineering Workstations Dedicated SCADA workstations shall be connected to the SCADA LAN only. Each SCADA workstation shall have ClearSCADA ViewX installed by and shall have a valid ClearSCADA ViewX licence installed There shall be at least two SCADA workstations in the plant. If required, there shall be additional SCADA workstations throughout the plant to allow efficient plant monitoring and operation. All SCADA workstations in the main control room shall have at least two monitors. Other workstations shall have at least one monitor Note that SCADA servers are not to be used as display clients with access to server racks physically restricted There shall be one SCADA workstation in an appropriate location that will be assigned as an Engineering Workstation. This machine shall additionally have all required programming and configuration software installed with valid licences for this software. This includes the server configuration and monitoring components of ClearSCADA and Unity Pro or Siemens Step7 PLC programming software, plus any other relevant configuration and monitoring software, such as instrument, VSD and Ethernet Switch configuration and monitoring software. The version of PLC programming software to be used shall be specified by ClearSCADA and the PLC programming software shall be installed by complete with licences. For any additional software required the Integrator shall liaise with to arrange installation and supply to any required licences Workstations and their associated peripherals, including monitors, shall be supplied from a UPS. Page 17 of 24

18 Inch Racks A 19 inch rack shall be installed in a suitable location within a building, such as the main switchroom, with the rack housing servers and communications equipment, including SCADA LAN network switches and remote access and security hardware. As far as practicable, an additional 19 inch rack shall be installed in another suitable location to house the secondary SCADA server. For both the primary and secondary SCADA servers to be located in the same 19inch rack, explicit approval is required from Any 19 inch racks housing servers shall be full height and full depth. If there is more than one server in a 19 inch rack, control of all servers in the rack shall be by a single monitor, mouse, and keyboard combination controlled by a KVM switch. There shall be a minimum of 2 spare network ports on each LAN at each 19 inch rack Corporate network equipment shall not be installed within the ICS equipment racks, nor shall it be powered by an ICS equipment rack UPS Refer to UWDSTD-E-SD-0004 Treatment Plant PLC and SCADA Control System Architecture - 19 inch Rack Power Supply Block Diagram for a block diagram of 19 inch rack power supply requirements Each 19 inch rack housing a server shall have a rack-mounted UPS which supplies only the equipment within the rack. All UPS s shall be sized to provide a minimum of 30 minutes of backup to all connected loads. All UPS s shall be monitored via SNMP by the ClearSCADA system as per Pr9834 SCADA and PLC Standard (formerly UWDMDR-D-TS-0003) Each full depth 19 inch rack shall have at least two power supply rails with standard Australian three pin outlets. Power outlets shall be provided so that all of the power rails can be supplied from UPS power or mains power easily At sites where the primary and secondary SCADA Servers are located in a single rack, two UPS s shall be provided in that rack. Each server will have one of its redundant power supplies connected to the first UPS and the other of its redundant power supplies connected to the second UPS SCADA LAN networking equipment, KVMs, monitors and all other equipment located in 19 inch racks shall be supplied by UPS A separate UPS sized to provide a minimum of 30 minutes of backup to all connected loads shall be provided at each location (switchroom, control room, external switchboard, etc.) as required to supply the PLCs, SCADA Workstations and DC power supplies (if not battery backed) supplying Remote I/O and network equipment Modifications to System Architecture Any modifications to the architecture of the system shall be approved by Software Licensing Requirements Table 2 Software Licensing Requirements Licence Licence Type Host Machine Quantity Supplied By Page 18 of 24

19 Licence Licence Type Host Machine Quantity Supplied By ClearSCADA ViewX Client ClearSCADA Server ClearSCADA WebX Client USB Dongle USB Dongle Floating (on Server USB dongles) Site machines running ViewX ClearSCADA Server ClearSCADA Server UnityPro Software Engineering Workstation Siemens Step7 Any other device configuration software Microsoft SQL Server Licence Microsoft SQL CAL Microsoft SQL CAL Microsoft Windows Server Microsoft Windows Software Software or USB Dongle Engineering Workstation Engineering Workstation Minimum 3 (2 Control Room SCADA Workstations and 1 Operator Laptop) 2 (1 Primary, 1 Secondary) 4 (2 primary, 2 Secondary) Refer note below table Integrator Server Data Warehouse and Reporting Server (ClearSCADA secondary server) 1 Device CAL N/A 1 per site PC User CAL N/A 1 per WAN report user ClearSCADA Minimum 2 servers All Site PCs 1 per site PC ClearSCADA Server Licences Note: shall supply a USB licence for each ClearSCADA server with the following details, Integrator to advise if point count will not be sufficient: Points: 25,000; Features: Main/Standby; ViewX: 0; WebX: 2. Page 19 of 24

20 9. Networks 9.1. IP Addresses Each device on the SCADA LAN shall have an IP address specified and configured by. No devices are to be connected to the SCADA LAN without having an IP address assigned by For the Site Control LAN and Device LAN if applicable the subnet ranges, subnet masks and default gateway settings shall be specified by for the Integrator to assign IP addresses for all devices. The assigning of the IP addresses must follow a logical documented structure The Integrator shall submit a request to for the Site Control LAN and Device LAN subnet information Backbone Configuration All Managed Ethernet switches on the Site Control LAN shall be connected in a redundant ring configuration and configured to use RSTP to allow ring connectivity. The backbone connection between switches shall be a fibre connection Any sub-rings shall have a fault tolerant connection to the main ring. Any single switch failure shall not result in communications loss between any other managed switches For each redundant ring, none of its redundant paths shall share a conduit. If all cables within a single conduit are severed, all managed switches shall still be able to communicate The speed of all ring backbone connections shall be at least 1GBit/s Network Switches Each Ethernet switch shall have at least 2 spare ports for future expansion Any configuration and management software for managed switches supplied by the integrator shall be installed and configured on the Engineering workstation All network switches shall be as per the Standard Equipment List Network monitoring shall be performed via SNMP V2 to Site Control LAN managed switches by the ClearSCADA system as per Pr9834 SCADA and PLC Standard Wireless Communication In general, Wireless Ethernet devices are not permitted to be connected to any network. Specific approval from is required before connecting any wireless devices Optical Fibre All Optical Fibre shall be installed as per the requirements of Pr9835 Specification for Electrical Installation at Treatment Plants (Design and Installation) Fibre types and fibre connectors shall be standardised across a single site for new fibre installations utilising SC type connectors. Page 20 of 24

21 9.6. Copper Ethernet All copper Ethernet cabling shall be installed as per the requirements of Pr9835 Specification for Electrical Installation at Treatment Plants (Design and Installation) Profibus Network Overview All of the field devices including Variable Speed Drives (VSDs), Direct Online Starters (DOLs), Soft Starters, Motor Protection Relays, valves, instruments, etc. that do not support Ethernet (Profinet) shall communicate with the PLC/s via Profibus DP Process data is transmitted using a cyclic master-slave method while alarms, configuration parameters or diagnostic data are transmitted using acyclic transmission The number of local Profibus networks depends on the number of CPU s DP ports. Each of CPU s Profibus DP ports (MPI/DP or DP) can be used to setup one or two local Profibus networks The Profibus DP network should be implemented with master-slave topology with mono-master system, i.e. only one master exists for cyclic data exchange with slave devices. Network Segments Depending on the plant size, the plant may be divided into multiple areas with each area connected to a separate Profibus network. In theory, each network segment can have up to 32 nodes and a maximum of 126 bus nodes (all masters and slaves together) can be connected to a single Profibus Network. In practice, however, the number of nodes connected is often smaller than this and depends on various factors such as the transmission rate of the line and the data quantities of the individual slaves. Cabling For Profibus DP networks, either copper or Fibre-Optic shall be used as transmission medium. Fibre Optic cables are required for links between buildings and links between physically separate process areas. Only manufacturer specified connectors shall be used All Profibus cabling shall be installed as per the requirements of Pr9835 Specification for Electrical Installation at Treatment Plants (Design and Installation). Redundancy using Optical Ring A ring topology is required for fault tolerance and requires Optical Link Modules (OLMs) to be used. 10. Security Firewall Each site shall have a hardware firewall that all external incoming or outgoing network traffic must pass through. The Firewall shall be supplied and configured by. Page 21 of 24

22 The firewall shall be located in one of the 19 inch server racks. The hardware firewall may be combined in the same unit as the Router Router Each site shall have a hardware router that all external incoming or outgoing network traffic must pass through. The router shall be supplied and configured by The router shall be located in one of the 19 inch server racks. The router may be combined in the same unit as the Firewall Remote Access s SCADA Domain features VPN connectivity and other security features to facilitate remote access from SCADA laptops to the domain Computers that are part of the SCADA Domain shall be able to access all devices on the SCADA LAN, whether the computer is connected to the local LAN or via the WAN. Computers on the Corporate Domain shall be limited to accessing WebX on the Primary and Secondary SCADA Servers and web reports on the Secondary SCADA Server. This access limitation shall be handled by the firewall configured by The Site Control LAN has no direct connection to any SCADA LAN networking equipment and thus cannot be accessed directly via the WAN. The only remote access to the Site Control LAN is via Remote Desktop Control of one of the site computers that is connected to the Site Control LAN No other remote connections shall be allowed other than the WAN Link Integrators shall apply for remote access to the SCADA Domain which will allow remote connection to the required devices. Each user requiring access to the SCADA Domain is required to complete and submit a SCADA User Request Form. Upon successful processing of this form, the user will then be provided with a username and password for use on the SCADA domain. Sharing of a person s login details is strictly forbidden as per s ICT Access Management Policy (OP8106) Network Segregation The SCADA LAN and Site Control LAN shall be kept separate with no interconnections. Where devices are required to access both the SCADA LAN and Site Control LAN they shall have a separate Network Interface Card (NIC) installed and configured for each LAN connection, with the connections not to be bridged. This includes the SCADA Servers and Engineering Workstation. Page 22 of 24

23 Appendix 1 - Treatment Plant PLC & SCADA Control System Typical Architecture Drawings The following drawings have been developed to indicate the Typical Architecture of a STP: a. UWDSTD-E-SD-0000 Treatment Plant PLC and SCADA Control System Architecture Overview Diagram b. UWDSTD-E-SD-0001 Treatment Plant PLC and SCADA Control System Architecture Information and SCADA Layer Overview Diagram c. UWDSTD-E-SD-0002 Treatment Plant PLC and SCADA Control System Architecture Control and Device Layer Overview Diagram (Schneider PLC) d. UWDSTD-E-SD-0003 Treatment Plant PLC and SCADA Control System Architecture - SCADA Information Transfer Block Diagram e. UWDSTD-E-SD-0004 Treatment Plant PLC and SCADA Control System Architecture - 19 inch Rack Power Supply Block Diagram f. UWDSTD-E-SD-0005 Treatment Plant PLC and SCADA Control System Architecture Control and Device Layer Overview Diagram (Siemens PLC). Page 23 of 24

24 Appendix 2 - Industrial Control Systems Standard Equipment List Table 3 Standard Network Equipment Item Ethernet Switch Managed Ethernet Patch Leads Details Moxa EDS-510A-1GT2SFP plus Gigabit optical SFP modules as required, e.g. SFP-1GSXLC Minimum Cat6 STP inside switchboards or where recommended by device manufacturer, otherwise UTP - SCADA LAN Yellow - Site Control LAN Green - Cross-over Red Table 4 Standard Equipment Schneider PLC Architecture Type Part Number Description Quantum Hot Standby Power Supply 140 CPS Power Supply CPU 140 CPU Hot Standby Processor kB Memory Card TSX MRP C007M Expansion Memory Card 7MB SRAM (where needed) RIO Head 140 CRP Remote IO Head Module - Ethernet Ethernet Comms 140 NOE Ethernet Communications Module Remote I/O - Advantys STB Power Supply STBPDT3100 Standard power distribution - 24Vdc Power Supply STBCPS2111 Auxiliary power supply NIM D/I D/O A/I A/O STBNIP2311 STBDDI3725 STBDDO3705 STBACI1400 STBACO0220 Standard Network Interface Module - Ethernet Modbus TCP/IP Basic digital input module STB - 24Vdc - 16 I Basic digital output module STB - 24Vdc - 16 O Standard analog input kit STB ma - 8 I Standard analog output kit STB ma - 2 O Page 24 of 24