T28 - Design Considerations for Robust EtherNet/IP Networking

Transcription

1 PUBLIC CO900H Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. T28 - Design Considerations for Robust EtherNet/IP Networking PUBLIC

2 PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 2 Abstract Learn about top design considerations that are developed by Rockwell Automation and our partners to help you design and deploy a more scalable, robust, secure and future-ready EtherNet/IP network infrastructure. A prior understanding of general Ethernet concepts is recommended.

3 Why Is This Important? Design Considerations for Robust EtherNet/IP Networking PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 3 Scalable, robust, secure and futureready infrastructure/architecture: Application Internet of Things, Internet of Everything Software Network

4 PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 4 Industrial Network Design Methodology Design Considerations for Robust EtherNet/IP Networking Industrial Network Design Methodology Single Industrial Network Technology Sizing and Selection of EtherNet/IP Devices Physical Infrastructure Structure, Hierarchy and Segmentation Broadcast Domains IP Addressing Network Availability Convergence-Ready Network Solutions

5 PUBLIC Industrial Network Design Methodology Understand application and functional requirements Devices to be connected industrial and non-industrial Data requirements for availability, integrity and confidentiality Communication patterns, topology and resiliency requirements Types of traffic information, control, safety, time synchronization, drive control, voice, video Develop a logical framework (roadmap) Avoiding Enabling OEM Convergence-Ready Because Network Network Sprawl Migrate from flat networks to structured and hardened networks Define zones and segmentation, place applications Solutions and devices in the logical framework that is based on requirements Infrastructure Matters Develop a physical framework to align with and support the logical framework Deploy a Holistic Defense-in-Depth Security Model Reduce risk, simplify design, and speed deployment: Use information technology (IT) standards Follow industrial automation technology (IAT) standards Use reference models and reference architectures AUDIT MANAGE / MONITOR ASSESS DESIGN/PLAN IMPLEMENT Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 5

6 Single Industrial Network Technology OSI 7-Layer Reference Model PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 6 Open Systems Interconnection Layer Name Layer No. Function Examples Application Presentation Layer 7 Layer 6 Network Services to User App Encryption/Other processing What makes EtherNet/IP industrial? CIP IEC Session Layer 5 Manage Multiple Applications Transport Network Routers Layer 4 Layer 3 Reliable End-to-End Delivery Error Correction Packet Delivery, Routing IETF TCP/UDP IETF IP Data Link Physical Switches Cabling/RF Layer 2 Layer 1 Framing of Data, Error Checking Signal type to transmit bits, pinouts, cable type IEEE 802.3/ TIA Physical Layer Hardening Infrastructure Device Hardening Common Application Layer Protocol

7 PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 7 Sizing and Selection of Devices Theoretical EtherNet/IP Performance Example Controller exchanges 36 bytes of I/O data with 10 I/O Adapters with a 1 ms Requested Packet Interval (RPI) RPI = 1 ms 1,000 frames/second in each direction Each I/O Adapter must be able to: Consume 1,000 frames/second Produce 1,000 frames/second The Controller must be able to: Consume 10,000 frames/second Produce 10,000 frames/second Design considerations Size the Controller Maximum # of Adapters (CIP Connections) Minimum RPI (how fast) Maximum I/O Data Size per RPI Size the Adapters Minimum RPI (how fast) Maximum I/O Data Size per RPI Physical Environment for example, EMI Interference for Copper Media Speed / Duplex (potential mismatch) Network Infrastructure Latency and Jitter This represents about 10% of the total network bandwidth

8 PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 8 Physical Infrastructure Design and implement a robust physical layer Environment Classification - MICE More than cable Connectors Patch panels Cable management Noise mitigation Grounding, Bonding and Shielding Standard Physical Media Wired vs. Wireless Copper vs. Fiber UTP vs. STP Single-mode vs. Multi-mode SFP LC vs. SC Standard Topology Choices Switch-Level and Device-Level ODVA Guide Fiber Guide ENET-TD003 Cable Selection ENET-WP007

9 PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 9 Physical Infrastructure Environmental Focus M.I.C.E. Increased Environmental Severity M.I.C.E. provides a method of categorizing the environmental classes for each plant Cell/Area Zone. The MICE environmental classification is a measure of product robustness: Specified in ISO/IEC Part of TIA-1005 and ANSI/TIA-568-C.0 standards This provides for determination of the level of hardening required for the network media, connectors, pathways, devices and enclosures. Examples of rating: TIA 1005 Office Industrial 1585 Industrial Ethernet Media: M 3 I 3 C 3 E 3 M12: M 3 I 3 C 3 E 3 RJ45: M 1 I 1 C 2 E 2

10 Physical Infrastructure Select best media for your needs UTP vs. STP CAT5e vs. CAT6a Copper vs. Fiber Multi-mode vs. Single-mode Fiber Unshielded Twisted Pair (UTP) Costs less Installs faster Shielded Twisted Pair (STP) Excellent immunity from EMI and RFI noise Can locate cable close to source of noise Smaller diameter, more flexible Well suited for more rigorous environments CAT5e CAT6a Costs Less Higher signal to noise ration; performance margins Suitable for speeds of less than a Gbps Designed to deliver Gbps performance Copper Fiber Termination and installation are faster Cost of fiber transceivers is higher Less fragile Use when excessive EMI noise is present Distances of less than 100 m Use when distance is a factor (over 100 m) Multi-mode Single-mode For distances of up to Gbps and Mbps Longer distances (up to 40 km) Lower-cost transceivers, connectors and installation High-bandwidth capabilities Higher fiber cost, but lower total system cost Lower fiber cost, but higher total system cost PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 10

11 PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 11 Structure, Hierarchy and Segmentation Smaller modular building blocks to help minimize network sprawl and build scalable, robust and future-ready network infrastructure Smaller broadcast domains (restrict broadcast traffic) Smaller fault domains (for example, Layer 2 loops) Smaller domains of trust (security) Multiple techniques to create smaller network building blocks (Layer 2 domains) Structure and hierarchy Logical model geographical and functional organization of IACS devices Campus network model - multi-tier switch model Layer 2 and Layer 3 Logical framework Segmentation Multiple network interface cards (NICs) for example, CIP bridge Network Address Translation (NAT) appliance Virtual Local Area Networks (VLANs) VLANs with NAT Integrated Services Router

12 Structure, Hierarchy and Segmentation Layer 2 Collision Domains PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 12 Switch L2-1 L /24 Switch L3-1 L /24 Switch L2-2 L2 - VLAN 10 L2 - VLAN 20 Fa1/1 Gi1/1 Gi1/1 Gi1/2 Fa1/1 Fa1/2 Controller 1 Fa1/2 L Controller 3 L L2 - E B44 L L2 - E B41 L L2-0000:BC10:1005 L2-0000:BC10:2005 Controller 2 L L2-001D:9C10:1006

13 Structure, Hierarchy and Segmentation Layer 2 Broadcast Domains - Switch Hierarchy PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 13 Switch L2-1 L /24 Switch L3-1 L /24 Switch L2-2 L2 - VLAN 10 L2 - VLAN 20 Fa1/1 Gi1/1 Gi1/1 Gi1/2 Fa1/1 Fa1/2 Controller 1 Fa1/2 L Controller 3 L L2 - E B44 L L2 - E B41 L L2-0000:BC10:1005 L2-0000:BC10:2005 Controller 2 L L2-001D:9C10:1006

14 PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 14 Structure, Hierarchy and Segmentation Flat and Open IACS Network Infrastructure Flat and Open Industrial Automation and Control System Structured and Hardened Network Infrastructure IACS Network Infrastructure

15 Structure, Hierarchy and Segmentation Multiple Network Interface Cards (NICs) - CIP Bridge Isolated networks - two NICs for physical network segmentation Plant Network Level 3 Control Network Benefits Clear network ownership demarcation line Challenges Limited visibility to control network devices for asset management Limited future-ready capability Smaller PACs may not support Layer 2 Network Layer 2 Network Converged networks logical segmentation Shared Layer 2 Network Plant Network Level 3 Converged Network Benefits Plant-wide information sharing for data collection and asset management Future-ready Challenges Blurred network ownership demarcation line PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 15 VLAN 102 Control Network

16 Structure, Hierarchy and Segmentation Virtual LANs (VLANs) PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 16 Plant-wide IACS VLAN 40 IP Subnet /24 Plant-wide IACS VLAN 40 IP Subnet /24 Plant-wide IACS 1734 Point I/O Stratix 8300 Ring OWS 1732E Slim ArmorBlock I/O VLAN 10 Plant-wide IACS 1734 Point I/O Stratix 8300 Stratix 8000 Layer 3 Ring OWS Stratix 5700 VLAN E Slim ArmorBlock I/O Machine #1 OEM #1 Stratix 8000 ControlLogix 1756-EN2T Layer 2 Stratix 5700 Large Layer 2 Broadcast Domain CompactLogix 5370 L3 Machine #2 OEM #2 Machine #1 OEM #1 Machine #1 (OEM #1) VLAN 20 IP Subnet /24 VLAN 10 IP Subnet /24 ControlLogix 1756-EN2T VLAN 20 CompactLogix 5370 L3 VLAN 30 Smaller Layer 2 Broadcast Domains Machine #2 OEM #2 Machine #2 (OEM #2) VLAN 30 IP Subnet /24 VLAN 5 IP Subnet /24

17 PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 17 Structure, Hierarchy and Segmentation Network Address Translation Network Address Translation (NAT)is a service that can translate a Source IP address to another IP address within a packet Can be a Layer 2 or Layer 3 device Has two forms: One to One (1:1) Allows for the assignment of a unique outside IP address to a specific inside IP address One to Many (1:n) a.k.a. TCP/UDP Port Address Translations (PAT). Allows Multiple devices to share one Outside address Outside Subnet (ex x) Many Outside IP addresses (One per device wishing to be accessible from the Outside Subnet NAT Enabled Device Many Inside IP addresses (One per connected device) Inside Subnet (ex x) Inside

18 PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 18 Structure, Hierarchy and Segmentation Network Address Translation Line Controller HMI.11 I/O Skid / Machine #1.14 Inside VLAN /24 Outside VLAN /24.13 Controller IES-1 VFD.12 IES-2 IES-4 Stratix 5700 w/nat IES-3 Cell/Area Zone - VFD Controller Industrial Zone Levels 0-3 (Plant-wide Network) I/O HMI.11 Skid / Machine #2 Inside VLAN /24 Multiple Skids/Machines Each Skid/Machine Aggregated by One Stratix 5700 Layer 2 NAT Switch Single VLAN Architecture Inside to Outside NAT Table Outside to inside NAT Table Inside to Outside NAT Table Outside to inside NAT Table IES-3 Stratix 5700 w/ NAT Inside Inside Outside Outside Inside IES-2 Stratix 5700 w/ NAT Outside Outside Inside

19 Structure, Hierarchy and Segmentation No Segmentation (not recommended) PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 19 Enterprise-wide Business Systems Levels 4 & 5 Data Center Enterprise Zone Level IDMZ Plant LAN VLAN17 - Layer 2 Domain Plant IP - Subnet /24, every device requires a unique IP address Plant-wide Site-wide Operation Systems Industrial Zone Level 3 - Site Operations Levels 0-3 Physical or Virtualized Servers (Plantwide Network) Application Servers and Services Platform Network Services for example, DNS, AD, DHCP, AAA Remote Access Server (RAS) Storage Array Cell/Area Zone #1 Subnet /24 Cell/Area Zone #2 Subnet /24 Cell/Area Zone #3 Subnet /24

20 Structure, Hierarchy and Segmentation Multiple NIC Segmentation Enterprise-wide Business Systems Levels 4 & 5 Data Center Enterprise Zone Level IDMZ Plant LAN VLAN17 - Layer 2 Domain Plant IP - Subnet /24 Plant-wide Site-wide Operation Systems Industrial Zone Level 3 - Site Operations Levels 0-3 Physical or Virtualized Servers (Plant-wide Network) Application Servers and Services Platform Network Services for example, DNS, AD, DHCP, AAA Remote Access Server (RAS) Storage Array Line/Area Controller Cell/Area Zone #1 Subnet /24 Cell/Area Zone #2 Subnet /24 Cell/Area Zone #3 Subnet /24 PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 20

21 Structure, Hierarchy and Segmentation NAT Appliance Segmentation Enterprise-wide Business Systems Levels 4 & 5 Data Center Enterprise Zone Level IDMZ Plant LAN VLAN17 - Layer 2 Domain Plant IP - Subnet /24 Plant-wide Site-wide Operation Systems Industrial Zone Level 3 - Site Operations Levels 0-3 Physical or Virtualized Servers (Plant-wide Network) Application Servers and Services Platform Network Services for example, DNS, AD, DHCP, AAA Remote Access Server (RAS) Storage Array Cell/Area Zone #1 Subnet /24 Cell/Area Zone #2 Subnet /24 Cell/Area Zone #3 Subnet /24 PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 21

22 Structure, Hierarchy and Segmentation Integrated Services Router Segmentation Enterprise-wide Business Systems Levels 4 & 5 Data Center Enterprise Zone Level IDMZ Plant LAN VLAN17 - Layer 2 Domain Plant IP - Subnet /24 Plant-wide Site-wide Operation Systems Industrial Zone Level 3 - Site Operations Levels 0-3 Physical or Virtualized Servers (Plant-wide Network) Application Servers and Services Platform Network Services for example, DNS, AD, DHCP, AAA Remote Access Server (RAS) Storage Array Cell/Area Zone #1 Subnet /24 Cell/Area Zone #2 Subnet /24 Cell/Area Zone #3 Subnet /24 PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 22

23 Structure, Hierarchy and Segmentation VLAN Segmentation without NAT PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 23 Enterprise-wide Business Systems Levels 4 & 5 Data Center Enterprise Zone Level IDMZ Plant LAN VLAN17 - Layer 2 Domain Plant IP - Subnet /24, every device requires a unique IP address Plant-wide Site-wide Operation Systems Industrial Zone Level 3 - Site Operations Levels 0-3 Physical or Virtualized Servers (Plant-wide Network) Application Servers and Services Platform Network Services for example, DNS, AD, DHCP, AAA Remote Access Server (RAS) Storage Array Cell/Area Zone #1 VLAN10 Subnet /24 Cell/Area Zone #2 VLAN20 Subnet /24 Cell/Area Zone #3 VLAN30 Subnet /24

24 Structure, Hierarchy and Segmentation VLAN Segmentation with NAT PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 24 Enterprise-wide Business Systems Levels 4 & 5 Data Center Enterprise Zone Level IDMZ Plant LAN VLAN17 - Layer 2 Domain Plant IP - Subnet /24 Plant-wide Site-wide Operation Systems Industrial Zone Level 3 - Site Operations Levels 0-3 Physical or Virtualized Servers (Plant-wide Network) Application Servers and Services Platform Network Services for example, DNS, AD, DHCP, AAA Remote Access Server (RAS) Storage Array Cell/Area Zone #1 VLAN10 Subnet /24 Cell/Area Zone #2 VLAN20 Subnet /24 Cell/Area Zone #3 VLAN30 Subnet /24

25 Network Availability Redundant vs. Redundant Path PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 25 Redundant Ethernet Networks Independent LANs Independent Paths Beacon Protocol Redundant Path Ethernet Network Common LAN Redundant Paths Resiliency Protocol

26 Network Availability Redundant Path Topologies with Resiliency Protocols Redundant Star Flex Links Cisco Catalyst 3750 StackWise Switch Stack Switch-level Topologies Ring Resilient Ethernet Protocol (REP) Cisco Catalyst 3750 StackWise Switch Stack Star/Bus Linear Cisco Catalyst 3750 StackWise Switch Stack Cisco Catalyst 2955 HMI HMI Controller HMI Controllers Controllers HMI Controllers, Drives, and Distributed I/O Cell/Area Zone Cell/Area Zone Controllers, Drives, and Distributed I/O Cell/Area Zone Device-level Topologies Stratix 8300 Switch-level and Device-level Topologies Controllers, Drives, and Distributed I/O Cell/Area Zone VFD Drive I/O Controller Servo Controllers, Drive Drives, and Distributed I/O Cell/Area Zone I/O VFD Drive HMI PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 26 HMI Controller I/O I/O Instrumentation VFD Drive

27 Network Availability Redundant Path Topologies with Resiliency Protocols PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 27 Industrial Zone HMI I/O Redundant Controllers CIP Class 3 IES IES IES CIP Class 1 Class 3 REP IES IES Controller VFD Drive I/O VFD Drive I/O DLR CIP Class 1 Safety Controller IES CIP Safety Safety I/O HMI Controller DLR CIP Sync Integrated Motion on the EtherNet/IP I/O network Servo Drive VFD Drive Instrumentation

28 PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 28 Convergence-ready Network Solutions Design and Implementation Considerations Partner Solution(s) for example, Machine Plant-wide Industrial Automation Systems Partner Solution(s) for example, Process Skid Plant-wide / Site-wide Industrial Automation Systems Design and deployment considerations that a partner (for example, OEM, SI, Contractor) has to take into account to achieve seamless integration of their solution (for example, machine, skid) into their customers plant-wide/site-wide network infrastructure. Early, open and two-way dialogue is critical!

29 PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 29 Convergence-ready Network Solutions Design and Implementation Considerations Alignment with End User - Security Stance: Business Practices Corporate/Local Standards Tolerance to Risk Current Status of Network Infrastructure (End User and OEM) Segmentation of Domains of Trust Application Requirements Industrial Security Policies Early, open and two-way dialogue is critical! Physical access, port security, access control lists, application security, remote access (avoidance of back doors) Alignment with industrial automation and control system (IACS) security standards such as ISA/IEC (formerly ISA99) and NIST

30 Convergence-ready Network Solutions Design and Implementation Considerations Alignment with End User - Network Services: Use of a common industrial network technology that fully uses standard Ethernet and IP networking technology as the multi-discipline industrial network infrastructure. Common network infrastructure devices asset utilization Future-ready - sustainability IP addressing schema Who manages? End User (OT/IT) or OEM? Address range (class), subnet, default gateway (routability) Implementation conventions static/dynamic, hardware/software configurable, NAT/DNS Use of Network Services Switches - managed vs. unmanaged, industrial vs. COTS, system vs. component approach Segmentation, data prioritization Topologies - switch-level, device-level, hybrid Availability loop prevention, redundant path topologies with resiliency protocols Time Synchronization Services IEEE 1588 Precision Time Protocol (PTP w/e2e) first fault, SOE, Motion Early, open and two-way dialogue is critical! PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 30

31 Additional Material CPwE Architectures - Cisco and Rockwell Automation PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 31 Websites Reference Architectures Design and Implementation Guides ENET-TD001E-EN-P - Converged Plantwide Ethernet (CPwE) Baseline Document ENET-TD005B-EN-P - Deploying the Resilient Ethernet Protocol (REP) in a Converged Plantwide Ethernet Architecture ENET-TD006A-EN-P - Deploying Wireless LAN Technology within a Converged Plantwide Ethernet Architecture ENET-TD007A-EN-P - Deploying Network Address Translation within a Converged Plantwide Ethernet Architecture ENET-TD008A-EN-P - Deploying Identity Services within a Converged Plantwide Ethernet Architecture ENET-TD009A-EN-P - Securely Traversing IACS Data Across the Industrial Demilitarized Zone ENET-TD010A-EN-P - Deploying A Resilient Converged Plantwide Ethernet Architecture

32 PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 32 Additional Material CPwE Architectures - Cisco and Rockwell Automation Application Guides ENET-TD003A-EN-E - Fiber-optic Infrastructure Application Guide (Panduit/Cisco/Rockwell Automation) Whitepapers ENET-WP022B-EN-P - Top 10 Recommendations for Plant-wide EtherNet/IP Deployments ENET-WP009A-EN-P - Achieving Secure Remote Access to plant-floor Applications and Data ENET-WP031A-EN-P - Design Considerations for Securing Industrial Automation and Control System Networks ENET-WP033A-EN-P - Resilient Ethernet Protocol in a Converged Plantwide Ethernet (CPwE) Architecture ENET-WP034A-EN-P - Deploying Wireless LAN Technology within a Converged Plantwide Ethernet Architecture ENET-WP036A-EN-P - Deploying Network Address Translation within a Converged Plantwide Ethernet Architecture ENET-WP037A-EN-P - Deploying Identity Services within a Converged Plantwide Ethernet Architecture ENET-WP038A-EN-P - Securely Traversing IACS Data Across the Industrial Demilitarized Zone ENET-WP039A-EN-P - A Resilient Converged Plantwide Ethernet Architecture

33 PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 33 Additional Material Training and Certifications Cisco Industrial Networking Specialist Training and Certification E-learning modules (pre-learning courses) Control Systems Fundamentals for Industrial Networking (ICINS) Networking Fundamentals for Industrial Control Systems (INICS) Classroom training Managing Industrial Networks with Cisco Networking Technologies (IMINS) Exam IMINS CCNA Industrial Training and Certification Classroom training Managing Industrial Networks for Manufacturing with Cisco Technologies (IMINS2) Exam IMINS2 Industrial IP Advantage: e-learning CPwE Design Considerations and Best Practices

34 PUBLIC Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. 34 Additional Material Education A go-to resource for educational information about industrial network communication and using standard Internet Protocol (IP) for industrial applications Community of like-minded companies Cisco, Panduit, and Rockwell Automation Receive monthly e-newsletters with articles and videos on the latest trends e-learning courses available on network design topics

35 Copyright 2015 Rockwell Automation, Inc. All Rights Reserved. Thank you for attending! PUBLIC