WW TSS-09 Advanced DAServer Troubleshooting

Transcription

1 Slide 1

2 WW TSS-09 Advanced DAServer Troubleshooting Presented by: Dan Scott / #SoftwareRevolution /InvensysVideos /Wonderware October 2013 /company/wonderware 2013 Invensys. All Rights Reserved. The names, logos, and taglines identifying the products and services of Invensys are proprietary marks of Invensys or its subsidiaries. All third party trademarks and service marks are the proprietary marks of their respective owners.

3 Introduction Troubleshooting Log Viewer and Log Flags DAServer Diagnostics Utilities Network Analyzers OPC Analyzers Slide 3

4 SMC Log Viewer Always check the logger Use Log Flags DASSend DASReceive DASTrace DASVerbose Slide 4

5 DAServer Diagnostics Statistics A quick way to check general DAServer Statistics Slide 5

6 DAServer Diagnostics Client Groups is good All Quality SuiteLink items Find in which items. willfor beall shown the device groups DDESLPlugIn individual Client Group belong Qualityitems is good for oneto. or more items and bad for one or more OPC Groups will items. be shown by the group name created by the Quality is bad OPC Client. for all items. Slide 6

7 DAServer Diagnostics Device Groups List of The <Default> Device Groupsusually group contains items advised from an OPC Clientgroups. All device listed here are An OPC Client can configured in the access anythe DAServer. configured Device <Default> group Group byneed to be does not configuring an created. Access Path in the Client. Slide 7

8 DAServer Diagnostics Messages Lists messages Detailed message being sentview. to the PLC Displays the number of items and errors in each message. Displays the status of the message. Slide 8

9 DAServer Diagnostics Transactions View Completed Transaction Write result will Diagnostic Backlog Size must greater than 0 to view Transactions. Shows Write Transactions. items Successfully give a description of the error which corresponds to the Result code. In Progress Demand Read and Refresh transactions are also displayed. Subscriptions are not displayed. Complete: at Generally used to troubleshoot performance and write issues. least one bad item Complete: all items bad. Demand Writes: the Quality/Results column shows the write complete code. All zeros ( ) indicates success. Slide 9

10 Connection Issues Ping Use to determine if the end device can be reached across the network. Winsock Errors Error values from the Windows Socket API Port Issues Use Portqry to determine if specific ports are listening Slide 10

11 Connection Issues Ping Ping is unsuccessful successful Tests basic connectivity to an end device Using Ping Open a command prompt. Type ping <ip address of end device> Slide 11

12 Connection Issues Winsock Errors Winsock is an Application Programming Interface (API) that developers use to make their programs network-enabled. When a DAServer or other application makes calls to the Windows Sockets API an error may be returned. An example of a logger message indicating a Winsock error: Connection to PLC Port.ModbusPLC at IP Address closed, error code = The DAServer will create new socket connection to send next message. Slide 12

13 Connection Issues Winsock Errors Connection to PLC Port.ModbusPLC at IP Address closed, error code = The DAServer will create new socket connection to send next message. WSAECONNABORTED Software caused connection abort. An established connection was aborted by the software in your host computer, possibly due to a data transmission time-out or protocol error. Some common Winsock errors: WSAECONNREFUSED (10061) Connection refused. No connection could be made because the target actively refused it. WSAETIMEDOUT (10060) Connection timed out. A connection attempt failed because the connected party did not properly respond after a period of time, or the established connection failed because the connected host has failed to respond. Slide 13

14 Connection Issues Winsock Errors Some common Winsock errors continued: WSAECONNRESET (10054) Connection reset by peer. An existing connection was forcibly closed by the remote host. Good sources of information regarding Winsock errors: Microsoft Description of Winsock errors Sockets.com Description of Winsock errors and possible causes. Slide 14

15 Connection Issues Port Issues Typically, a PLC has a specific port listening for connections. The DAServer specifies an IP address and a port number when it makes a connection to a PLC. The PLC must be listening on this port number. Firewalls between the DAServer and the PLC must not be blocking this port. Slide 15

16 Connection Issues Port Issues PC PC PC Connect to port 502 Connect to port 502 Connect to port 502 Block Port 502 Firewall Port 502 Listening Port 502 Not Listening Port 502 Listening PLC PLC PLC Slide 16

17 Connection Issues Port Issues Using Portqry Run Portqry from the DAServer node. Name or IP address to query. Single port to query. Result Port 502 is listening Slide 17

18 Connection Issues Port Issues Using Portqry Result Port 502 is not listening Not Listening indicates Portqry reached the PLC but the PLC is not listening on the port number specified. Slide 18

19 Connection Issues Port Issues Using Portqry Result Port 502 is filtered Filtered indicates Portqry did not receive a response from the target port and the target port is being filtered. A firewall may be blocking the port number specified. Slide 19

20 Connection Issues Port Issues - Common Port Numbers ControlLogix PLCs AB PLC GE PLC Modbus TCP 502 Siemens 102 Suitelink 5413 DCOM 135 Slide 20 The standard port number for TCP to is 502. PortModbus 135 is used initiate Some Modbus devices DCOM communication. allow user configurable Once athe OPC Client and port number. Server are able to The Modbus TCP communicate, theydaserver will also allowsnew theport user to negotiate configure numbers. a port other than 502.

21 Connection Issues Port Issues Resources Portqry can be downloaded from: Description of the Portqry.exe command-line utility: Using PORTQRY for troubleshooting: Slide 21

22 Performance Common parameters that affect performance Number of device groups per PLC object. Number of outstanding / queued messages. Block I/O Size Optimization Parameters Contiguous PLC Addresses PLC Response Time Slide 22 When Some Use The number System to DAServers use Tags of more messages PLC support and Device items Logthat in Groups Flags a PLC can to Generally, reading contiguous configurable determine be single sentpoll. to a latency. device optimization before a reply addresses allows the DAServer tois Whenless parameters. received. to use fewer Device send polls to the PLC. Groups Higher values = Better performance Make sure =the PLC Performance supports the Less Polls Better number Make sure of outstanding the PLC supports messages the configured block sizes in configured. the DAServer.

23 Performance Multiple device groups per PLC object? Consider multiple device groups if some items need to be updated at a fast rate while others don t. Multiple device groups per PLC object can dramatically increase messages sent to the PLC Increasing messages sent to the PLC can dramatically decrease performance. Slide 23

24 Performance Number of device groups per PLC object Test 3 Device Groups Slide 24 3 Device Groups Desired update interval is 501 to 573 milliseconds. Use WWClient to advise 178 items in each device group. PLC addresses are not contiguous.

25 Performance Number of device groups per PLC object Test 3 Device groups 534 Total items split evenly between 3 Device Groups 300 messages being sent to the PLC. Less than 2 items per poll message. Slide 25

26 Performance Number of device groups per PLC object Test 3 Device Groups $SYS$MaxInterval displays the current update interval in milliseconds for all items in the corresponding device group Slide 26

27 Performance Number of device groups per PLC object Test 1 Device Group 532 Total items advised through 1 Group. Same PLC items as the 3 group test 100 messages being sent to the PLC. More than 5 items per poll message. Slide 27

28 Performance Number of device groups per PLC object Test 1 Device Groups $SYS$MaxInterval displays the current update interval in milliseconds for all items in the corresponding device group Slide 28

29 Performance Outstanding Message Configuration Increasing the number of outstanding messages can increase performance. The PLC must support the value entered here. Slide 29

30 Performance Outstanding Message Configuration Example: Outstanding Messages = 1 Advise 500 items = 5 polls to the PLC 12:00: Poll through :00: PLC responds 12:00: Poll through :00: PLC responds 12:00: Poll through :00: PLC responds 12:00: Poll through :00: PLC responds 12:00: Poll through :00: PLC responds Slide 30 Total time to poll all items 75 ms

31 Performance Outstanding Message Configuration Example: Outstanding Messages = 4 Advise 500 items = 5 polls to the PLC 12:00: Poll through :00: Poll through :00: Poll through :00: Poll through :00: PLC responds ( through ) 12:00: Poll through :00: PLC responds ( through ) 12:00: PLC responds ( through ) 12:00: PLC responds ( through ) 12:00: PLC responds ( through ) Slide 31 Total time to poll all items 30 ms

32 Performance Outstanding Message Configuration Test items 4 1 Outstanding Messages Message ms Slide 32

33 Performance Optimization Parameters ABCIP DAServer ABCIP DAServer Optimize No Optimization User Defined Data for Startup Read time Types Thetags DAServer uses the All are accessed bymost from the basicdaserver The communication will attempt method to predefining PLC processor messages using the in the device s available group requests by using fortable. the aoptimizing UDT s tag name controller, memory location therefore for communication elements into a request with the forthe the blocks of information from controller whole structure. controller. This option provides the best overall performance. This option is most effective with a large number of tags on continuous scan. Slide 33

34 Performance Optimization Parameters SIDirect DAServer SIDirect DAServer Cyclic Services Optimization Auto S7 SAPI Block read Cyclic services best have when a reliable youtoado update not If This youmode know ishow always implemented many registers services keep whole the frequently frequency activate/deactivate and need access. items. PLC the byte server array can handle, compatible containing youless can some withbus distribute older items. S7 Theavailable DAServercyclic tries services to register as the PLCs. among manymode items the services as different This device isinbest groups ifcyclic youby frequently using the possible. Limit This activate/deactivate mode cyclicisservices the less-preferred items to: parameter. that have optimization similar addresses. mode.this mode causes Otherwise, less activations/deactivations you can use the on the Maximum available services. protocol. Slide 34

35 Performance PLC Response Time Check Ping response time Statistics Roundtrip Time HowPackets Sent to send long it takes andpackets Received receive the data packet. Packets Lost Slide 35 Minimum round trip time Maximum round trip time Average round trip time

36 Performance PLC Response Time Slow ping time? Use Tracert Tracert Hop number Traces the route between the source and the Round trip time for each destination. packet. 3 packets are sent to eachhow hop.many hops it Shows takes to get to the The IP address each hop. destination. Alsoofdisplays the amount of time each hop takes. Slide 36

37 Performance PLC Response Time Performance Example Test Setup: Computer running DASMBTCP located in Lake Forest, CA DASMBTCP configured for 2 PLCs, 1 topic per PLC with an update rate of 100ms. DASMBTCP set to 1 outstanding message. wwclient advising integer items through DASMBTCP to 2 PLCs. 82 poll messages being generated for each PLC. PLC-1 Located in Lake Forest, CA Ping time = 1ms Time to update all items = <500ms Slide 37 PLC-2 Located in Wyomissing, PA Ping time = 85ms Time to update all items = 7000ms

38 Log Flags DASSend DASReceive DASTrace DASVerbose Displays protocol messages sent to the PLC. Displays protocol messages received from the PLC. Displays protocol-specific diagnostic messages. Modifies other log flags to provide more detailed messages. PoLL message Message 4 sent sent to4for PLC sent READ_4X_REG: Port.ModbusPLC by PLC for READ_4X_REG: => (0 d => ff 3 0SentMessageSize=1, 001) SentMessageSize=0, MessagesToSendSize=0, MessagesToSendSize=0, MaxQueuedMsgs=1 MaxQueuedMsgs=1 Message received from PLC Port.ModbusPLC (0 d ff f2 ) Slide 38

39 Troubleshooting Examples Missing Data for some PLC items Check SMC Logger The PLC Port.ModbusPLC reported receiving an ILLEGAL DATA ADDRESS from the DAServer. PLC errorcode 02 Turn on log flags (DASSend, DASReceive, DASTrace, DASVerbose) Message sent to PLC Port.ModbusPLC (1 e ff 3 4e 15 0 c ) Message received from PLC Port.ModbusPLC (1 e ff 83 2 ) Hex Hex Hex ff 1 e1 3 ff 00 4e c2 Hex 1 e Slide 39 DescriptionDescription Value Value Description Value Description Value PLC Unit IDTransaction ID FF = 255 decimal 01e1 hex = 481 decimal Transaction ID 01e1 hex = 481 decimal PLC Unit ID = 255holding decimal Modbus Function Code 03FF = Read Protocol ID 0000 = register Modbus protocol Protocol ID 0000 = Modbus protocol Modbus Function Code 83 == Read holding register Exception Starting Address (0-based) 4E decimal (PLC ) Number of bytes to follow 0003 = 3 bytes to address follow Number of bytes to follow 0006 = 6 bytes to follow Exception Code to read 02 == Illegal Data Address Number of registers 000C Read 12 registers ( to )

40 Troubleshooting Examples Missing Data for some PLC items We have verified through the protocol that there is an illegal data address when polling addresses through To find the specific illegal data address we can: Slide 40 Set the holding register read block size to 1. Use wwclient to advise through Locate the bad addresses wwclient Diagnostics

41 Troubleshooting Examples Checking for correct data values Scenario: Customer indicates that InTouch is displaying a value of 2449 for Pump1 RPM and believes this value is incorrect. Check the Tagname configuration in InTouch to determine the item name and access name. Check the access name to determine the DAServer and topic. In this example, the DAserver is DASMBTCP, the topic is plc and the item name is Check the name of the Port and PLC object name. (Will be used later when analyzing log file) Slide 41

42 Troubleshooting Examples Checking for correct data values Item name = DAServer = DASMBTCP Topic = PLC Port object name = Port PLC object name = ModbusPLC Use wwclient to advise item Value matches InTouch Slide 42

43 Troubleshooting Examples Checking for correct data values Check the value in DASMBTCP Diagnostics Value matches InTouch and wwclient Slide 43

44 Troubleshooting Examples Checking for correct data values Turn on log flags (DASSend, DASReceive, DASTrace, DASVerbose) Message sent to PLC Port.ModbusPLC (c 6c ff ) Message 3180 sent for READ_4X_REG: => SentMessageSize=1, MessagesToSendSize=0, MaxQueuedMsgs=1 Message received from PLC Port.ModbusPLC (c 6c b ff c ) PoLL message 3180 sent by PLC for READ_4X_REG: => SentMessageSize=0, MessagesToSendSize=0, MaxQueuedMsgs=1 Find the correct send message. Verify the Port object name, PLC object name and item name. Make sure the Transaction ID matches for the Send and receive Verify the Send message: PLC ID = FF (255), Function code 3 = read holding registers, Starting address = 0000 (400001), Number of registers = 0014 (20) Slide 44

45 Troubleshooting Examples Checking for correct data values Message received from PLC Port.ModbusPLC (c 6c b ff c ) Hex Description Value ff PLC Unit ID FF = 255 decimal 3 Modbus Function Code 3 = Read holding register 28 Number of data bytes to follow 28 = 40 decimal 039c Data for Register 0 (400001) 039C = 924 decimal 0991 Data for Register 1 (400002) 0991 = 2449 decimal Result: InTouch is displaying the correct value from the PLC. Slide 45

46 Other Tools Wireshark Network protocol analyzer Captures and logs network traffic Analysis and Statistics Logs can be saved for later analysis Slide 46

47 Other Tools Wireshark Filter by IP address of PLC. Select packet for details. Packet details. Packet bytes. Slide 47

48 Other Tools OPC Analyzer / OPC Sniffer Captures and logs communication between OPC servers and clients Use to troubleshoot issues which require more data than the OPC server and client logs provide. Logs can be saved for later analysis Slide 48

49 Other Tools OPC Analyzer / OPC Sniffer Select OPC Server Shows OPC methods Sent by OPC Client Sent by OPC Server Slide 49

50 Resources Wonderware Tech Notes, Tech Articles, Tech Alerts, Documentation, Downloads Portqry Download Description of the Portqry.exe command-line utility: Using PORTQRY for troubleshooting: Winsock Microsoft Description of Winsock errors Sockets.com Description of Winsock errors and possible causes. Slide 50

51 Resources Network Tools Wireshark Sysinternals Modbus Technical Resources, Modbus Specifications OPC OPC Foundation Slide 51

52 Slide 52