2015 Softing Inc. All Rights Reserved. tmanager Troubleshooting

Transcription

1 tmanager Troubleshooting

2 Softing Softing is a leading global provider of products for Industrial Automation and for Automotive Electronics. For over 30 years, the core business of Softing Industrial Automation has been industrial data communication with products ranging from chips & stacks up to dedicated gateways and data transfer modules. Softing acquired Psiber Data GmbH and OLDI in Psiber Data sells cable testing products. OLDI is an Original Design Manufacturer (ODM) that designs and sells PAC/PLC modules under other company brands and a few with the OLDI brand. OLDI as a company will continue as an ODM. OLDI and Psiber Data products will transition to the Softing brand in Softing also acquired the industrial communications product line from Trebing and Himstedt (TH) in As of January 1, 2015, the official North America company name is Softing Inc.

3 Softing Inc. Sells Softing and OLDI branded products Global Encompass Product Partner Products for Rockwell Authorized Distributors: Data Exchange Modules Communication and Protocol Modules Network Monitoring and Cable Test Tools Headquartered in Knoxville, TN

4 Products sold through the Rockwell Automation Channel Move Data Run Custom Code Certify Cable Connect Networks Monitor Networks eatm Enterprise Appliance Transaction Modules catm Controller Appliance Transaction Modules SAM Special Application Modules WireXpert Advanced Cable Certifier for Ethernet copper or fiber PCI; PCI Express Network Cards for High Temp Environments TH Scope; TH Link Monitor; Diagnose EtherNet/IP and other networks ControlLogix

5 Topics CompactFlash Storage Logs System and Audit Logs Transaction Logs tmanager Statistics StatusTags Supervisory Controller Project Endpoint Message Path / Trigger (Handshaking) Trigger and Endpoint Overrun Timers Trigger & Endpoint Status Screens Cautions when Configuring Status Options Trigger & Endpoint States Errors Trigger, Endpoint & Map Errors Database Validation Viewing Map Values (Debug Map Option) Network Diagnostics Cross-Reference Export Spare Modules Where to get Additional Information / Help

6 tmanager On-Board Compact Flash Storage A minimum of 2 GB of non-volatile flash storage is available for Store & Forward queues, System & Audit logs, and Transaction Logs Total amount available is shown on the eatm Manager Status Screen Even if some tmanager versions allow more than 2 GB total, only 2 GB is guaranteed in future versions The tmanager won t allow you to allocate more space than is available. You will see an error message when you try to save your configuration Breakdown System Log: 20 MB (cannot be changed) Audit Log: 20 MB (cannot be changed) Transaction Logs For each trigger or endpoint that has transaction logging enabled, you can have 2-20 transaction log files (user-defined), & each log file can be 1-10 MB (user-defined) Configurable for each trigger/endpoint Store & Forward Queues Up to 200 MB per endpoint (configured individually for each endpoint)

7 Audit & System Log Details Audit log -- contains user changes that have been made. 20 Mb is allocated System Log -- contains errors and events that have occurred. 20 Mb is allocated Log files are divided into multiple 1 MB log files The date by each individual log file shows the last day it was updated. Double-click on a date to open that particular log file If there s at the end of the message, double-click on the message to read more The messages are displayed in the order they are received into the log If the clock on the tmanager changed at some point in the past, the messages may appear to be out of order You can filter the types of messages that are shown, and search within the current file If a Warning, Error, or Fatal event occurs, the STS LED on the tmanager will blink red, & the icon will appear on the Project Editor & xatm Manager screens Note: After version the STS LED will remain green, even during error conditions. However, the ERR: Check log message will continue to scroll between other information Solid Green means the tmanager is running; Blinking Green means it is Halted The Requires Attention icon can be cleared via the xatm Manager Status screen

8 Audit Log

9 System Log

10 System and Audit Log Configuration Screen

11 System and Audit Log Files - Configuration The xatm Manager s Configure Logs screen lets you view and modify the following System and Audit log settings Log detail level: Specifies the level of detail to log in the System log INFO -- will see Informational, Warning, Error and Fatal messages (default) WARN -- will see Warning, Error, and Fatal messages ERROR -- will only see Error and Fatal messages DEBUG -- will see all the messages, plus additional messages that may be helpful with troubleshooting. Don t keep DEBUG selected long-term as it may have an impact on the tmanager performance Enable notification: If notification is enabled, you can specify the settings for sending an notification when Warning, Error, or Fatal entries are logged to the System log Note that after version the STS LED will no longer turn Red when a new Warning or Error occurs. Thus, enabling notifications is recommended. FTP export: If the enabled, when a log file is full it will be automatically exported and sent to the configured FTP site SMTP export: If the enabled, when a log file is full it will be automatically exported and sent to the configured address

12 Transaction Logging A Transaction Log keeps a log of transactions that have occurred on a trigger or endpoint. Transaction Logging can be individually selected and configured for each trigger and each endpoint Uses: For startup and troubleshooting Database Endpoint Transaction Log -- includes the SQL messages sent to the database, data being transferred, and any database error messages Both successful and unsuccessful transactions are logged. Data is displayed in decimal Separate entry for each message path Automation Endpoint Transaction Log -- includes the hex data sent to the controller Separate entry for each message path Trigger Transaction Log -- includes the values of the trigger Variable and Reference fields (if used), along with the values of all Input tags used in the map and all tags used in a message path Where field Values of tags used in the map and message path are displayed in both decimal and hex New entry each time the trigger is fired. If the trigger is associated with multiple message paths, data from all the associated messages paths is in the same entry. To maintain a historical record of what was sent for regulatory requirements or ongoing machine analysis You can have the Transaction Log automatically exported via FTP or SMTP when a file becomes full Recommended if you wish to maintain the information for longer periods of time For tmanager higher speed applications, enable Transaction Logging only during trouble-shooting

13 Trigger Transaction Log Examples Trigger used in an Insert or Procedure/Function Message Path - Values of map Input tags - If used, trigger Variable and Reference values Trigger used in a Select or Update Message Path - Values of tags used in the message path Where field - If used, trigger Variable and Reference values

14 Endpoint Transaction Log Examples Database Endpoint Example Trigger transaction log: Automation Endpoint

15 Transaction Logging Configuration The following settings are available on the Transaction Log tab on the Trigger & Endpoint configuration screens: Transaction Log Settings: Enables the transaction log Number of log files (min 2): The no. of transaction log files to store on the compact flash The minimum allowed with the transaction log enabled is 2; the maximum is 20 Log file size (MB): The size, in megabytes, the transaction log is allowed to grow to before creating a new log file The minimum size is 1 MB; the maximum is 10 MB Message bytes to log: The amount of the message (in bytes) to log. A message larger than this size will be truncated Log entire message: If this option is selected, the whole message is logged, regardless of size FTP export enabled: If enabled, the tmanager exports the log via FTP when the log file reaches its maximum size and a new log file is created SMTP export enabled: If enabled, the tmanager exports the log as an attachment to an when the log file reaches its maximum size and a new log file is created

16 tmanager Status Information tmanager Statistics viewed on the xatm Manager Status Screen Supervisory Controller Status Tags defined on the Project Editor Supervisory Controller Settings screen Project Status Tags defined in the Project Editor under Edit --> Project Settings Endpoint Status Tags defined in the Project Editor on the Endpoint Configuration screen Message Path Status Tags Defined in the Project Editor on the Trigger Configuration screen Trigger & Endpoint Status Screens Show success and error counts, min/max/average times, dropped messages, etc Trigger and Endpoint Overrun Timers Defined under the Advanced tabs on the Trigger and Endpoint Configuration screens Requires Attention status Answer 113 in the OLDI Knowledgebase ( ) describes the various ways to monitor if there are unacknowledged warnings or errors

17 tmanager Statistics Displayed on the xatm Manager Status screen, and updated every 60 seconds If Memory Used increases to 90% the tmanager halts. It will restart once memory usage decreases to 85% The Requires Attention icon is displayed when one or more Warning, Error, or Fatal events have occurred To clear it, double-click on the icon and then select Clear Events

18 Supervisory Controller Status Tags Update Rate: If a Supervisory Controller is defined, tmanager status information is updated and sent to the Supervisory Controller every second. However, status information is only updated on the xatm Manager Status Screen every minute. Index 0 (Status Bits) Bit 0 (Needs Attention): When this bit is on, there are one or more errors that need attention that have occurred on the appliance. This means the Needs Attention indicator is active Bit 1 (PLC Command Error): The last controller command executed had an error Bit 2 (Supervisory Confirmation Flag): This bit is not displayed in the status data. Bit 2 is set only in the controller that owns the Control portion of the AOP connection. If a backplane AND an EtherNet/IP controller have a valid AOP connection, bit 2 will be set in one or the other (depending on the Supervisory Controller settings). The one with bit 2 set has Control (supervisory privileges) and may send commands to the appliance. If bit 2 is clear, the connection is read-only Note: Multiple backplane and/or multiple EtherNet/IP AOP connections is NOT supported. However, one backplane controller and one EtherNet/IP controller may simultaneously connect, with one having Control and the other just able to read Status Data. Bit 16 (Data in S & F): One or more Store & Forward queues are more than 1% full. Index 1 (Appliance Mode): The mode of the appliance. 1 = appliance is halted 2 = appliance is suspended (free memory on the appliance dipped to less than 10%, and hasn t returned to 15%) 3 = appliance is running Index 2 (Command Trigger Echo): This index is set to the output index 0 command trigger value after the command has completed. Index 3 (Free Running Counter): A free running counter to indicate that the appliance is still alive.

19 Project Status Tags To configure, go to Edit Project Settings within the Project Editor If you are using a Supervisory controller to start/stop projects: Either the Supervisory Controller or Project Editor can stop a project Both have to be in agreement to start a project (needs to be started in the software and the Supervisory Controller state for the project must also be Run Mode) The value of the Project Status Tag bits are: Bit 0: User Interface flag for the project (0 = Stop, 1 = Run) Bit 1: Supervisory Controller flag for the project (0 = Stop, 1 = Run) The Project Status Tags are updated as the project status changes, or at a minimum of once every 20 seconds Probably don t need if not using a Supervisory Controller to start/stop projects

20 Endpoint Status Tags Value reflects the status of the last transaction to the endpoint Database Endpoints: Status Tags reflect the status of the last transaction to that endpoint, both read and write Automation Endpoints: Status Tags reflect the status of only the latest write transaction to that endpoint. An error on reading information from an Automation device (i.e. a CompactLogix PAC) would not change the status The Endpoint Status Tags are updated only on change Status will not be updated if the communications continue to be successful If communications fail, the status will change Define Endpoint Status Tags only if the PLC/PAC will be monitoring the overall health of this connection Otherwise, unnecessary overhead

21 Endpoint Status Tag Values The status values reported in the Endpoint Status Tag are: Success = 0 Last transmission was successful MsgFailed_WillRetry = 1 Last transmission timed out will retry message (for example, couldn t communicate with the database) If S&F is enabled, means the last message through the endpoint was placed in the S&F queue MsgFailed_Discarded = 2 Last transmission failed will discard message Either the message was discarded because the S&F queue is full, or an error code was received from the database Stopped = 3 The destination is stopped on the tmanager (for example, the project or endpoint is stopped) Failover = 4 The destination was unavailable, so the tmanager has switched over to the defined Failover endpoint

22 Message Path Status Tags Message Path Status Tags report the handshaking status, and are updated on change Message Path Status Tags are define on the configuration screen for the associated trigger Only define if the PLC/PAC logic will be monitoring the completion status of the individual message path. Otherwise, unnecessary overhead If a trigger is used in multiple message paths, a separate status tag can be set for each message path Handshake Mode defines the Handshake position the message path must be in, in order for the trigger to fire again

23 Handshake Modes Handshake Modes (in order of execution) Received from Src: The trigger is rearmed when data is received from the source trigger (seldom used) In Queue: The trigger is rearmed when the message is sent to the destination endpoint or goes into the endpoint's queue. This is the option recommended for Store & Forward, and is the default for Insert and Update Message Paths Sent to Dest: The trigger is rearmed when the message is sent to the destination endpoint Store & Forward won t work if this mode is selected for a message path Use if you want to stop the line if your Insert or Update is not completed for any reason Response in Queue: This is only an option for Select or Procedure/Function message paths. The trigger is rearmed when the message response is sent back to the source (inbound endpoint) or goes into the inbound endpoint s queue. This mode is seldom used Store & Forward won t work if this mode is selected for a message path Sent to Src: This is only an option for Select or Procedure/Function message paths, and is their default handshaking mode. The trigger is rearmed when the message response is sent back to the source (inbound endpoint) Store & Forward won t work if this mode is selected for a message path. Change to In Queue if you want to use S & F in conjunction with a Select or Stored Procedure Message Path

24 Message Path Status Tag - Values Each Message Path goes through all 5 handshaking stages (10 thru 50), independent of the handshaking value defined for the message path, unless the trigger re-fires before the entire cycle is complete Message Path Tag Values: Received from Src: 10 In Queue: 20 (stays here while data is in the Endpoint s Queue) Sent to Dest: 30 Response in Queue: 40 Sent to Src: 50 Message Dropped: 101 (S & F Queue is full or not enabled) Error: 102 (error code returned from adapter) Expired: 103 (message exceeded Time To Live in S & F Queue) Trigger Faulted: 104 (this may be caused by a tag missing in the map or trigger event) Message was Flushed: 105 (this is caused when the user flushes the S & F Queue)

25 Trigger and Endpoint Overrun Timers Endpoint Transaction Overrun Timer: If set to non zero, an error will be reported if the time for an endpoint to send data and receive a response exceeds the specified value. Trigger Scan Overrun Timer: For periodic triggers, if set to non zero, an error will be reported if the time from trigger evaluation to the next trigger evaluation exceeds this value. Execution Overrun Timer: If set to non zero, an error will be reported if the time from trigger evaluation to trigger completion exceeds this value. Trigger completion occurs when the handshake mode for all message paths linked to this trigger have been fulfilled. Total number of overruns is shown on the Trigger and Endpoint Status screens (next 2 slides).the amount of each overrun is recorded in the System Log error messages

26 Trigger and Endpoint Status Screens Used to monitor overall success and errors concerning triggers and endpoints, as well as execution times, and the status of the S&F Queues Can also view the Endpoint Transaction Overrun & Trigger Execution Overrun totals Can be accessed from the Status menu in either software package

27 Trigger and Endpoint Status Screens Individual status screens can be accessed by right-clicking on a trigger or endpoint and picking Status This is the only screen where you can see the Trigger Scan Time Overruns total

28 Cautions Status Tags Since they will add loading on the controller and could affect performance, only define Status Tags for the triggers and endpoints where they are needed and will be used by the controller s application program Trigger and Endpoint Overrun Timers The overrun features should be enabled sparingly since they add overhead to the system which effects overall throughput of all the triggers on the module The values entered for the trigger overrun timers need to be greater than the Scan Period configured for the trigger Recommend a multiplier of 1.5 or greater If the values are too low, the system log will get filled with overrun errors, the module throughput will slow down, and other important events and/or errors may be missed The value entered for the endpoint Transaction Overrun Timer needs to be higher than the acceptable Maximum Execution Time for that endpoint (Max Xact on the Endpoint Status screen)

29 Trigger and Endpoint States If a trigger or endpoint is gray, and you are unsuccessful in starting the project with the tmanager software Check to see if the Supervisory Controller is stopping the Project. Reference the Supervisory Controller Quick Start at If a trigger is running, but not firing when it should Verify that the endpoint it is associated with is running (is green). Depending on the Handshaking mode selected (discussed later), a stopped endpoint could prevent the trigger from re-arming

30 Trigger, Endpoint and Map Errors If a trigger, endpoint or map is yellow or has a red exclamation point (error indicator) Check the System Log for error messages related to which tag or map source wasn t available. If the tmanager cannot access a tag or database column, there is usually an error in the system log Hover over any faulted/error icons to see a descriptive message concerning the source of the problem If the trigger never fired (tag associated with the trigger is not available), there might not be a system log message If the trigger is faulted (is yellow) due to a missing tag on the input side of a map or the condition on the trigger, if the tag becomes available again the trigger will automatically restart If a trigger is faulted and you update the tag list, the trigger state will change to red (stopped). If the trigger is faulted due to items in a map being unavailable, after the tag list is updated a red exclamation point will appear before the affected map, and the map s unavailable items will be highlighted in red

31 Error: Invalid request reading trigger Error: Bad parameter reading trigger Possible Causes A tag associated with the trigger or the input side of a related map does not exist in the controller External Access for the tag is set to None in RSLogix 5000 (v19 or later) Example System Log Text yyy-mm-dd hh:mm:ss,zzz [YFTriggerServerReadThread-8] ERROR - Error on trigger \BasicDBDemo_eATM\Automation\Trigger_Production_Report: <6019h> CLXBP Invalid request reading trigger (BasicDBDemo_eATM\Trigger_Production_Report) group tag (CLX_Demo_eATM\CYCLE_TIME) yyy-mm-dd hh:mm:ss,zzz [YFTriggerServerReadThread-12] ERROR - Error on trigger proj1automationcabinetenteringmes_sta5705frommezz: <6002h> CLXBP Bad parameter reading trigger (proj1cabinetenteringmes_sta5705frommezz) variable Postfoam 2CMES_Mezz_To_Sta5705.Flags.Enable) Resolution 1. Add the referenced tag to the PLC/PAC. If the tag exists, and the External Access for the tag is set to None, use the RSLogix 5000 Tag Editor to change the External Access field to Read/Write or Read Only 2. Update the tmanager Tag List via the tmanager Project Editor

32 Error: Invalid request Possible Causes A tag that's on the output side of a map does not exist in the controller External Access for the tag is set to Read Only or None in RSLogix 5000 (v19 or later) Example System Log Text yyy-mm-dd hh:mm:ss,zzz [\BasicDBDemo_eATM\Automation\Trigger_Change] [\BasicDBDemo_eATM\Path_Select] \BasicDBDemo_eATM\Automation\Endpoint_1: Error writing tags to AutomationAdapter: Error "CLXBP Invalid request" writing tag CLX_Demo_eATM\ScouringPadProfile.Model - Retry in 30 seconds Resolution 1. Add the referenced tag to the PLC/PAC If the tag exists, and the External Access for the tag is set to None or Read Only, use the RSLogix 5000 Tag Editor to change the External Access field to Read/Write 2. Update the tmanager Tag List via the tmanager Project Editor

33 Error: Data type mismatch reading trigger Possible Cause The tag is marked with a different revision number by the controller This is typically only seen with components of a UDT that has been modified. If this occurs, data integrity requires an enumeration update before you are allowed to write to this tag Example System Log Text yyy-mm-dd hh:mm:ss,zzz <601Ah> CLXBP Data type mismatch reading trigger (WetMix_B01B01BatchDataTrigger) group tag (WetMixProgram:B01.BatchData_Buffer[0].ProdID) Resolution Update the Tag List for the referenced controller

34 Error: Connection Timeout Problem You can communicate with the database, but the System Log contains a Connection Timeout error for an endpoint Possible Cause The Transaction Timeout value for an adapter is set too low Resolution Increase the Transaction Timeout value on the Adapter configuration screen to allow more time for the database transactions

35 Viewing Map Values (Debug Map icon) The Debug Map icon allows you to have a one-shot view of the values of the map elements One the icon is selected, the map is armed for debugging When the next trigger fires which executes the map, the values of the map elements (including functoids) are captured and displayed on the map You can press the Debug Map icon again to clear the debug values. You can press it again to re-arm the map for debugging and see another set of values

36 Validating Communications with a Database Within the Project Editor, open the database adapter and select Validate Connection If the Connection Status box doesn t say The connection is valid then Check the System Log for clues to the source of the problem. For instance, if an invalid usename/password is being used If the error is similar to Destination not available or Destination Time-Out Try to ping the MSSQL computer s IP address from the tmanager. Go to the xatm Manager software and select Administration Network Network Diagnostics Ping If the Ping isn t successful, possibly your Windows Firewall was not configured to allow communication with the MSSQL port To adjust your firewall, and double-check other MSSQL setting that need to be set correctly, reference the Configuring MSSQL for Communication with a tmanager or xcoupler document which is found in the OLDI Knowledgebase ( You may have to involve your IT department If the validation is successful, but you can t enumerate, verify that the user has sufficient rights (must be a Project Editor or Administrator)

37 Network Diagnostics The Network Diagnostics pane in the xatm Manager software allows you to run network diagnostics on the tmanager appliance The diagnostics are from the point of view of the tmanager For example, a Ping of a host computer indicates if the tmanager module is communicating with the computer Available Diagnostics: Ping Trace route Path ping IPConfig Also available on the Network screen: Network Snooper Network Routes (mainly for IT use)

38 Cross-Reference Export To create a Cross-Reference Export, launch the Project Editor software, go to the Application drop down list and select Export Cross-Reference This report is often used as part of the system documentation The cross references are stored in a comma separated (CSV) file that can be easily viewed using programs such as Microsoft Excel For each object in the appliance, the CSV file will show where the object is used in all of the tmanager projects. Information for the following objects is included: Tag Database Column XML Schema Element ASCII Schema Entry Binary Schema Entry Prototype Trigger Map Message Path Endpoint

39 Spare Options Spare Compact Flash/ CF (56eATM-tManager-SPARE_CF) Only OLDI compact flashes may be used in a tmanager appliance, and they cannot be opened or copied by other devices When the tmanager backup file is Restored to the Spare CF, a Temporary Key is automatically generated. This Temporary Key will be valid for 10 days. After that, the tmanager's triggers will stop firing. A Permanent Key needs to requested from OLDI. The preferred method is via an OLDI Service Request. Spare module (56eATM-tManager-SPARE) If the original module's CF is used in the Spare Module, when the backup file is Restored the Original License Key will be automatically loaded from the backup file. No further actions related to the License Key are required. If the Spare Module's CF is used, when the backup file is Restored a Temporary Key will be automatically generated. This Temporary Key will be valid for 10 days. After that, the tmanager's triggers will stop firing. A Permanent Key needs to requested from OLDI. The preferred method is via an OLDI Service Request. Fully licensed tmanager appliance When the tmanager backup file is Restored to the Fully Licensed Spare Module, select Yes when asked "Do you want to preserve the current valid license on the module?" If you don't, the Spare Module's license will be over-written. If you accidentally answer No, contact OLDI to get the Permanent Key restored

40 Where to Get Additional Information / Help Access context-sensitive Help by pressing [F1] from within the Project Editor or xatm Manager software programs Go to the Knowledgebase at If you cannot find what you are looking for in the Knowledgebase or Online Help, or need assistance with any of our products, connect Support: Service Request: Go to Select the Support tab and click on the Submit a Service Request Phone: support@softing.us

41 Softing Sales Contacts Scott Sibenac Jim Ralston Pete Desmarais Christophe Renard Regional Sales Western & Mexico Regional Sales Eastern US & Canada WireXpert Business Manager Regional Sales EMEA & Asia Pacific

42 Resources Softing US Web Softing US main website: To place an order Product information Knowledge Base Service Request Phone & FAX In the US: International: FAX: Address Softing Inc Chapman Hwy Knoxville, TN Rockwell Web Softing s Encompass Listing Knowledge Base (Softing reposts many Knowledge Base articles in Rockwell s Knowledge Base)