Technical Support Files Needed for Troubleshooting Abstract Check Point Technical Services requests files or information to help facilitate problem resolution. The following document is provided to customers and partners may anticipate what information or files will be requested based on the type of problem they are experiencing. Document Title: Files Needed for Troubleshooting Creation Date: 7-Jan-2004 Modified Date: 8-Jan-2004 Document
TABLE OF CONTENTS ABSTRACT...1 OVERVIEW...3 FIREWALL-1...4 General...4 CORE Crash...4 Dr. Watson...4 INSPECT...4 Kernel Crashes...4 LOG...4 Network Address Translation...4 Resources: CVP...5 Rule Base Problems...5 Security Server...5 APPLIANCE PRODUCTS...5 CVP & UFP Problems...5 Nokia...5 OSE...5 SecurePlatform...5 Small Office Products...6 OPSEC Application...6 HIGH AVAILABILITY...6 ClusterXL...6 Rainfinity Rainwall...6 Stonesoft Stonebeat Full Cluster...6 Reporting Module...6 FloodGate-1...7 ENTERPRISE PRODUCTS...7 General...7 Provider-1...7 SiteManager-1...7 User Authority...7 FireWall-1 GX (Wireless)...8 Customer Logging Module...8 Management Logging Module...8 LDAP Account Management...8 VSX...8 ENCRYPTION PRODUCTS...8 VPN-1 Pro...8 VPN-1 Net...9 VPN-1 Edge...9 SecuRemote...9 SecureClient...9 VPN-1 Mac Client...10 VPN-1 Accelerator Cards...10 SecureXL TurboCard...10 PKI...10 DOCUMENTING TROUBLESHOOTING PRIOR TO CONTACTING SUPPORT...11 Files Needed for Troubleshooting Page 2 of 11
Overview This document will provide a list of information or files that may be requested by Check Point Technical Services when a customer or partner is experiencing a problem with any of the following technologies: FireWall-1 Appliance Products High Availability Products Enterprise Products Encryption Products Additionally, this document will detail how a customer or partner can provide information about troubleshooting steps he or she may have already done prior to contacting support. Files Needed for Troubleshooting Page 3 of 11
FireWall-1 General Complete contact information, (name, title, company name, e-mail address, phone number, pager number, fax number, onsite phone number, time zone) for all parties involved in the issue. Execute the $FWDIR/bin/fwinfo, cpinfo, or ipsoinfo command on all FireWall-1 modules and the FireWall- 1 management station in question, divert the output to a file, and attach the file to a web request. Describe the hardware platform(s) involved in this issue, including the amount of memory, disk space, and NIC card types (manufacturer and model). Describe the operating system(s) involved in this issue, including the version number and patch level information. (Include which service pack and hotfixes for NT, which patches for Solaris, etc.). Provide a detailed description of the problem or issue, including any symptoms noted, any patterns seen (time of day or only certain users affected, etc ) and any specific error messages received. Log file contains relevant log errors Updated SVN Mapping of all the network related to the problem including Hardware/Software detailed descriptions, Network Map, Connections types, bandwidth, and IP addresses of all segment routers and transitional gateways. General information about the network, including: approximate number of users, approximate number of simultaneous sessions per user, types of applications in use, network traffic passing through the software at the time of error, CPU utilization, memory allocation and utilization. An electronic topology diagram is preferred Visio or PowerPoint are good applications to use for this. If this is not feasible, a fax of hand drawn diagrams is an acceptable alternative, provided the IP addresses or Host ID information is legible upon receipt. CORE Crash Core File Dr. Watson Dr. Watson file (drwtsn32.log) User.dmp file (system.dmp in case of a blue screen). INSPECT If a specific SERVICE was mentioned, specify the following: o How does the service work o On which protocol does the service work o On which ports does the service work fwmonitor + a list of the relevant IPs (client, server, FireWall). Kernel Crashes vmcore.x file unix.x file LOG If the problem is related to the Log Viewer, issue the command fw logexport in order to see if all the columns are full. If the log records are not written to the log file ( fw log and fw logexport show no new records), you may want to run fw d d D, which includes special debugging option for FW1_LOG connections for VPN- 1/FireWall-1 v4.1. o fw debug fwd on --> log/fwd.elg o fw debug fwm on --> log/fwm.elg Network Address Translation fwmonitor + a list of the relevant IPs (client, server,firewall) Files Needed for Troubleshooting Page 4 of 11
Issue the command o fw ctl debug -buf o fw ctl debug xlate o fw ctl kdebug -f > /tmp/kdebug.out and send the file (In case of FTP or TELNET, you can add the option xltrc after the option xlate )..After the problem occurs, stop this command with ^C, and run fw ctl debug 0. Resources: CVP Issue the command snoop on port 18181 fwopsec.conf file cvp.conf file on the CVP side Set the environment variable OPSEC_DEBUG_LEVEL to 3, and restart fwd. Send the output received in fwd.log. Rule Base Problems fwmonitor + a list of the relevant IPs (client, server, FireWall). Security Server fwmonitor + a list of the relevant IPs (client, server, FireWall). Run the Authentication daemon in Debug and send the log/ahttpd.elg file. If the problem is related to SMTP, send the spool directory and run the mail dequeuer and the asmtpd in debug mode. Appliance Products CVP & UFP Problems cpinfo from FireWall-1 Enforcement module cpinfo from SmartCenter Management module CVP or UFP product name and version URL of web site if the problem is with accessing a certain web site ahttpd, aftp etc. debug (in case it's http related issue) fw monitor (including the IP addresses of all parties) Web/FTP site trying to be accessed fw.log file (when there are error messages in the log viewer.) or an export of the relevant log records Important: Make sure you verify whether the problem occurs with/without UFP/CVP Nokia ipsoinfo from FireWall-1 Enforcement module ipsoinfo from SmartCenter Management module OSE cpinfo from SmartCenter Management module Router type and OS version For Cisco and Nortel (Bay), obtain a copy of the routers configuration (*cfg file) SecurePlatform cpinfo from FireWall-1 Enforcement module cpinfo from SmartCenter Management module For user mode crash - send the user dump o Use the 'ulimit -c unlimited' command to configure the machine to generate cores. For kernel mode crashes: Files Needed for Troubleshooting Page 5 of 11
o Send the crash dump file located in: /var/log/dump/x (where x is the crash number) o Send the /var/log/dump/analysis file Did customer add patches? Which ones? Hardware NIC Drivers (if the problem related to NIC) Small Office Products cpinfo from FireWall-1 Enforcement module cpinfo from SmartCenter Management module Small Office product name & model number Hot Fix number (if any used) History of RPM installations OPSEC Application Vendor and version of OPSEC application cpinfo from management and module Log files from the OPSEC vendor application (when available) OPSEC debug on the Application side (when available) o Usually to run it simultaneously with FireWall-1 OPSEC debug (on the FireWall-1 module side) High Availability ClusterXL cpinfos from the SmartCenter Server and Enforcement points fw ctl debug buf 4096 fw ctl debug m cluster all fw ctl kdebug f > <file name> Rainfinity Rainwall cpinfo Rainfinity version *.cfg files from Rainwall fw ctl debug buf 4096 fw ctl debug misc fw ctl kdebug f > <file name> Stonesoft Stonebeat Full Cluster cpinfo StoneBeat version sbinfo $sbfchome/etc directory from StoneBeat fw ctl debug buf 4096 fw ctl debug misc - only if they use sync fw ctl kdebug f > <file name> Reporting Module cpinfo (from SmartCenter only) Files Needed for Troubleshooting Page 6 of 11
reporting server directory (Program Files/Checkpoint/Reporting Module or /opt/cprt-50 directory disregarding the database directory) rtserver debug log consolidator debug The fw log files $FWDIR/log directory FloodGate-1 cpinfo fw ctl debug -m FG-1 Enterprise Products General Latest cpinfo file Provider-1 cpinfos from MDS environment and CMA environment SIC problems o cpd debug on MDS o cpd debug on individual CMA Copy of $MDSDIR/conf/mdsdb directory (the latest cpinfo includes it) fwd debug for logging/status/connectivity issues fwm debug for gui/management issues mds_backup SiteManager-1 cpinfos from MDS environment and CMA environment SIC problems o cpd debug on MDS o cpd debug on individual CMA Copy of $MDSDIR/con/mdsdb directory fwd debug for logging issues fwm debug for GUI/management issues mds_backup User Authority cpinfo from management and gateway netsod debug on gateway SIC problems o cpd debug on domain controller Information from Domain Controller for authentication problems: cpinfo, netsod debug, ipconfig /all output Netcat between Domain controller and Secure Agent. Netcat between Module and Domain Controller Files Needed for Troubleshooting Page 7 of 11
FireWall-1 GX (Wireless) cpinfo from management/gateway Good topology description fw.log Customer Logging Module GUI problems - fwm debug Logging problems - fwd debug SIC problems - cpd debug cpinfo Check to determine if there are crashes Management Logging Module cpinfo on MDS for MDS and problematic CMA environment cpinfo from MLM $MDSDIR and corresponding CLM environments GUI problems o fwm debug in proper CLM $FWDIR Logging problems o fwd in proper CLM $FWDIR mds_backup LDAP Account Management cpinfo from SmartCenter Server and Enforcement module fw monitor of traffic between Enforcement module and LDAP server output of ldapsearch command fwd debug output Product name and version of the LDAP server and any relevant logs or errors messages from it. VSX mds_backup from Provider-1 VSX MDS cpinfo from the problematic CMA environment (mdsenv <cma name>) output of fw vsx stat v command on the VSX Gateway cpd.elg (cpd_admin debug on) from the VSX MDS and Gateway for virtual system creation, policy installation and SIC issues fw monitor vs <vsid> from problematic Virtual System cpinfo c <vsid> -o <file> from the VSX Gateway fw ctl debug with necessary flags Encryption Products VPN-1 Pro Monitor from VPN-1 Enforcement modules involved in VPN vpnd.elg file from VPN-1 Enforcement modules involved in VPN (vpn debug on) IKE.elg file from VPN-1 Enforcement modules involved in VPN (vpn debug ikeon) Any error messages seen in log viewer cpinfo from VPN-1 Enforcement modules involved in VPN Files Needed for Troubleshooting Page 8 of 11
cpinfo from SmartCenter Management module(s) of the above VPN-1 Enforcement modules Network description Core files if any VPN-1 Net Monitor from VPN-1 Enforcement modules involved in VPN vpnd.elg file from VPN-1 Enforcement modules involved in VPN (vpn debug on) IKE.elg file from VPN-1 Enforcement modules involved in VPN (vpn debug ikeon) Any error messages seen in log viewer cpinfo from VPN-1 Enforcement modules involved in VPN cpinfo from SmartCenter Management module(s) of the above VPN-1 Enforcement modules Network description Core files if any VPN-1 Edge http://my.firewall/pub/test.html diagnostics output from http://my.firewall, setup> firmware> diagnostics exported configuration (.cfg) from http://my.firewall, setup> tools> export cpinfo from central site VPN-1 Enforcement module(s) and SmartCenter Server involved in VPN vpnd.elg and ike.elg from central site VPN-1 Enforcement modules involved in VPN (vpn debug on, vpn debug ikeon) SecuRemote Monitor from VPN-1 Enforcement modules involved in client to FireWall VPN Monitor (or anlz) output from client involved in client to FireWall VPN vpnd.elg file from VPN-1 Enforcement modules involved in VPN (vpn debug on) IKE.elg file from VPN-1 Enforcement modules involved in VPN (vpn debug ikeon) IKE.elg file from client involved in client to FireWall VPN Any error messages seen in log viewer cpinfo from VPN-1 Enforcement modules involved in VPN cpinfo from SmartCenter Management module(s) of the above FireWalls srinfo from client *.log files form log directory Network description SecureClient Monitor from VPN-1 Enforcement modules involved in client to FireWall VPN Monitor (or anlz) output from client involved in client to FireWall VPN o The command "srfw monitor.." - starting from NG FP2 vpnd.elg file from VPN-1 Enforcement modules involved in VPN (vpn debug on) IKE.elg file from VPN-1 Enforcement modules involved in VPN (vpn debug ikeon) IKE.elg file from client involved in client to FireWall VPN Any error messages seen in log viewer cpinfo from VPN-1 Enforcement modules involved in VPN cpinfo from SmartCenter Management Module(s) of the above FireWalls srinfo from client o If it's a problem getting the policy, or logging onto the Policy Server, we'll need the dtpsd.elg file (dtps debug on) *.log files from log directory Network description vpnd.elg file from VPN-1 Enforcement modules involved in VPN (vpn debug on) Files Needed for Troubleshooting Page 9 of 11
VPN-1 Mac Client Monitor from VPN-1 Enforcement modules involved in client to FireWall VPN IKE.elg file from VPN-1 Enforcement modules involved in VPN (vpn debug ikeon) *.alf files from VPN-1 Client folder on the Macintosh in question cpinfo from VPN-1 Enforcement modules involved in VPN cpinfo from SmartCenter Management Module(s) of the above FireWalls VPN-1 Accelerator Cards Output of 'vpn accel stat -l' Collect console error messages o Windows - Error messages in event viewer (copy of event logs) o Solaris - /var/adm/messages o Linux - /var/log/messages lunadiag (test #9) bcmdiag used via the GUI in Win NT/Win 2000 or via commands: bcmdiag -(vsx) in Linux and Solaris SecureXL TurboCard Output of fwaccel stat Output of fwaccel conns Output of vpn accel stat for encryption issues fw ctl debug buf 4096 fwaccel dbg <flag> fw ctl debug f > <file> PKI output of vpn crlview d obj <fw object name> -cert <cert nickname> vpnd.elg (with vpn debug on) ike.elg with (vpn debug ikeon) cpinfos Certificate authority product name and version and output of any relevant logs or error messages from the server. Files Needed for Troubleshooting Page 10 of 11
Documenting Troubleshooting Prior to Contacting Support Check Point encourages customers and partners to provide any troubleshooting information they may have done prior to contacting Check Point. To help our technical advisors easily determine what a customer or partner may have already reviewed, please be ready to provide or document the as much of the following information as possible: Additional/Alternate Customer's Contact name, email address & phone # Problem description including: current OS & FW (include hotfix) version, what triggered the problem (include specific error messages) Business Impact Network topology (Include other CP products/builds and other involved machines) If other servers are involved, state product name, version etc What was checked /tested (detail tests and results) What databases were used for reference/troubleshooting (SecureKnowledge/Manuals/etc.) and what were the results Suggested next steps Attached files If you believe you have discovered a bug, please provide the following information: Bug information: Brief problem summary Test results summary Test bed configuration (test rack setup) Test methodology (procedure used to replicate) Any relevant crash or debug files Files Needed for Troubleshooting Page 11 of 11