VSX Troubleshooting. Quick guide

Size: px

Start display at page:

Download "VSX Troubleshooting. Quick guide"

Hugo Barrett
5 years ago
Views:

1 VSX Troubleshooting Quick guide

2 Agenda How VSX is built (in brief) Management scheme Gateway architecture Licensing Issues to fix Tools and methods 2

3 Reference Note Pictures from Check Point publicly available documents are used in this presentation Information from Check Point troubleshooting documentation used in this presentation 3

4 Management Side VSX Architecture

5 MGMT model of VSX Three tear infrastructure Two types SmartCenter Provider-1 5

6 SmartCenter Model Nothing special 6

7 Provider-1 Model Virtual Systems are managed by different CMAs Special so called Main CMA to manage VSX cluster objects Target CMAs to manage particular VSs 7

8 Provider-1 Model 8

9 MGMT DB objects Two types instead of one as for regular FW network_object - security aspects of a Virtual Device vs_slot_objects - networking aspects of a Virtual Device network_object - on the Target CMA vs_slot objects - on Main CMA 9

10 vs_slot objects Special DB table called vs_slot_objects Network interfaces of VS Routes of VS Other VS specific attributes, such as reference to hosting VSX object Vital info for creation and change of VS 10

11 Network Configuration Scripts Configuration changes are pushed to VSX from MGMT with NCS NCS are generated on MGMT On VSX GW NCS are parsed and executed 11

12 Types of NCS local.vs - NCS file - last configuration change local.vsall - NCS file, full configuration, executed on startup local.vskeep - contains list of existing VSIDs, used at system startup 12

13 local.vs Interfaces lists of each vs_slot: interfaces - interfaces configuration to be interfaces_installed - existing interfaces configuration Each vs_slot object has 2 attributes containing routes lists: routes - routing table to be routes_installed existing routing table local.vs file is created by comparing and calculating the difference of interfaces to interfaces_installed and routes to routes_installed 13

14 local.vsall and local.vskeep Each Virtual Device has 2 NCS files kept on the management: VS_name.vsnew - NCS file containing interfaces VS_name.vsrt - NCS file containing routes These files are updated each time configuration change is done using SmartDashboard. local.vsall is created by concatenating all the files of all the Virtual Devices. local.vskeep is created by going over vs_slo_objects table and writing all the Virtual Devices VSIDs to it. 14

15 Location of NCS files All.vsnew and.vsrt files - in $FWDIR/conf/ vs_repository/vsx_name of Main CMA MGMT: local.vs, local.vsall and local.vskeep files - in $FWDIR/state/VSX_NAME/VSX/ VSX GW: local.vs, local.vsall and local.vskeep files in $FWDIR/state/ tmp/ VSX/ If scripts processed successfully, they copied to to $FWDIR/state/local/VSX/ directory 15

16 Provider-1 forwarding 16

17 VSX private network Funny IPs For internal communication Default cluster private network: /22 Cluster Private Network can be changed: - In SmartDashboard, if there is no VDon the VSX - By using vsx_util change_private_net 17

18 gateway Side VSX Architecture

19 Important note VS is NOT a virtual machine Common file space Common kernel VRFs and kernel contexts are different 19

20 VRFs A Multiple routing domain (VRF) has separate: VRF ID Interfaces Unicast routing table Routing cache Multicast forwarding cache ARP table Loopback interface Sockets VRFs enable overlapping IP addresses. 20

21 VRF: CLI changes Mind context when working with ip route, iputils, ping, traceroute, arp, route, ifconfig and netstat: - traceroute Z vrfid - ip route vrf vrfid - -z vrfid for the rest (arp z 1, netstat z 2 -rn) Use all instead of vrfid to show information for all VRFs VRF context can be changed by vsx set <vsid> or vrfctl s <vrf> commands. VRF 0 - physical machine 21

22 VSX user mode Processes: Multi context: fwd, cpd, cplogd and fibmgr Single context: vpnd and gated System resources like CPU and HDD space are shared 22

23 File Structure VS folders - CTX under $FWDIR / $CPDIR CTX00xxx - Virtual Device ID xxx. VSX machine (VSID 0) - $FWDIR / $CPDIR 23

24 Creating VS object 1 License validation Update context database with Virtual Device information $CPDIR/conf/ctxdb.C) Create Virtual Device registry entries (OTP for SIC certificate) Create Virtual Device directories and soft-links: $CPDIR/CTX/CTX00xxx/conf $FWDIR/CTX/CTX00xxx/log, database, 24

25 Creating VS object 2 Create initial policy Create the OS VRF instance Create the VS instance in the FW kernel and load security policy Send a message that notifies cpd and fwd that a new context was added. cpd adds the new context to its db. 25

26 Troubleshooting techniques

27 Useful knowledge Management debugging (ref. P-1 lecture) Gateway architecture and troubleshooting techniques ClusterXL SecureXL 27

28 Things to Check First Licensing on both MGMT and GW sides Connectivity between VSX and MGMT All the jazz: - local time settings - static routes - IP addressing - mind funny IPs - etc... 28

29 Management Debugging

30 Management Issues Provisioning Changes vsx_util operations policy installation 30

31 Important Do not lock Main CMA while working with VSX on Target CMAs 31

32 Debbuging fwm TDERROR_ALL_ALL - might be too much vsx provisioning and vsx_util: TDERROR_ALL_VSXM Policy installation: TDERROR_ALL_INSTMGR 32

33 How to set debug flags Mind context!!! fw debug fwm on TDERROR_ALL_VSXM=INFO Or export TDERROR_ALL_VSXM=INFO and restart fwm process 33

34 Debug output $FWDIR/log/fwm.elg 34

35 Turning it off fw debug fwm TDERROR_ALL_VSXM=0 fw debug fwm off 35

36 Which CMA? Most of the cases - Main CMA Policy installation - Target CMA 36

37 Gateway Debugging

38 Common Issues Connectivity Policy Interfaces Clustering 38

39 To check first Connectivity Topology of VSX cluster and adjacent networks Local times Licenses 39

40 Overvew vsx stat -v VSX Gateway Status ================== Name: test1 Security Policy: Standard Installed at: 25Jul2010 3:42:11 SIC Status: Trust Number of Virtual Systems allowed by license: 25 Virtual Systems [active / configured]: 7 / 7 Virtual Routers and Switches [active / configured]: 0 / 0 Total connections [current / limit]: 4994 / Virtual Devices Status ====================== ID Type & Name Security Policy Installed at SIC Stat S test1_xxxxxxxxxxxx1... Standard 25Jul2010 3:42 Trust 2 S test1_xxxxxxxxxxxx2... Standard 25Jul2010 3:42 Trust 3 S test1_xxxxxxxxxxxx3... Standard 25Jul2010 3:42 Trust 4 S test1_xxxxxxxxxxxx2... Standard 25Jul2010 3:42 Trust 5 S test1_xxxxxxxxxxxx2... Standard 25Jul2010 3:42 Trust 6 S test1_xxxxxxxxxxxx2... Standard 25Jul2010 3:42 Trust 7 S test1_xxxxxxxxxxxx2... Standard 25Jul2010 3:42 Trust 40

41 Tools tcpdump -i <IF name> fw monitor [ v <vsid>] -e <Your filter> Example: fw monitor v 2 e port(80) and ip_p=17, accept; Note: changing context does NOT help fw monitor to limit output 41

42 Acceleration Most of fw monitor output is accelerated, so you will see just the first packet. fwaccel [-vs <vsid>] [conns templates stat on off] Mind VS number 42

43 Cluster Issues Get status cphaprob [-vs vsid] stat Interfaces status cphaprob -a [-vs vsid] if Problem notification list: cphaprob [-vs vsid] list Force member UP or DOWN, for failover tests: clusterxl_admin up/down 43

44 Kernel Debug One kernel for all! Massive output, mind performance Some express debugs: fw ctl zdebug drop grep <your filter> - to see drop reason on specific traffic Mind kernel buffer size 44

45 System Tools arp, route, netstat, ifconfig - to have -z X flag where X is VS number -z all prints info for all VSs ping -z... traceroute -Z... (Capital letter) 45

46 VS Policy To fetch the last installed policy: fw [-vs <vsid>] fetch local Fetching the last policy that failed to be installed fw fetchlocal -d $FWDIR/state/ tmp/fw1/ To unload policy: fw [-vs <vsid>] unloadlocal Unload policy for all VSs: fw vsx unloadall 46

47 VS configuration To fetch configuration: fw vsx fetch For specific VS: fw vsx fetchvs vs 2 To see NCS script for a specific VS: fw vsx showncs <vsid> 47

48 Other tips Double check topology If you cannot figure connectivity issues, especially some traffic degradation, suspect ClusterXL before others 48

49 Questions And Answers

50 Thank You For Your Time!

Check Point Troubleshooting and Debugging Tools for Faster Resolution January 24, 2006

Check Point Troubleshooting and Debugging Tools for Faster Resolution January 24, 2006 IMPORTANT Check Point recommends that customers stay up-to-date with the latest service packs, HFAs and versions of