ISE under magnifying glass. How to troubleshoot ISE

Transcription

1

2 ISE under magnifying glass. How to troubleshoot ISE Serhii Kucherenko, TAC CSE, CCIE #35182 Eugene Korneychuk, TAC CSE, CCIE #43253

3 Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot# 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

4 Abstract Nowadays Security is on the top of the mind of everyone. Such products as ISE are considered to be in the centre of Security Infrastructure. Being in TAC we can see significant growth in ISE cases opened by partners and customers. It is important not just to install it but also to maintain. ISE is a complex solution, it integrates with your DNS, AD, NGFW, Enterprise Network, etc. It is vital to quickly find where are the issues, what can cause them, and how to fix them. Session will focus on the structured approach on troubleshooting ISE. The session is intended for the engineers who have already deployed ISE and looking forward on how to get the best from troubleshooting ISE. Real life examples will be covered from TAC experience by means of demos and troubleshooting techniques. Hidden slides are here for your reference and future use Cisco and/or its affiliates. All rights reserved. Cisco Public 4

5 Welcome to the mystery world of ISE troubleshooting Stay tuned for next 90 minutes with TAC AAA engineers from Krakow TAC Eugene Korneychuk Customer Support Engineer AAA Team Krakow 6 years in TAC 12 years in Networking Serhii Kucherenko Customer Support Engineer AAA Team Krakow 3 years in TAC 11 years in Networking 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 5

6 Warning! Slavic Accent Ahead 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 6

7 What do you imagine might be some essential elements in successful troubleshooting? structured approach which is similar to a deductive method is one of the key elements in successful troubleshooting 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 7

8 Session Objectives Session will cover: ISE Troubleshooting; Theory of ISE and 802.1x operations; Complex examples. We want you to learn Session will not cover: Marketing; Roadmaps; All possible ISE features. and have fun 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8

9 You are at Level 3 Breakout Session Level 1 Sessions Level 2 Sessions Level 3 Sessions 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 9

10 Agenda Introduction to EXAMPLE IT Meet with John H. Watson, EXAMPLE ISE admin ISE authentication troubleshooting with John ISE deployment troubleshooting with John

11 Introduction to EXAMPLE IT EXAMPLE IT is a huge international IT company with offices all around the globe and head office in London; Network security is one of the major concerns for EXAMPLE top management; Identity networking is implemented based on Cisco ISE, EXAMPLE started from ISE 1.2, currently deployment is on 2.2 Patch 2; 2 PAN, 2 MNT and 2 PSNs are located in London, there are also 2 PSNs in APAC (Singapore) and 2 PSNs in Americas (New York) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 11

12 Meet with John H. Watson, EXAMPLE ISE admin John is experienced Network and IT administrator CCIE Security certification holder from 2012 For last couple of years responsible for ISE in EXAMPLE 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 12

13 EXAMPLE IT has completed site opening Brings a lot of income to the company Brings a lot of headache to the IT department New child domain was configured for the new site with the name of child.example.com. John worked hard with his colleague Sherlock from Active Directory team to make sure all users will be authenticating just fine. They tested multiple authentication scenarios and all tests were successful 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 13

14 9 am GMT 1st of February 2018 Sherlock calls Mrs. Hudson Site leader of newly opened location cannot login to the network Troubleshooting performed by AD team: The issue was not seen before; The issue is constant Mrs. Hudson can t login at all; 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 14

15 What else was checked? Error message Password was incorrect for the network: wired Though, Mrs. Hudson provided clear evidence (sticky note) of that she is entering the right password, it was changed few times from AD side to exclude client mistyping issue it didn t help 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 15

16 What now? Use MS-RPC for password based authentications Use Lookup for authentications without password, like EAP-TLS Authentication Result can be SUCCESS or FAILED 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 16

17 Test User Authentication Authentication Result (Failure) ISE Node, where authentication was performed Steps taken by ISE while authenticating the client 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 17

18 Test User Authentication Authentication Result (Success) Domain, which authenticated the client UPN and DN of the client Total number for Groups/Attributes retrieved for the client Time it took to: Perform authentication Fetch Groups/Attributes Useful when troubleshooting latency 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 18

19 Test User Authentication Groups Active Directory Groups, which the user belongs to SID values to confirm the real groups in AD 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 19

20 Test User Authentication Attributes Full list of User Attributes Attributes can be used in Authorization Policies 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 20

21 How about Detailed Authentication Report? 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 21

22 Detailed Authentication Report Overview Authentication Result Username sent by the client Mac address of the client 1 st Entry Policy Set selected (will be Default Policy Sets are not enabled 2 nd Entry Selected Authentication Policy within Policy Set 3 rd Entry Selected Identity Policy within selected Authentication Policy 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 22

23 Detailed Authentication Report Authentication Details Timestamp from the Radius Timestamp from ISE PSN, where authentication took place Troubleshooting section, very useful information, which contains the reason for the failure, root cause of it, and potential resolution. The first thing to look into if you are facing the authentication issues. Radius attribute which should show client s mac address, ip address in vpn usecase. IP address received from Radius Accounting (Framed IP address field) Identity Store used for authentication 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 23

24 Detailed Authentication Report Authentication Details Session Id created by the NAD, based on NAD IP Address, Session Count and Timestamp, used for CoA Authentication Method and Protocol. Can be EAP-TLS, PEAP, PAP_ASCII Service-Type is used to distinguish MAB (Call Check) from 802.1x (Framed) in case of Cisco NAD NAD information, Network Device, Device Type, Location as it is configured on ISE; NAS IPv4 Address, Port ID, Port Type as it is coming from Radius, can be used during troubleshooting to narrow down the issues, to specific NAD s And much more in the Other Attributes section 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 24

25 PEAP-MS-CHAPv2 process Supplicant Layer 2 Link EAP over LAN (EAPoL) EAPoL Start Authenticator Layer 3 Link RADIUS AuthC Server Layer 3 Link MS- RPC/Kerberos Request Identity Server Cert Validation Response Identity: mhudson TLS Client Hello TLS Change cipher spec EAP Proposal PEAP Response Identity: mhudson TLS Server Hello, Server Certificate RADIUS Access Request [AVP: EAP-Response:mhudson] RADIUS Access Challenge [AVP: EAP Proposal PEAP] RADIUS Access Request [AVP: TLS Client Hello] RADIUS Access Challenge [AVP: TLS Server Hello, Server Certificate] RADIUS Access Request [AVP: TLS ] Request Identity Check proposal LDAP mhudson lookup Response Challenge Success MS-RPC - mhudson Challenge MS-RPC - Response Success EAP Success RADIUS Access Accept [AVP: EAP-Response: mhudson] 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 25

26 Detailed Authentication Report Steps New Radius Session creation To select proper Policy, ISE needs to query specific PIP, if it exists in the conditions. PIP stands for Policy Information Point, Queried PIP is indication of ISE looking for attribute values to be provided. In this case ISE needs to match the rule based on Radius Flow Type (MAB vs 802.1x vs default) Matched Authentication Policy, note Dot1X it is the name of the Policy EAP protocol negotiation, please note EAP method is something being agreed between the client and the server, in this case Server proposed EAP- TLS, while client didn t agree with it (send NAK, proposing PEAP). Since PEAP is allowed on ISE (Allowed Protocols) PEAP was negotiated Cisco and/or its affiliates. All rights reserved. Cisco Public 26

27 Detailed Authentication Report Steps Phase 1. Following steps are showing regular TLS handshake during EAP- PEAP authentication, it is also called an outer method, within TLS tunnel credentials can be securely sent using such protocols as MS-CHAPv2, the process of sending credentials is also being referred as inner method Cisco and/or its affiliates. All rights reserved. Cisco Public 27

28 Detailed Authentication Report Steps Phase 2. Following steps are showing regular TLS handshake during EAP- PEAP authentication, it is also called an outer method, within TLS tunnel credentials can be securely sent using such protocols as MS-CHAPv2, the process of sending credentials is also being referred as inner method Cisco and/or its affiliates. All rights reserved. Cisco Public 28

29 Detailed Authentication Report Steps Identity Policy Selection. In this case Identity Sequence called All_User_ID_Stores was selected, which contains Internal Users and all AD users Since there is no mhudson in Internal users, it is expected not to find her ISE is doing authentication via Active Directory, the first step is Identity resolution, the goal is to check if there are any other accounts with the same username, goal is to get User principle name (UPN) of the user. It is used later on for password verification Cisco and/or its affiliates. All rights reserved. Cisco Public 29

30 Now What? But mhudson is part of child.example.com domain, why do we see 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 30

31 SAM vs UPN SAM or SAM-Account Name short name used for authentication. Like mhudson User First Name: May User Second Name: Hudson Logon name: mhudson UPN: UPN or User Principle Name long form of username used for authentication. Like or User First Name: Martha User Second Name: Hudson Logon name: mhudson UPN: 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 31

32 Wait a minute Shouldn t ISE take care of it? Multiple accounts found Authentication failed for mhudson@example.com Authentication succeeded for mhudson@child.example.com It Should! For password based protocols (PEAP) password is used to locate the correct user For EAP-TLS binary certificate comparison can be used to confirm that the user is unique 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 32

33 Thanks to John Be aware of: CSCvf ISE failing to resolve ambiguity for AD accounts Affected versions: 2.2 p4 2.1 p5 2.0 p5 Fixed in: 2.2 p5 2.3 p2 Quick workaround to use UPN for the users with ambiguity 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 33

34 11 am GMT 1st of February 2018 Helpdesk calls Helpdesk is reporting that most of the wireless phones can t connect to the network Troubleshooting performed by Helpdesk team: The issue seen since authentication was enabled; The issue is intermittent, phones can t connect 7 times out of 10; Issue is seen only with phones, computer authentication is working fine. Phones are using EAP-TLS authentication, computers PEAP authentication Cisco and/or its affiliates. All rights reserved. Cisco Public 34

35 What else? That s not it. IP Telephony infrastructure is managed by 3 rd party company which is not available at the moment. John has only ISE to troubleshoot phones connectivity issues 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 35

36 What else? ISE Reporting with combination of Authentication Summary report and Radius Authentication Report issue is narrowed down further: There is no pattern, first authentication can work and 5 subsequent fails, and vice versa; Issues are seen only on WLC s; There are only wireless phones connecting to WLC s All authentications are failing with the same error message during EAP-TLS negotiation: Invalid or unexpected EAP payload received Operations > Reports > Reports > Endpoints and Users > Authentication Summary/Radius Authentications 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 36

37 EAP-TLS process Supplicant Layer 2 Link EAP over LAN (EAPoL) EAPoL Start Authenticator Layer 3 Link RADIUS AuthC Server Layer 3 Link MS- RPC/Kerberos Request Identity Server Cert Validation Response Identity: TLS Client Hello TLS Change cipher spec Client Certificate EAP Proposal TLS TLS Server Hello, Server Certificate RADIUS Access Request [AVP: EAP-Response:] RADIUS Access Challenge [AVP: EAP Proposal TLS] RADIUS Access Request [AVP: TLS Client Hello] RADIUS Access Challenge [AVP: TLS Server Hello, Server Certificate] RADIUS Access Request [AVP: Client Certificate] Check proposal Client Cert Validation Success LDAP lookup EAP Success RADIUS Access Accept [AVP: EAP-Response:] 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 37

38 Who s fault is that? 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 38

39 What now? Select the ISE Node from which you would like to take the packet captures Select the interface from which you would like to take the packet captures Operations > Troubleshoot > Diagnostic Tools > TCP Dump In sniffer trace we trust 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 39

40 How to get the best from Wireshark, while troubleshooting AAA Question: How do I filter Radius packets Answer: Applying the basic filter radius will leave only radius packets displayed Note: if there is IP fragmentation in place, fragments will not be displayed Question: There are too many authentications, how do I filter only those coming from specific NAD Answer: The easiest way is to filter with ip.addr == <NAD ip adddress>, since the captures are taken on ISE itself, placing this filter will result in full communication between NAD and ISE Note: If the device is behind NAT, the better filter will be radius.nas_ip_address == <NAD ip adddress>, but it will filter only Access-Requests Question: How do I filter only authentications from specific endpoint Answer: The best way is to filter with radius.calling_station_id == mac address", however it will capture only Access-Requests 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 40

41 There are millions of packets, how do I navigate? Question: How do I correlate Access-Request to Access-Challenges and Access-Requests to Access-Accepts/Rejects? Answer: Access-Challenges, Access-Accepts and Access-Rejects are all responses to Access- Requests, they should have the same Radius Packet identifier as an Access-Request, it can be also used as a filter radius.id == <radius id> Question: How do I filter only Access-Accepts or only Access-Challenges or only Access-Accepts/Rejects Answer: Use the filter radius.code = <id> to filter Radius packets of particular type. 1 Access-Request, 2 Access-Accept, 3 Access-Reject, 11 Access- Challenge 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 41

42 There are millions of packets, how do I navigate? Question: What if I want to filter on the fly, based on attributes I found in the Radius Packet? Answer: It is possible as well. Find an attribute of your choice, right click, Apply as Filter, Selected. Operation above will result in the filter automatically applied to the packet capture you have opened Cisco and/or its affiliates. All rights reserved. Cisco Public 42

43 There are millions of packets, how do I navigate? Question: How can I quickly find which packet Access-Challenge is being replied to. Or where is the Accept for particular Access-Request Answer: In the Packet Details view, expand Radius. Text [The response to this request is in frame 21] indicates that the reply is there, when you click on it, wireshark will open the packet 21 itself 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 43

44 There are millions of packets, how do I navigate? Question: How can I check which ciphers client supports while establishing a tunnel. Answer: Supported Ciphers are sent within the Client Hello Message, which is part of the EAP message, EAP in encapsulated into Radius attribute 79. For the Access- Request in the Packet Details view, expand Radius, find and expand AVP (Attribute Value Pair) EAP- Message (79), expand EAP, expand SSL, expand Client Hello, and look into Cipher Suites. Note: selected Cipher Suite will be sent in Client Hello, use the same procedure to find it Cisco and/or its affiliates. All rights reserved. Cisco Public 44

45 There are millions of packets, how do I navigate? Question: How can I check which certificates are sent by the server and if the full chain is sent. Answer: Navigate to Access-Challenge which contains Server Hello, this message should also contain Certificate Chain. During EAP exchange ISE is sending full certificate chain to the client. From the screenshot above you can see that 3 certificates were sent, ISE certificate, Sub CA and CA Cisco and/or its affiliates. All rights reserved. Cisco Public 45

46 There are millions of packets, how do I navigate? Question: How can I check which certificates are sent by the client and if the full chain is sent. Answer: Navigate to Access-Request which contains Client Certificate. During EAP exchange Client is also sending full certificate chain to the server. From the screenshot above you can see that 3 certificates were sent, phone certificate, Sub CA and CA Cisco and/or its affiliates. All rights reserved. Cisco Public 46

47 There are millions of packets, how do I navigate? Question: How about fragmentation? How the packets being reassembled. Answer: Fragmentation can happen at multiple layers: IP RADIUS Attribute Value Pairs (AVP) EAP-TLS Screenshot shows IP Fragmentation Fragmentation can occur only between the Network Access Device (NAD) and the AAA server (IP/UDP/RADIUS used as a transport). This situation occurs when NAD (Cisco IOS switch) tries to send the RADIUS Request that contains the EAP payload, which is bigger then MTU of the interface. In this case Wireshark reassembled 2 IP fragments (size 1408 and size 333) into one of size Cisco and/or its affiliates. All rights reserved. Cisco Public 47

48 There are millions of packets, how do I navigate? Question: What is Radius Fragmentation? Answer: It is not the real fragmentation. As per RFC 2865, a single RADIUS attribute can have up to 253 bytes of data. Because of that, the EAP payload is always transmitted in multiple EAP-Message RADIUS attributes. On the screenshot EAP-Message is sent in 4 Segments Cisco and/or its affiliates. All rights reserved. Cisco Public 48

49 There are millions of packets, how do I navigate? Question: What is EAP Fragmentation? Answer: EAP Fragmentation happens in: RADIUS Access-Challenge sent by the AAA server, which carries the EAP-Request with the Secure Sockets Layer (SSL) Server Certificate with the whole chain. RADIUS Access-Request send by NAD, which carries the EAP-Response with the SSL Client Certificate with the whole chain. On the screenshot Wireshark reassembles 4 EAP- TLS Fragments (sizes 1002, 1002, 1002, 623 bytes) from 4 Access-Challenges into a single EAP-TLS Frame with size of 3629 bytes. The selected EAP fragment (one of those 4) indicates that there will be more EAP fragments coming, because More fragment flag is set to true Cisco and/or its affiliates. All rights reserved. Cisco Public 49

50 There are millions of packets, how do I navigate? Wireshark is capable of detecting the fragments, what is really convenient is that you don t need to expand the packet details. In the Packet List menu, next to the packet number you can see small circle, next to packets which are related to each other. On the screenshot the selected packet has #10. Packet #9 is marked because packet #10 is a response to the request in packet #9. Packets #4, #6, #8 are marked because together with packet #10 they contain whole EAP-TLS payload 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 50

51 Sniffer trace for the non working scenario 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 51

52 Dissecting EAP-TLS Packet #1 Access-Request Packet #3 Access-Request Packet #2 Access-Challenge 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 52

53 Dissecting EAP-TLS Packets #4, #6, #8, #10 Access-Challenges Packets #5, #7, #9 Access-Requests 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 53

54 Dissecting EAP-TLS Packets #11, #12 Access-Request Packets #13 Access-Reject 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 54

55 Dissecting EAP-TLS Not Working Scenario Working Scenario 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 55

56 Dissecting EAP-TLS Not Working Scenario But wait, shouldn t ISE first reply with Access-Challenge? RFC 5216 (EAP-TLS), section (fragmentation) states: The EAP peer MUST wait until it receives the EAP- Request before sending another fragment. Access-Reject is sent almost immediately after Access-Request 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 56

57 Dissecting EAP-TLS Error message: Invalid or unexpected EAP payload received In working scenario EAP-TLS payload reported is 4236 bytes, while in non working it is bytes, which is 1 Mb RFC 5216 (EAP-TLS), section (fragmentation) states: maximum acceptable message length might be 64 KB 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 57

58 Switches ISE Troubleshooting Wireless Controllers NTP Endpoints Load Balancers 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 58

59 Now it is time for John to get some rest Job well done Complicated issues have been identified So its time for a coffee But 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 59

60 Phone always rings unexpectedly Mycroft called from Singapore office to inform John that today night one of the PSN servers become unavailable, After fast troubleshooting faulty hard drive have been identified New server is already pre-configured and ready to be joined to the deployment 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 60

61 Page which is known to any ISE administrator FQDN of the node which needs to be joined SuperAdmin username User password 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 61

62 Some other well known thing For majority of the common problems nice error messages are presented Problem Solution But this is not todays case Cisco and/or its affiliates. All rights reserved. Cisco Public 62

63 John have tried to join new node to the deployment Initially everything was looking good But after some time 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 63

64 What is the goal? At this point troubleshooting may look very challenging During next 40 minutes we will learn how exactly Join process works in depth so this knowledge can be applied to help John and in any other real-life scenarios Cisco and/or its affiliates. All rights reserved. Cisco Public 64

65 To help John we need to understand process in depth Join process in a nutshell - a process of authenticated configuration database replication from Primary Policy Administration (PAN) to the node which needs to be joined. PSN2 is PAN is TLS Connection DNS communication Who is PSN2 Who is PAN 7 Node internal operations Primary PAN Standalone Node Client Hello Server Hello 3 4 Certificate Validation 5 Credentials, PAN info 6 User Authentication/ Authorization Standalone node info Cisco and/or its affiliates. All rights reserved. Cisco Public 65

66 Join operation continue TLS Connection DNS communication Node internal operations Primary PAN 10 9 Standalone Node Node details saved to DB Save Certs to DB 11 Certificates Download* Certs saved to NSSDB FW rules updated to permit incoming J-Group Connection 12 Shared Certificates replication/fw rules update** FW rules updated according to new node role * - Only certificates are downloaded, private keys stay on the node ** - Shared certificates are any System wildcard certificates (Ex: Portal) plus PAN SAML certificate 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 66

67 Be patient, join almost done TLS Connection DNS communication Node internal operations J-Groups TCP Primary PAN DB export up to current SCN* 15 DB Changes Publishing DB dump destroyed DMP file transferred along with check sum Client Hello Server Hello 21 DB dump destroyed DB import process** PSN2 DB Update J-Group Process Update Sync Status/Request Sequence numbers starting from SCN*** Missing DB changes Certificate Validation J-Group Process 19 PAN is Who is PAN 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 67

68 Some clarification * - Current SCN (System Change Number) ISE database is dynamic in nature. PAN DB is constantly updated by other nodes when new endpoints (EPs) are seen on the network or when EPs ownership change occurs. Due to this fact it is impossible to stop DB on PAN, instead copy of DB is created up to certain change (SCN), after successful join newly added node needs to request all changes which have been made after SCN. ** - DB import process before import started node stops all services except DB and after it import operation performed. After successful import some additional steps are executed like local Redis (Profiler Database) reset. *** - Sync Status Update after import is finished new node must contact new PAN over J-Group and inform that it that DB import has been complotted. At this stage node needs to resolve PAN FQDN and as well verify PAN certificate Cisco and/or its affiliates. All rights reserved. Cisco Public 68

69 Main troubleshooting tools - Captures Packet captures ISE built in capture can be used. Packets between PAN and nodes for which join operation is failing need to be captured. What to look for: - Fatal TLS errors on port 443/12001, - Certificates and certificate chains on port 443/ Un-answered SYNs on port Cisco and/or its affiliates. All rights reserved. Cisco Public 69

70 Collecting TCP dump on ISE ISE has built in packet capture utility which can be used from GUI. Operations > Troubleshoot > Diagnostic Tools > TCP Dump Start capture Choose ISE node whew capture should be collected Select interface for capture. Note: ISE supports deployment only over G0 3 Place filet if needed 4 Choose capture format. RAW date is recommended as it can be later opened by Wireshark or similar utility Download capture after it been stopped Cisco and/or its affiliates. All rights reserved. Cisco Public 70

71 Example of fatal TLS error In current example PAN IP , Standalone Fatal TLS error sent by PAN Standalone Node replies with Server hello which includes Admin certificate chain PAN sends TLS client hello 4 Failure reason After join button is pressed by Admin PAN tries to resolve FQDN of standalone node and establish TLS connection to it over port 443. Standalone node must reply with server hello which includes Full chain of Admin certificate. In case of any certificate/chain validation problems PAN indicates it with fatal TLS error Cisco and/or its affiliates. All rights reserved. Cisco Public 71

72 Extracting certificates from the packet capture Some times It may be needed to confirm which certificates are sent on the packet level. Certificates can be extracted from "Server Hello" packet with standard Wireshark feature "Export packet bytes. Certificates can be later validated by OpenSSL $ openssl verify -verbose -CAfile CA.pem PSN.crt 2 Navigate to certificates in Server Hello and choose Export Packet Bytes 1 Save file as.crt Check certificate in Windows. To confirm trust chain issues of the certificate needs to be installed in OS trusted store Cisco and/or its affiliates. All rights reserved. Cisco Public 72

73 J-Group communications Below is example of communication over port (J-groups) between PSN ( ) and PAN ( ). It is ultimate responsibility of Any Secondary node to establish connection to the Primary PAN after DB export: This connection should happen after Primary PAN name resolution on the Secondary node. In case if you are not able to see any connections attempt to PAN IP over port TCP on the secondary node this could indicate DNS issue Cisco and/or its affiliates. All rights reserved. Cisco Public 73

74 J-Group Decode As Wireshark does not recognize TCP as an SSL by default. We can change this with standard Wireshark feature Decode As Select J-Group destination port Instruct Wireshark which protocol needs to be used for Decode Choose packet in the flow and apply Decode As 4 Save your selection 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 74

75 Deployment troubleshooting intro to ISE log types In general ISE logs can be divided in two main categories: Application logs related to ISE applications like (guest, profiler, replication, runtime, ise-psc, ). Available through GUI, CLI, included into the support bundle, System logs related to underlying OS and generic services (ADE, dmesg, messages, ). Available through CLI and support bundle only 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 75

76 ISE Log analysis recommendations CLI rare case as filtering is limited, plus it's not really convenient to check huge amount of data in terminal. Normally used by TAC/Customer when all other mechanism are not available. Individual log file analysis Analysis of individual log files downloaded from ISE GUI. Normally this method includes usage of specialized text editors like Notepad++/Sublime and so one. Most convenient method for the end user. Support bundle most convenient method for TAC which includes individual files analysis plus some internal automation available in TAC Cisco and/or its affiliates. All rights reserved. Cisco Public 76

77 Deployment troubleshooting log files For effective troubleshooting of join operations three log files can be used. In 90% of the cases there is no need to enable any debugs. Default logging levels are enough. - ise-psc.log main ISE log file, in context of join operation troubleshooting in it we can find information about join process, like: DNS queries, certificate validation, certificates replication, sync status control - ADE.log underlying OS log file. Contains information about DB import/export process along with services start/stop - replication.log can be used to troubleshoot issues after DB import or problems with DMP file transfer * - please see slides for more details * * * 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 77

78 Main troubleshooting tools What to choose and when? Short answer both captures and logs should be used Some recommendations: If process fails before step 9 and GUI error is not clear enough or recommendations has been followed already - Packet capture may be a first point to start In case if step 9 can be passed then logs are the first option. Packet capture can be used later for J-Group verification Cisco and/or its affiliates. All rights reserved. Cisco Public 78

79 ISE Logs how to find in CLI? ISE logs can be viewed in CLI with the following commands: pan1cl/admin# show logging application <file name> This command can be used to view and filter ise-psc log files and replication log files pan1cl/admin# show logging application ise-psc.log i ERROR Display all messages with severity ERROR from current ise-psc log file pan1cl/admin# show logging system ade/<file name> This command can be used to view and filter ADE log files pan1cl/admin# show logging system ade/ade.log i sync_export.sh Display all messages related to DB export script 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 79

80 Navigate through the log files in CLI Find and filter specific log file 1 pan1cl/admin# show logging application i ise-psc Dec :38:00 ise-psc.log Dec :59:18 ise-psc.log Dec :59:16 ise-psc.log Dec :59:31 ise-psc.log pan1cl/admin# show logging application ise-psc.log Tailing of specific file, analog of "tail -f" i apac-psn2cl 4 Display All ISE application log files Filer output by displaying only files which contains ise-psc in name Display specific log file Limit output to only strings which contains - apac-psn2cl apac-psn2cl/admin# show logging application ise-psc.log tail :05:35,195 INFO [Thread-69][] api.services.persistance.dao.distri butiondao -::::- In DAO getrepository method for HostConfig Type: PRIMARY 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 80

81 ISE Logs how to find in GUI? Operations > Troubleshoot > Download logs 2 1 Navigate to Debug Logs Tab Select node from which files needs to be downloaded 3 Click on the files which needs to be downloaded Open downloaded files locally Cisco and/or its affiliates. All rights reserved. Cisco Public 81

82 ISE Logs Support bundle collection Operations > Troubleshoot > Select node from which bundle needs to be downloaded Download logs DB dmp file, rarely needed, TAC normally uses configuration backup ISE application log files Local Radius logs core files needed for ISE crash analysis Data specific to MNT node Underlying OS log files 10 XML file with policies configuration Generate bundle 11 Without defined time range all data will be attached, this can make size of support bundle significant Public key only Cisco TAC can decrypt bundle Shred key bundle can be decrypted by GPG utility 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 82

83 Coming back to our issue Clear problem definition is a key element to successful troubleshooting Newly installed ISE node apac-psn2cl.example.com has failed into the state Registration or Sync failed around 20 minutes after successful registration. With this information we can: - Eliminate steps from 1 till 9 as those which passed successfully - During logs analysis focus on steps which are left From which log file John should start? ise-psc.log ADE.log replication.log * - please see slides for more details * * * 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 83

84 Now we know where to start, but Smallest ISE log file is couple megabytes in size, just scrolling through it will take a lot of time. We need to have some indicators before analysis can be started. What possible indicators could you imagine? Time range starting from registration start till failure for all log files on both PAN and Secondary Severity ERROR or WARN plus hostname of Secondary on PAN or PAN on secondary for ise-psc and replication Filter by sync_export.sh on PAN and sync_import.sh on Secondary for ADE 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 84

85 Let s try to apply this approach Display only lines which are having WARN or ERROR followed by hostname of a secondary node pan1cl/admin# show logging application ise-psc.log i "WARN\ ERROR.*apac-psn2cl" :45:11,494 ERROR [admin-http-pool4][] 3 cpm.infrastructure.deployment.client.deploymentregistrationclient -:admin:apac-psn2cl:registernode:- An error occurred while importing deployment shared system certificates to the registering node 4 Actual error Output filters Now we know that some certificate is causing the issue, but how to fix it? 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 85

86 Should we look at the certificates? PAN Secondary For this case it may be possible to figure out straight from here, but in general it is always better to know for sure which certificate is causing the problem 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 86

87 I need the name We know exact problematic step now (13th), logs on PAN are not giving much so let's check on Secondary How we can filter them? Severity ERROR can be good point to start As well we can search for lines with ERROR and "certificate" Only lines which are having ERROR followed by certificate apac-psn2cl/admin# sh logging application ise-psc.log i "ERROR.*certificate :45:11,489 ERROR [admin-http-pool8][] infrastructure.certmgmt.service.impl.localcertificateserviceimpl -::::- Another certificate with Friendly Name 'ISE-wildcard' already exists. Friendly Names must be unique. 2 Actual error which contains exact certificate name and error reason 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 87

88 Ah, here they are PAN Secondary Solution Change the friendly name of admin certification on the Standalone node 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 88

89 Changing the certificate friendly name 1 Change friendly name to something else 2 Save your changes Cisco and/or its affiliates. All rights reserved. Cisco Public 89

90 Now it should work right? Again initially all looks good But even after couple of hours node still not in sync And finally 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 90

91 Lets go further DB export As now we should pass step 13 th successfully we can start DB import/export validation This can be done either by downloading support bundle from both Primary and Standalone, or by validation the logs through CLI DB export on PAN Beginning and end of db export process. Time in between indicates how long it took PAN to create DB dump* Limit output to only lines which contain script name Sync identifier, can be used to filter ise-psc on Primary PAN and ADE on secondary node. Unique value generated for every sync process pan1cl/admin# show logging system ade/ade.log i sync_export.sh T15:42: :00 pan1cl ADE-SERVICE[1119]: [20627]:[info] utils: vsh_pipe.c[117] [admin]: pattern after filtering dashes: sync_export.sh T15:45: :00 pan1cl logger: info:[sync_export.sh] Starting the datadump export for sync identifier f603da90-dcf7-11e d T15:45: :00 pan1cl logger: info:[sync_export.sh] SCN retrieved for sync identifier f603da90-dcf7-11e d now starting export for CSCN and host pan1cl T15:52: :00 pan1cl logger: info:[sync_export.sh] Export success for sync identifier f603da90-dcf7-11e d * - huge time gap between export start and finish may indicate issues with DB size on PAN 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 91

92 Be aware of CSCuv ISE 1.4 DB growing large due to EDF_DB_LOG EDF_DB_LOG is one of the ISE Oracle tables. When table size is too big it is not just affect backup operations but as well it can affect any DB dump operations like Registering of new node or Manual Sync for existing nodes Fixed releases 1.4 (P8); 2.0 (P4); (P1); 2.1; and upper releases CSCvc EDF_DB_LOG Still Growing in Size May Cause Large Backups or Backup Failures Previous defect has not been fully fixed and DB dump failures have been still seen due to EDF_DB_LOG table size Fixed releases 2.1 (P4); 2.2 (P2); Cisco and/or its affiliates. All rights reserved. Cisco Public 92

93 Secondary node DB import On secondary node we can filter ADE log with the sync_import.sh or sync ID taken from the primary node DB import process finished successfully DB import process started apac-psn2cl/admin# show logging system ade/ade.log i sync_import.sh T14:45: :00 apac-psn2cl admin: info:[patchinstall.sh] Taking backup of patch file: /opt/cscocpm/bin/sync_import.sh T14:46: :00 apac-psn2cl admin: info:[patchinstall.sh] Updating patched file: /storeddata/installing/ /filesystem/opt/cscocpm/bin/sync_import.sh T15:42: :00 apac-psn2cl ADE-SERVICE[1116]: [13089]:[info] utils: vsh_pipe.c[117] [admin]: pattern after filtering dashes: sync_import.sh T15:53: :00 apac-psn2cl logger: info:[sync_import.sh] Dropping the check constraints T15:53: :00 apac-psn2cl logger: info:[sync_import.sh] f603da90-dcf7-11e d2955: Starting the datadump import T15:56: :00 apac-psn2cl logger: info:[application:operation:sync_import.sh] Elapsed: 00:00: T15:56: :00 apac-psn2cl logger: info:[application:operation:sync_import.sh] f603da90-2 dcf7-11e d2955: DB Sync Import successfully completed 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 93

94 What to do next From ADE logs we can confirm that first 19 th steps passed successfully. Now it is time to focus on replication log. We can filter current replication.log on secondary node to see what is going on there. In the logs we can see same WARN messages repeated again and again 1 apac-psn2cl/admin# sh logging application replication.log tail i "WARN\ ERROR.*pan1cl" :29:48,945 WARN [Timer-3,ISERepCluster-CljFl,apac-psn2cl-27946][] org.jgroups.protocols.tunnel -::::- failed reconnecting stub to GR at pan1cl.example.com/ :12001: java. lang.exception: Could not connect to pan1cl.example.com/ : :29:52,909 WARN [pool-43-thread-1][] cisco.cpm.deployment.replication.clientsyncstatusupdaterimpl -:: dd88-11e d2955:FullSync:- PAP failed to acknowledge the sync status...will try again later 3 Display only lines which are having WARN or ERROR followed by hostname of a PAN node Actual problem Result We will try to connect later 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 94

95 Verification Logs clearly indicate that connection attempt to pan1cl.example.com ( ) over TCP port is failing. Let us check the configuration pan1cl/admin# sh running-config Generating configuration...! hostname pan1cl! ip domain-name example.com! ipv6 enable! interface GigabitEthernet 0 ip address ipv6 address autoconfig ipv6 enable But wait a second PAN has an IP Why secondary tries to connect to ? apac-psn2cl/admin# nslookup pan1cl Trying "pan1cl.example.com" pan1cl.example.com IN A Received 52 bytes from #53 in 3 ms Solution Correct PAN A record on DNS server 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 95

96 And finally Detailed authentication report, Thor hammer in the world of ISE troubleshooting Ambiguity - can happen good that we know how to fix it In Packet capture we trust! ISE Deployment grow step by step Dear ISE we are not afraid of your logs anymore 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 96

97 Score of this session is really important for us In case if you really enjoyed this session and content was useful 5 is the right score to give Otherwise please leave your comment so we can make session even better next time 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 97

98 Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot# 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

99 Please complete your Online Session Evaluations after each session Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt All surveys can be completed via the Cisco Live Mobile App or the Communication Stations Complete Your Online Session Evaluation Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at Cisco and/or its affiliates. All rights reserved. Cisco Public

100 Thank you

101