How Secured Search Works in IBM Watson Content Analytics

Transcription

1 How Secured Search Works in IBM Watson Content Analytics 1

2 How secured search works in IBM Watson Explorer Analytical Component 3 Introducing some concepts... 3 Document level security... 3 How is that achieved?... 3 What is Pre-filtering and Post-filtering?... 3 Document ACL information stored in index... 4 Figure 1. Enable Document Level Security... 4 User security context attached to the search request... 5 In pre-filtering, USC is compared with ACL information stored in index to filter search results... 6 Configure Identify Management Component... 6 Figure 2. Configure Identity Management... 7 My Profile... 7 Figure 3. My Profile setting... 7 Skipping My Profile... 8 Application SSO vs Secure Search SSO... 8 Application SSO... 8 Secure search SSO... 8 In crawler configuration, enable SSO:... 8 Secure Search Scenarios... 9 Scenario 1. No Application SSO, in crawler configuration SSO is not enable either... 9 What happens when a user logs into application? Figure 3. ssotoken Scenario 2 Enabled Application SSO, in crawler configuration, SSO is enabled. My Profile should be able to be skipped in this scenario What happens when a user launch search application? Troubleshooting What to check if search result is not expected? What logs do we need? Clear IMC cache to force discovery session to connect to data source server to get latest user information Identify whether the problem happens in pre-filtering or post-filtering Continue secured search even when crawler server is down Calling discovery command to connect to data source server directly How to check document ACL stored in index? Resources

3 How secured search works in IBM Watson Explorer Analytical Component In Watson Explorer Analytical Component, or Watson Content Analytics, you can enable application login and enable document level security. Once properly configured, search users will perform search to only return documents that they are authorized to see. Introducing some concepts Document level security Document-level access control ensures that the search results contain only documents that the user who submitted the search request is authorized to see. Refer to IBM Watson Explorer knowledge center on this topic for more. Click here How is that achieved? Configure document-level security options. If security was enabled when the collection is created, the crawler can associate security data with documents in the index. This data enables enterprise search applications (or content analytics collection to enforce access controls based on the stored access control lists or security tokens. Refer to IBM Watson Explorer knowledge center on this topic for more. Click here What is Pre-filtering and Post-filtering? There are two distinct approaches to filtering documents to ensure that search results contain only the documents that the user who submitted the search request is authorized to view. The first approach is to replicate the document's source access control lists (ACLs) at crawl time into the index and to rely on the search engine to compare user credentials to the replicated document ACLs. Pre-filtering the documents, and controlling which documents are added to the index, results in the best performance. However, it is difficult to model all of the security policies of the various back-end sources in the index and implement comparison logic in a uniform way. This approach is also not as responsive to any changes that might occur in the source ACLs. The second approach is to post-filter documents in the result set by consulting the back-end sources for current security data. This approach allows the contributing back-end sources to be the final arbiters of the documents returned to the user, and ensures that the result set reflects current access controls. However, this approach results in degraded search performance because it requires that connections exist with all of the back-end sources. If a source is not accessible, then links to documents must be filtered out of the result set along with documents that the user is not authorized to view. 3

4 Please refer to more details in Knowledge Center, click here. In this article, I will use the term pre-filtering and post-filtering for these two approaches. Document ACL information stored in index To have document ACL information stored in index, you must configure document level security for the crawler. For example, when you create Windows File System crawler, you can configure document level security here: Figure 1. Enable Document Level Security The first check box Validate current credentials during query processing is about Postfiltering. Options for indexing access controls is about pre-filtering. During pre-filtering, users security information is compared with the ACL stored in index. The options here specify what kind of security information is stored in ACL for documents. An example from Notes documents In Domino security, there are 3 levels. Server level security, database level security, and document level security. As you can see from DumpIndex output, we mark them by [1 0]-Server level, [2 0]-Database level, [3 0] Document level. As you can see from the dumpindex output, the "-default-" group already have [1 0] and [3 0] access to all documents, so it really depends on whether the user (or any groups it belongs) to have [2 0] access to that document. If the user has [2 0] access, then the user can see this document. Here is an example for a Notes documents crawled by NRPC Notes crawler: ID: 119 4

5 URI: domino://watson.test.ibm.com/492576f a5/archive%5ctest55%6b nsf//f22d00f64f C Security: on $sec$watson!!mydomain!!notes: [-default- 1 0] [group01 2 0] [cn=user01 testgroup/ou=o/o=ricoh 2 0] [group02 2 0] [cn=user02/o=mygroup 2 0] [group03 2 0] [group04 2 0] [cn=user02/ou=r/o=watson 2 0] [*/ou=r/o=watson 2 0] [cn=user03 supports/ou=o/o=watson 2 0] [-default- 3 0] An example from Windows shared files crawled by a Windows File System crawler: C:\Users\Administrator.WATSONSUPPORT>dumpindex SecureSearch --uri --security Found 4 documents: ID: 0 URI: file:///c:/data/test/wexca%2520v11.0%2520performance%2520tuning%2520guide.pdf Security: on $sec$watsonsupport.adl.ibm.com: [system 1 0] [wexcrawler01 1 0] [user01 1 0] [system 2 0] [wexcrawler01 2 0] [user01 2 0] ID: 1 URI: file:///c:/data/test/test.html Security: on $sec$watsonsupport.adl.ibm.com: [system 1 0] [wexcrawler01 1 0] [user01 1 0] [system 2 0] [wexcrawler01 2 0] [user01 2 0] ID: 2 URI: file:////shanghai/test/a.txt Security: on $sec$watsonsupport.adl.ibm.com: [system 1 0] [administrator 1 0] [wexcrawler ] [user01 1 0] [system 2 0] [administrator 2 0] [wexcrawler01 2 0] [user01 2 0] ID: 3 URI: file:////shanghai/test/dir1/b.txt Security: on $sec$watsonsupport.adl.ibm.com: [system 1 0] [administrator 1 0] [wexcrawler ] [user01 1 0] [system 2 0] [administrator 2 0] [wexcrawler01 2 0] [user01 2 0] The user (or any group the user belongs to) needs to have both [1 0] and [2 0] in the security information, to be able to see this document in search result. User security context attached to the search request What is user security context (USC)? As explained above, to perform a secured search, search server must let data source server know who is performing the search so that it can bring back only the documents that the users can access. For that purpose, User Security Context (USC) string is attached to each search request to identify who the user is for each data source server that is being searched for. USC must account for the different identifiers that a single user must use to access the various back-end data source server. USC can be attached either by custom code or utilizing the Identity Management Component (IMC) the product is provided with. How does USC look like? The following is an example of USC for Windows File System secured crawl space, after decoding. <identities id="dxnlcjax"><ssotoken>8vo3zkzfwaecvucxkmi27pyw55ygdmhqhmlfps9dk24hv0drpf7yan5th8qbfggng5 XuSVgEDAE/WX/gZLNPX/7n2C+wl1xj6TR9pwlvnlmIbSa7RwGYBqwylza6pp8ka5umXHlEV7XwIF2S4Csk/TvFEJgd uhjgtn7hrfywq7nrcvajmkprfkmmszy2an1r1onmfbhewodbtecmgt2/nq3utchgxohoalz8l9q3bgfa4hjfugxnps kcmv8d8wgrwnz9gjfy82oy3qhzcr9ozxybluvnkgtoy1i1eesbkwv2smydkar0jdojck2mn0hewjnquf0gfpwyje2d yks32mqcjlbddocxf1u5vk5/ag7tmajr5ukaoeobnbw3jyhs</ssotoken><cookies><cookie name=" 5

6 LtpaToken2 ">OHZvM3pLWkZXQUVDVnVjWGttSTI3UFl3NTVZR0RtaHFITWxGUHM5RGsyNGh2MERScEY3WWFuNXRIOFFiZkdHTkc1 WHVTVmdFREFFL1dYL2daTE5QWC83bjJDK3dsMXhqNlRSOXB3bHZubG1JYlNhN1J3R1lCcXd5bHphNnBwOGthNXVtWE hsrvy3whdjrjjtnenzay9udkzfsmdkdwhqz3run0hyrnl3ctduukn2ywpta3byrktnbvnaetjhtjfsmu9utuzcsevx b0rcvevdbud0mi9uctnvvenir3hvse9bbho4tdlrm2jnrme0sgpgvudybnbta2nndjheohdncnduwjlhsmzzodjvwt NxaFpjcjlPelhZYmx1dm5LZ3RveTFJMUVlc0Jrd1Yyc01ZRGthUjBKRG9qQ2syTU4waGVXSm5RVWYwR2Zwd3lqZTJE euttmzjtcunqbejere9degyxvtvwazuvyuc3dg1bsni1vwthb0vpqm5idznqwuht</cookie><cookie name=" JSESSIONID">MDAwMExPaTRDbVZZRnVJZGFkV0J5Z1l2dWhpOjZlOTE5MDU1LWNjYjQtNGU2OS04ZWJmLWMwNTkyOG NiZTZlNA==</cookie></cookies <identities id="user01"> <identity id="watsonsupport.adl.ibm.com"> <username>user01</username> <type>winfs</type> <password encrypt="yes">ftbufzxczpurvwxiqsg2ww==</password> <groups> <group id="domain Users"/> <group id="everyone"/> <group id="users"/> <group id="interactive"/> <group id="console LOGON"/> <group id="authenticated Users"/> <group id="this Organization"/> <group id="wcausers"/> <group id="ntlm Authentication"/> </groups> <properties> <property name="valid">true</property> <property name="connectionid"> </property> <property name="username">user01</property> <property name="creationdate"> </property> <property name="enable">true</property> <property name="aclvl">2</property> <property name="crwid">securesearch.win_64181</property> <property name="spaceid">t </property> <property name="ssoenabled">false</property> </properties> </identity> </identities> In pre-filtering, USC is compared with ACL information stored in index to filter search results User and user group information from USC will be used when pre-filtering happens. It compares the USC with the ACL information stored in index. For example, ID: 0 URI: file:///c:/data/test/wexca%2520v11.0%2520performance%2520tuning%2520guide.pdf Security: on $sec$watsonsupport.adl.ibm.com: [system 1 0] [wexcrawler01 1 0] [user01 1 0] [system 2 0] [wexcrawler01 2 0] [user01 2 0] Here it means user01 can access both the directory and the file itself. In case user01 does not appear on this list, we will find out all the groups that user01 belongs to, and see if any of the groups can see to this document that to determine whether user01 can see this document in search result. Configure Identify Management Component To manage user credentials, you need to configure Identity Management Component. In customized application, you can also choose to attach USC to search request by your own custom code. It is out of our discussion in this article. 6

7 Figure 2. Configure Identity Management For details please refer to Knowledge Center. My Profile When a user logs in to Enterprise Search application, or Content Analytics application, on the upper right hand corner, there is My Profile. From here the user can provide user name and password to each data source, to identify himself. Figure 3. My Profile setting For details please refer to this page in Knowledge Center. 7

8 Skipping My Profile With secure search SSO, users can run secure searches without having to map credentials in the My Profile dialog of the content analytics application or enterprise search application. Application SSO vs Secure Search SSO Application SSO With application SSO, users can log in one time and be authenticated across several systems. For example, users can log in to the content analytics application or enterprise search application and then open links to see documents in the results without being prompted to log in to the data source servers. This type of SSO authentication relies on Lightweight Third-Party Authentication (LTPA tokens). Secure search SSO With secure search SSO, users can enter secure searches without having to map credentials in the My Profile dialog of the content analytics miner or enterprise search application. Secure search, which is also known as document-level security, means that users see only the documents that they are authorized to see. The My Profile dialog maps the user's login identity to the user's credentials on various data source servers, which typically use data source-specific credential formats. Application SSO is a prerequisite for implementing secure search SSO. If you remove the requirement to map credentials in the My Profile dialog, users are still prompted to log in when they access results unless you also implement application SSO. If the following settings are configured to support SSO authentication, secure search SSO is in effect. The application stops requiring users to map their login credentials to data source credentials in the My Profile dialog. When you configure the identity management component in the Security view, select the check box for each crawler type that you want to enable for SSO support. When you configure security settings for an individual crawler, enable SSO support. For more details please refer to following topics in knowledge centre: Support for single sign-on authentication Identity management and SSO authentication In crawler configuration, enable SSO: Single sign-on security (SSO) 8

9 If security was enabled for the collection when the collection was created, and if you specify that you want to use the identity management component (you specify this option in the Security view), you can specify whether you want to use single sign-on (SSO) security to control access to documents: Enabled for SSO Select this option if the server to be crawled is protected by a product that provides SSO security and you want to use SSO security to control access to the documents that this crawler crawls. Later, when you configure document-level security options for a data source, you can specify that you want to validate the user's current credentials. The system will then use SSO methods to authenticate users when they search the collection. Not enabled for SSO Select this option if SSO security is not available on the server to be crawled or if you do not want to use SSO security to authenticate users when they search the collection. Enable SSO option is available to these types of crawlers: Content Integrator Domino Document Manager Notes Quickr for Domino IBM Connections SharePoint FileNet P8 Secure Search Scenarios Scenario 1. No Application SSO, in crawler configuration SSO is not enable either Use IMC to manage user credentials As mentioned earlier, you can also write your own code to attach USC (user security credential information) to each search request in your customized search application, but it will not be covered in this article. After the user log in, click on My Profile on the upper right corner of the application to open it, you can enter username/password to access each secured data source. This is called user credential information, it will be stored in IMC, so that later when user logs in search application, he does not need to do the mapping each time in My Profile. The user credential information is stored in IMC cache (an internal database). Please note that user credentials are still validated against the data source server each time user logs in, or click Apply button on the My Profile page. Using the IMC cache does not mean user validation 9

10 (to each secured data source) is skipped, it only means the mapping information is stored in My Profile, so that you do not need to enter it in My Profile each time after you log in. What happens when a user logs into application? 1. A user logs in search application, username/password is authenticated against the LDAP server 2. Check which collections are enabled for search currently. 3. In each collection enabled for search, find out secured crawl spaces. 4. For each secured crawl space (configured in crawler settings), get user credentials respectively. 5. If IMC is used to manage user credentials, read IMC cache database. 6. If the user credential is still valid, use the user information (including groups extracted for this user) stored in IMC cache to construct USC. Then connect to data source server to validate the user id and password. If validation passes, continue to next step. (Note: in case of Notes NRPC crawler, only checks if the user id exists in Domino server s names.nsf. And group information will not be stored in IMC, instead it will be obtained in next step. ) 7. If the user credentials have expired, or the Apply button on My Profile dialogue box is clicked, the user credential data needs to be refreshed. In this case, it connects to data source server to validate the user, and to extract group information for this user, then update the IMC cache with latest user information. If user credentials have not expired yet, will read user information from IMC cache database. 8. Repeat this process for all crawl spaces matched to the login user, until a complete USC for all relevant data source servers is constructed for this user. 9. With this USC attached to the search request, search server will be able to return search results only matching to this user, either by pre-filtering or post-filtering. The following is taken from search server trace when data source is crawled by Windows File System crawler [FINER] Retrieved the Searchable object from the local server. LocalServerItem host:localhost port:57786 [FINE] getactivesecurespaces - collection: SecureSearch :search1 : is enabled for search check vault.xml, find spaces that matched to this collection ID [FINER] 2 spaces are recorded in the vault information. [FINER] 2 spaces are match to vault information. Get secured data sources according to above space information, retrieve by each crawler: [FINE] getsecureddatasources - Secure SpaceInfo : [FINER] createsecureddatasource - entered [FINER] createsecureddatasource - adding type winfs id watsonsupport.adl.ibm.com domainname watsonsupport.adl.ibm.com [FINER] createsecureddatasource - conid spid t [FINER] RETURN [FINER] Number of detected secured data sources: 1. IDs are watsonsupport.adl.ibm.com [INFO] getimcproperties - config file path: C:\IBM\es\nodeinfo\es.cfg 10

11 [INFO] getimcproperties - sso file path: C:\IBM\es\esadmin\config\sso.properties As Windows File System crawler is not supported with SSO, so you will find return SSO disabled in the log: [INFO] issourcetypessoenabled - returning SSO disabled [FINE] getsecureddatasources entered [INFO] getusercredentials entered [INFO] getusercredentials - database connection is null, initializing connection to cloudscape connect to IMC cache db [FINER] createsecureddatasource - entered [FINER] createsecureddatasource - adding type winfs id watsonsupport.adl.ibm.com domainname watsonsupport.adl.ibm.com [FINER] Number of detected secured data sources: 1. IDs are watsonsupport.adl.ibm.com [INFO] issourcetypessoenabled - returning SSO disabled [FINER] Number of filterred sources is: 1 [FINER] InternalBaseAction successfully serialized the given result as com.ibm.es.api.srv.actions.dockresponse [FINER] RETURN Get USC from IMC cache database [FINE] getusercredentials - credentials: <identities (omitted) /identities> Here <ssotoken> value is taken from browser, Figure 3. ssotoken This value was encoded and stored into cookie LtpaToken2 part in above USC information. JSESSIONID part was also taken from browser, then was encoded and stored here. After decoding the information, it will look like [FINE] getusercredentials - credentials: <identities id="dxnlcjax"><ssotoken>8vo3zkzfwaecvucxkmi27pyw55ygdmhqhmlfps9dk24hv0drpf7yan5th8qbfggng5 XuSVgEDAE/WX/gZLNPX/7n2C+wl1xj6TR9pwlvnlmIbSa7RwGYBqwylza6pp8ka5umXHlEV7XwIF2S4Csk/TvFEJgd uhjgtn7hrfywq7nrcvajmkprfkmmszy2an1r1onmfbhewodbtecmgt2/nq3utchgxohoalz8l9q3bgfa4hjfugxnps kcmv8d8wgrwnz9gjfy82oy3qhzcr9ozxybluvnkgtoy1i1eesbkwv2smydkar0jdojck2mn0hewjnquf0gfpwyje2d yks32mqcjlbddocxf1u5vk5/ag7tmajr5ukaoeobnbw3jyhs</ssotoken><cookies><cookie name=" LtpaToken2 ">OHZvM3pLWkZXQUVDVnVjWGttSTI3UFl3NTVZR0RtaHFITWxGUHM5RGsyNGh2MERScEY3WWFuNXRIOFFiZkdHTkc1 WHVTVmdFREFFL1dYL2daTE5QWC83bjJDK3dsMXhqNlRSOXB3bHZubG1JYlNhN1J3R1lCcXd5bHphNnBwOGthNXVtWE hsrvy3whdjrjjtnenzay9udkzfsmdkdwhqz3run0hyrnl3ctduukn2ywpta3byrktnbvnaetjhtjfsmu9utuzcsevx b0rcvevdbud0mi9uctnvvenir3hvse9bbho4tdlrm2jnrme0sgpgvudybnbta2nndjheohdncnduwjlhsmzzodjvwt NxaFpjcjlPelhZYmx1dm5LZ3RveTFJMUVlc0Jrd1Yyc01ZRGthUjBKRG9qQ2syTU4waGVXSm5RVWYwR2Zwd3lqZTJE euttmzjtcunqbejere9degyxvtvwazuvyuc3dg1bsni1vwthb0vpqm5idznqwuht</cookie><cookie name=" JSESSIONID">MDAwMExPaTRDbVZZRnVJZGFkV0J5Z1l2dWhpOjZlOTE5MDU1LWNjYjQtNGU2OS04ZWJmLWMwNTkyOG NiZTZlNA==</cookie></cookies <identities id="user01"> <identity id="watsonsupport.adl.ibm.com"> <username>user01</username> <type>winfs</type> <password encrypt="yes">ftbufzxczpurvwxiqsg2ww==</password> 11

12 <groups> <group id="domain Users"/> <group id="everyone"/> <group id="users"/> <group id="interactive"/> <group id="console LOGON"/> <group id="authenticated Users"/> <group id="this Organization"/> <group id="wcausers"/> <group id="ntlm Authentication"/> </groups> <properties> <property name="valid">true</property> <property name="connectionid"> </property> <property name="username">user01</property> <property name="creationdate"> </property> <property name="enable">true</property> <property name="aclvl">2</property> <property name="crwid">securesearch.win_64181</property> <property name="spaceid">t </property> <property name="ssoenabled">false</property> </properties> </identity> </identities> IMC cache database was updated with the above SSO token and cookie values The last entry was for user user01, as you can see the ssotoken value was updated with the encoded LtpaToken2 taken from browser just now. Please note in Scenario1 SSO token is not used as security is not enabled, but it is still captured. ## [ T12:27: AEST] [DockExecutorThread-4] [com.ibm.es.api.srv.operations.imc.imcoperations] [getsecureddatasources] [FINE] getsecureddatasources entered [FINER] createsecureddatasource - entered ## [ T12:27: AEST] [DockExecutorThread-4] [com.ibm.es.api.srv.operations.imc.imcoperations] [createsecureddatasource] [FINER] createsecureddatasource - adding type winfs id watsonsupport.adl.ibm.com domainname watsonsupport.adl.ibm.com ## [ T12:27: AEST] [DockExecutorThread-4] [com.ibm.es.api.srv.operations.imc.imcoperations] [createsecureddatasource] [FINER] createsecureddatasource - conid spid t ## [ T12:27: AEST] [DockExecutorThread-4] [com.ibm.es.api.srv.operations.imc.imchelper] [issourcetypessoenabled] [INFO] issourcetypessoenabled - returning SSO disabled ## [ T12:27: AEST] [DockExecutorThread-4] [com.ibm.es.api.srv.operations.imc.imchelper] [createuserprompts] [INFO] Identity is already stored, creating user prompt for t with watsonsupport.adl.ibm.com_winfs [INFO] updateidentities - identity domain: watsonsupport.adl.ibm.com IMC cache was updated [FINE] ValidateUserRequest - entering [FINE] ValidateUserRequest - user: user01 then use customcommunication session to validate user [FINER] Start Check Session customcommunication.node1 [FINE] ValidateUserRequest - isvalid: true ## [ T12:27: AEST] [DockExecutorThread-2] [com.ibm.es.api.srv.operations.imc.imcoperations] [validatewindowsfsuser] [FINE] ValidateUserRequest - returning 12

13 Now the user is validated against this data source crawl space. If there is another crawl space, then continue this process until all are validated. ## [ T16:02: AEST] [DockExecutorThread-1] [com.ibm.es.api.srv.operations.imc.imcoperations] [validatewindowsfsuser] [FINE] ValidateUserRequest - returning ## [ T16:02: AEST] [DockExecutorThread-1] [com.ibm.es.api.srv.operations.imc.imchelper] [checkidentityflag] [INFO] checkidentityflag - identity property valid is true Then get group information is not already in IMC cache, or cache has expired ## [ T16:02: AEST] [DockExecutorThread-1] [com.ibm.es.api.srv.operations.imc.imcoperations] [getusergroups] [INFO] getusergroups - source type: winfs ## [ T16:02: AEST] [DockExecutorThread-1] [com.ibm.es.api.srv.operations.imc.imcoperations] [getwindowsfsusergroups] [FINE] GetGroupsForUserRequest - entering ## [ T16:02: AEST] [DockExecutorThread-1] [com.ibm.es.api.srv.operations.imc.imcoperations] [getwindowsfsusergroups] [FINE] GetGroupsForUserRequest - user: user02. [FINER] GetGroupsForUserRequest - groups: Domain Users;;Everyone;;Users;;INTERACTIVE;;CONSOLE LOGON;;Authenticated Users;;This Organization;;WCAUsers;;NTLM Authentication;; ## [ T16:02: AEST] [DockExecutorThread-1] [com.ibm.es.api.srv.operations.imc.imcoperations] [getwindowsfsusergroups] [FINE] GetGroupsForUserRequest - returning ## [ T16:02: AEST] [DockExecutorThread-1] [com.ibm.es.api.srv.operations.imc.imcoperations] [setusercredentials] [INFO] setusercredentials - entered ## [ T16:02: AEST] [DockExecutorThread-1] [com.ibm.es.api.srv.operations.imc.imcoperations] [setusercredentials] [INFO] setusercredentials returning store the user information in IMC cache, so next time it won't need to do it again until cache expires. [FINER] facetedsearch - read property to turn off impersonation tentatively : null bypassfilter:false ## [ T15:54: AEST] [DockExecutorThread-2] [com.ibm.es.search.wrapper.ozeimpersonatedsearchable] [search] [FINER] facetedsearch - with post-filtering. Scenario 2 Enabled Application SSO, in crawler configuration, SSO is enabled. My Profile should be able to be skipped in this scenario Use IMC to manage user credentials As mentioned earlier, you can also write your own code to attach USC (user security credential information) to each search request in your customized search application, but it will not be covered in this article. What happens when a user launch search application? 1. A user logs in search application, username/password is authenticated against the LDAP server (or SSO token is used to validate if user first visited other application then to WCA search application) 13

14 2. Get a list of searchable collections, find out secured spaces, and find out the spaces that match to the searchable collections. 3. Read IMC cache if cache is still valid, construct USC from user credentials stored in cache (including groups information for the user of that particular data source). 4. If the user credentials in IMC has become expired, the user credential data needs to refresh. In this case, it connects to data source server to validate the user, and extract group information for this user, then update the IMC cache with latest user information. 5. Get IMC properties from sso.properties file, if IMC is in use, and the relevant checkbox is ticked for the crawler that support SSO. 6. Then validate the user id on the data source server using SSO token taken from current browser session s cookies (ltpa token or ltpa v2 token). It always uses ltpa2 token if using WCA is installed with embedded application server (as WebSphere Liberty Profie only support ltpa v2) 7. Repeat this process for all crawl spaces matched to the login user, until a complete USC for all relevant data source servers is constructed for this user. 8. With this USC attached to the search request, search server will be able to return search results only matching to this user, either by pre-filtering or post-filtering. Troubleshooting What to check if search result is not expected? We often see problems that search result is not expected, such as documents should be returned in the result as the user should have access; or documents that the user does not have access returned in search result unexpectedly. What logs do we need? Search server trace Please refer to technote: Enabling detailed logging to troubleshoot document-level security issues Detailed crawler trace Please refer to technote: This technote also introduces how to enable trace for discovery, customcommunication session. For almost all types of data source, discovery session is used to connect to data source server to do user validation. All types of crawlers except those using Adapter Framework, which are SharePoint crawler, FileNet P8 crawler and Agent for Windows file system crawler. 14

15 For these data source, customcommunication session is used to connect to data source server to do user validation. For crawler using Adapter Framework, customcommunication is the only session in charge of pre-filtering and post-filtering Clear IMC cache to force discovery session to connect to data source server to get latest user information. Please refer to technote: Identify whether the problem happens in pre-filtering or post-filtering From admin console, select the collection that have unexpected secure search result, click Edit collection, you will see options to ignore pre-filtering or post-filtering. Continue secured search even when crawler server is down From above you will understand that discovery session is called when validating user id to the data source server, or extracting group information. Discovery session runs on crawler server, so it becomes single point of failure if crawler server is down, you cannot perform secured search, even when the data source server and search servers are working normally. Crawler server becomes the single point of failure. Fortunately, there is a method to overcome this restriction, please refer to KC Supporting secure search when the crawler server is not available Calling discovery command to connect to data source server directly How to examine user validation and group extraction for document-level security issues in OmniFind Enterprise Edition How to check document ACL stored in index? You can use DumpIndex command to check how document ACL is stored in index. DumpIndex <cid> --uri security This command will show you all documents in the index with security information attached to each document. Please refer to technote: Resources IBM Watson Content Analytics 3.5 Knowledge Center ery.es.ad.doc/iiysasecnocrawler.htm 15

16 IBM Watson Explorer Analytical Components Knowledge Center ery.es.nav.doc/explorer_analytics.htm 16