MuseKnowledge Proxy Source Profiling

Transcription

1 MuseKnowledge Proxy Source Profiling MuseGlobal, Inc. One Embarcadero Suite 500 San Francisco, CA MuseGlobal S.A Calea Bucuresti Bl. 27B, Sc. 1, Ap. 10 Craiova, România EduLib, S.R.L. Calea Bucuresti Bl. 27B, Sc. 1, Ap. 2 Craiova, România Version: 1.0 Date: 22nd June 2016 Author: EduLib, S.R.L.

2 MuseKnowledge Proxy Highly customizable multi-platform proxy server Gateway to authenticated restricted content; Rewriting web server; Web Access Management (WAM); Multi tenant hosting for all customers; MuseKnowledge Proxy Applications: Fully configurable interfaces allowing administrators to setup remote or local Data Sources ("Sources"); End users enjoy a single sign-on for all of the subscribed data; The number of MuseKnowledge Proxy Applications and sources installed in an application are controlled by the license; More than 10 years within the Muse Federated Search Platform to manage the authentication to resources and the navigation to full text. PAGE 2

3 MuseKnowledge Proxy MuseKnowledge Proxy Terminology Source content provider resource (platform, website); MuseKnowledge Proxy Source profile XML file with the configuration options for a specific resource; MuseKnowledge Proxy files location on disk: ${MUSE_HOME}/proxy Where ${MUSE_HOME} is default %ProgramFiles%/muse on Windows Operating Systems and /opt/muse on Linux Operating Systems. MuseKnowledge Proxy Application location on disk: ${MUSE_HOME}/proxy/webcontexts/Applications/Application_ID Where Application_ID is the ID of the MuseKnowledge Proxy Application. Source profiles location on disk: ${MUSE_HOME}/proxy/webcontexts/Applications/Application_ID/profiles/sources/ Assumptions for adding a new MuseKnowledge Proxy Source profile A MuseKnowledge Proxy Application already exists; The MuseKnowledge Proxy Application is accessible; The Source Groups are defined; The Source is defined and placed within at least one Source Group. PAGE 3

4 MuseKnowledge Proxy Software used for profiling Web browser: Google Chrome, Mozilla Firefox, Internet Explorer; Network sniffer: Fiddler, Browser Developer Tools; XML Editor (optional). XML files can be edited with any text editor (like Notepad for example) however it is good to verify the XML is well formed by loading it into a browser. Skills for MuseKnowledge Proxy Admin Good networking knowledge; Good understanding of HTTP; Good understanding of HTML; Good knowledge of the MuseKnowledge Proxy and its structure. PAGE 4

5 Source Configuration Source Profile XML file: ${MUSE_HOME}/proxy/webcontexts/Applications/Application_ID/profiles/sources/S ourceid.xml Where Application_ID is the MuseKnowledge Proxy Application identifier and SourceID.xml is the XML configuration file associated with that resource. Sources index file: ${MUSE_HOME}/proxy/webcontexts/Applications/Application_ID/profiles/Sources.x ml It contains the list of sources present in the MuseKnowledge Proxy Application. Sample entry: <SOURCE> <IDENTIFIER code="0">forensicnetbase</identifier> <NAME>FORENSICnetBASE</NAME> <DESCRIPTION>Featuring works by esteemed criminologists and forensic practitioners, this unparalleled e-library boasts an ever-growing number of essential references in crisis management and negotiation methods, arson and homicide investigation, expert witnessing, and forensic pathology. This versatile invaluable product is well-suited to the needs of the smallest of sheriff&apos;s departments to large governmental law enforcement agencies, such as the CIA, DEA, and FBI. </DESCRIPTION> </SOURCE> <MODULE> </MODULE> <CLASS>com.edulib.muse.proxy.application.sources.modules.impl.HttpModule</CLASS> <CONFIGURATION_FILE>${WEB_CONTEXT_HOME}/profiles/sources/FORENSICnetBASE.xml</CONFIGURATION_FILE> More PAGE 5

6 Source Configuration Sources Groups definitions: ${MUSE_HOME}/proxy/webcontexts/Applications/Application_ID/profiles/SourcesGr oups.xml It contains the groups definitions. Sample entry: <SOURCES_GROUP> <IDENTIFIER>1</IDENTIFIER> <NAME>Default Sources</NAME> <SOURCES> </SOURCES> <IDENTIFIER>FORENSICnetBASE</IDENTIFIER> <IDENTIFIER>ProjectMuse</IDENTIFIER> </SOURCES_GROUP> 3 steps for enabling a MuseKnowledge Proxy source 1. Make sure the source XML profile exists, if not, create it by copying an existing one or download it from our EduLib web repository : ${MUSE_HOME}/proxy/webcontexts/Applications/Application_ID/profiles/sources/SourceID.xml 2. Add a new entry in the sources index file, e.g. in the file: ${MUSE_HOME}/proxy/webcontexts/Applications/Application_ID/profiles/Sources.xml 3. Add a new entry in the sources groups file, e.g. in the file: ${MUSE_HOME}/proxy/webcontexts/Applications/Application_ID/profiles/SourcesGroups.xml PAGE 6

7 Basic Source Settings <URL> The URL of the target site service accessed by the source. Example: <URL> <NAME> The name of the source. Example: <NAME>IEEE Xplore Digital Library</NAME> <DESCRIPTION> A description of the source. Example: <DESCRIPTION>The IEEE Xplore digital library is a powerful resource for discovery and access to scientific and technical content published by the IEEE (Institute of Electrical and Electronics Engineers) and its publishing partners.</description> <REWRITING_PATTERNS> The URLs rewriting patterns. In the target site service pages only the URLs matching these patterns will be rewritten. Values: [include: PATTERNS], [exclude: PATTERNS] where PATTERNS = semicolon-delimited list of regular expressions matching the site's domain[:port][/url_file]. Example: <REWRITING_PATTERNS>*.ieee.org;*/*.js*;</REWRITING_PATTERNS> PAGE 7

8 Advanced Source Settings <TRANSPARENT_CONTENT_PATTERNS> URLs matching these patterns are rewritten to follow-up links if they match the REWRITING_PATTERNS, so they will be proxied, but their response content is not processed for rewrite. This is necessary because some resources needs to be requested from the same origin but should remain un-touched. Example: <TRANSPARENT_CONTENT_PATTERNS>*/*.js*;ieeexplore.ieee.org/xpl/mostPopular*.ajax*;ieeexplore.ieee.org/xpl/mostRecent*.ajax*;</T RANSPARENT_CONTENT_PATTERNS> <REWRITE_BY_HOST> true or false. This option states if the type of the follow-up links is Rewrite by Host. When set to true the follow-up links will contain the Muse Proxy marker values as part of the proxy host name subdomain. Wildcard DNS is required to have the Rewrite by Host functional. Example of a follow-up link when using rewrite by host: <ENCODING> The encoding of the site (e.g. UTF-8). <REFERER> The referer for the source. It is required to be filled for the sources that authenticate only from an explicit referer. Can be also used if it is required to have a custom referer in order for the target source provider to easily track the request coming from this module. <CUSTOM_HTTP_HEADERS> Custom HTTP headers to be sent when performing the request. Structure: <CUSTOM_HTTP_HEADER> The custom HTTP header. <NAME> The custom HTTP header name. </NAME> <VALUE> The custom HTTP header value. </VALUE> </CUSTOM_HTTP_HEADER> More PAGE 8

9 Advanced Source Settings <REPLACE_HOST>The value to be used for replacing the host from the URL obtained after the source authentication in order to be redirected to another page from the target site. Example: <REPLACE_HOST>es.mysite.com</REPLACE_HOST> <REPLACE_PATH> The value to be used for replacing the path from the URL obtained after the source authentication in order to be redirected to another page from the target site. Example: <REPLACE_PATH>/search/searchForm.html</REPLACE_PATH> <SSL_CERTIFICATES> The SSL certificates full path to be used for the source in order to obtain HTTPS authentication to the target source. <SSL_TRUST_ALL> If set to true it will accept all certificates from the remote source. <FILTER> Configurable filter to be applied on the content for this source. This rewriting filter is a general configurable Find and Replace filter which can rewrite/unrewrite or do other replacements based on JDK regular expressions and/or Muse Proxy specific Token Rule expressions. There can be multiple FILTER elements and also multiple find and replace rules and Token rules inside a FILTER element. Example: <FILTER> <PATTERNS> <FIND scope="first"> <REPLACE rewrite="0">$0</replace> </FILTER> PAGE 9

10 Additional Source Configuration Settings <EXTRACTOR>, <URL>, <POST_PARAMETERS> Extract and Navigate. Example: <URL> <EXTRACTOR ref="nexturl"><![cdata[<a class="ul" href="([^"]+)"\s*onmouseover="popup\('ihs Standards]]></EXTRACTOR> <URL>${nextURL_1}</URL> Processing options with refprocess attribute: "urlencode", "urldecode", "xmlescape", "xmlunescape. Example: <URL> <EXTRACTOR ref="eventtarget" refprocess="urlencode">type="hidden"\sname=" EVENTTARGET"\sid=" EVENTTARGET"\svalue="([^"]*?)"</EXTRACTOR> <EXTRACTOR ref="eventargument" refprocess="urlencode">type="hidden"\sname=" EVENTARGUMENT"\sid=" EVENTARGUMENT"\svalue="([^"]*?)"</EXTRACTOR> <EXTRACTOR ref="viewstate" refprocess="urlencode">type="hidden"\sname=" VIEWSTATE"\sid=" VIEWSTATE"\svalue="([^"]+?)"</EXTRACTOR> <EXTRACTOR ref="eventvalidation" refprocess="urlencode">type="hidden"\sname=" EVENTVALIDATION"\sid=" EVENTVALIDATION"\svalue="([^"]+?)"</EXTRACTOR> <URL> <POST_PARAMETERS><![CDATA[ EVENTTARGET=${eventtarget_1}& EVENTARGUMENT=${eventargument_1}& VIEWSTATE=${views tate_1}& EVENTVALIDATION=${eventvalidation_1}&_ctl0%3AphContent%3AtxUserName=${userName}&_ctl0%3AphContent%3AtxPass word=${userpassword}&_ctl0%3aphcontent%3abtnlogin=login]]></post_parameters> Define own JavaScript functions to be referred in the refprocess attribute: <DEF><![CDATA[ function jsonjoin(input) { //JS code here } ]]></DEF> <EXTRACTOR ref="sectek" refprocess="jsonjoin">(^.*+$)</extractor> More PAGE 10

11 Additional Source Configuration Settings <SHOW_GET_PARAMETERS> Show the URL details in the follow-up link. Example: <SHOW_GET_PARAMETERS>true</SHOW_GET_PARAMETERS> <CONNECT_TIMEOUT>, <READ_TIMEOUT> Connection time-outs. Example: <CONNECT_TIMEOUT>60000</CONNECT_TIMEOUT> <READ_TIMEOUT>60000</READ_TIMEOUT> <PROXY_USED>, <PROXY_HOST>, <PROXY_PORT>, <PROXY_PAC>, <PROXY_AUTHORIZATION_USER_NAME>, <PROXY_AUTHORIZATION_USER_PASSWORD>, <PROXY_AUTHORIZATION_SCHEME> Proxy Chaining configuration elements. Example: <PROXY_USED>SOURCE_LEVEL</PROXY_USED> <PROXY_HOST> </PROXY_HOST> <PROXY_PORT>3128</PROXY_PORT> <PROXY_AUTHORIZATION_USER_NAME>univ</PROXY_AUTHORIZATION_USER_NAME> <PROXY_AUTHORIZATION_USER_PASSWORD>fadwadOrp0</PROXY_AUTHORIZATION_USER_PASSWORD> <PROXY_AUTHORIZATION_SCHEME>Basic</PROXY_AUTHORIZATION_SCHEME> <LINK_OUT> Link out to other source profiles. Example: <LINK_OUT>true</LINK_OUT> <COOKIES> Specify cookies to be sent. Example: <COOKIES> <COOKIE>persistentSession=hl1k61cn32ecu1iu979cr5rbk2; path=/; domain=mydomain.com;</cookie> </COOKIES> More PAGE 11

12 Additional Source Configuration Settings <X_FORWARDED_FOR> Reveal the IP of the end-user to the Target Source. Example: <X_FORWARDED_FOR>true</X_FORWARDED_FOR> <FOLLOW_REDIRECTS> Automatic HTTP Redirects. Example: <FOLLOW_REDIRECTS>true</FOLLOW_REDIRECTS> <REDIRECT> Redirecting to Remote Source, e.g. drop out of rewriting. Example: <REDIRECT>true</REDIRECT> <CLIENT_KEYSTORE> Client Side SSL Authentication. Example: <CLIENT_KEYSTORE password="changeit">${web_context_home}/certificates/seap.jks</client_keystore> <PARAMETERS> General use definition of parameters. Example: <PARAMETERS> <PARAMETER> <NAME> </NAME><VALUE>MY_ </VALUE> </PARAMETER> <PARAMETER> <NAME>password</NAME><VALUE>MY_PASSWORD</VALUE> </PARAMETER> </PARAMETERS> <AUTHENTICATION_TYPE>, <LAST_UPDATED> Informational elements. Example: <AUTHENTICATION_TYPE>IP</AUTHENTICATION_TYPE> <LAST_UPDATED> </LAST_UPDATED> More PAGE 12

13 Recap: Adding a new Source Edit Edit Copy an existing ${MUSE_HOME}/proxy/webcontexts/Applications/Application_ID/profiles/Sources.xml ${MUSE_HOME}/proxy/webcontexts/Applications/Application_ID/profiles/SourcesGroups.xml source profile under Duplicate an existing entry and change accordingly the values of IDENTIFIER, NAME and Duplicate a new filename, for CONFIGURATION_FILE. an existing entry and change accordingly the values of IDENTIFIER. example NewSource. Edit the NewSource.xml file and: 1) Change the URL value with the one of your new source. 2) Change the values for the NAME and DESCRIPTION fields accordingly. 3) Change accordingly the values for REWRITING_PATTERNS. PAGE 13

14 Username/Password Controlled Resource Use a network sniffer or browser developer tools to see the request made when submitting the username/password; URL value: POST_PARAMETERS value: <POST_PARAMETERS>autho=${userName}&password=${userPassword}&language=en&next=html%2Fhome.html&bad=error%2Fbadlo gin_en.html&entityjavascript=true&entityscreensize=large</post_parameters> PARAMETERS values: <PARAMETERS> <PARAMETER><NAME>userName</NAME><VALUE> </VALUE> </PARAMETER> <PARAMETER><NAME>userPassword</NAME><VALUE>test%26Password</VALUE> </PARAMETER> </PARAMETERS> PAGE 14

15 Use Case 1: How to correct unrewritten links Sniff communication and search for the unrewritten URL. Once found identify the URL pattern of the request and create a filter. PAGE 15

16 Use Case 2: A Complex Username/Password Authentication Scenario Second most popular authentication type after IP authentication. IPs shortage is becoming an issue; Make use of Extract and Navigate features: EXTRACTOR, URL and POST_PARAMETERS; Applicable where providing the Username/Password values in a single request is not working; More PAGE 16

17 Use Case 2: A Complex Username/Password Authentication Scenario <URL> <POST_PARAMETERS></POST_PARAMETERS> First request to the logon page. From the logon page extract the dynamic hidden fields. <EXTRACTOR ref="eventtarget" refprocess="urlencode">type="hidden"\sname=" EVENTTARGET"\sid=" EVENTTARGET"\svalue="([^"]*?)"</EXTRACTOR> <EXTRACTOR ref="eventargument" refprocess="urlencode">type="hidden"\sname=" EVENTARGUMENT"\sid=" EVENTARGUMENT"\svalue="([^"]*?)"</EXTRACTOR> <EXTRACTOR ref="viewstate" refprocess="urlencode">type="hidden"\sname=" VIEWSTATE"\sid=" VIEWSTATE"\svalue="([^"]+?)"</EXTRACTOR> <EXTRACTOR ref="viewstategenerator" refprocess="urlencode">type="hidden"\sname=" VIEWSTATEGENERATOR"\sid=" VIEWSTATEGENERATOR"\svalue="([^"]+?)"</EXTRACTOR> <EXTRACTOR ref="eventvalidation" refprocess="urlencode">type="hidden"\sname=" EVENTVALIDATION"\sid=" EVENTVALIDATION"\svalue="([^"]+?)"</EXTRACTOR> More PAGE 17

18 Use Case 2: A Complex Username/Password Authentication Scenario <URL> <POST_PARAMETERS><![CDATA[ EVENTTARGET=${eventtarget_1}& EVENTARGUMENT=${eventargument_1}& VIEWSTATE=${viewstate_1}& VIEWSTATEGENERATOR=${viewstategenerator_1}& EVENTVALIDATION=${eventvalidation_1}& ctl00%24maincontent%24userlogin%24username=${username}&ctl00%24maincontent%24userlogin%24password=${userpassword}& ctl00%24maincontent%24userlogin%24button1=giri%c5%9f]]></post_parameters> <PARAMETERS> <PARAMETER><NAME>userName</NAME><VALUE>YYY</VALUE> </PARAMETER> <PARAMETER><NAME>userPassword</NAME><VALUE>ZZZ</VALUE> </PARAMETER> </PARAMETERS> Second request to submit the login details using the values extracted previously. Complete configuration source profile can be downloaded from here. PAGE 18

19 Use Case 3: Transparent Navigation Scenario Perform actions transparently to the end-user: authenticate with username/password, change the default language, land on the desired page. And reach to the desired landing page. Logon page Landing page after logon from where change the language to English. In the new page click on UK United Kingdom and then on Contract notice More PAGE 19

20 Use Case 3: Transparent Navigation Scenario <URL> <EXTRACTOR ref="step1" refprocess="xmlunescape"><![cdata[<a href="( loginrequestid=[^"]+?)"\sclass="main_domain external_hover"\stitle="external"><span class="external"></span>]]></extractor> <URL>${step1_1}</URL> <EXTRACTOR ref="posturl">id="loginform"\sname="strongloginform"\saccept-charset="utf-8"\saction="([^"]+?)"</extractor> <EXTRACTOR ref="txid">type="hidden"\sname="txid"\svalue="([^"]+?)"</extractor> <EXTRACTOR ref="lt">type="hidden"\sname="lt"\svalue="([^"]+?)"</extractor> <EXTRACTOR ref="loginrequestid">type="hidden"\sname="loginrequestid"\svalue="([^"]+?)"</extractor> <URL>${postURL_1}</URL> <POST_PARAMETERS><![CDATA[TxId=${TxId_1}&lt=${lt_1}&domain=external&loginRequestId=${loginRequestId_1}&useDefaultStrength=true& timezone=gmt%2b03%3a00&selfhost=webgate.ec.europa.eu&username=${username}&zpassword=&password=${userpassword}&submit=login%21]]> </POST_PARAMETERS> <EXTRACTOR ref="lasturl"><![cdata[<a\sid="fallbacklink"\shref="( <URL>${lastURL_1}</URL> <URL> <URL> <POST_PARAMETERS><![CDATA[action=cl&lgId=en]]></POST_PARAMETERS> Perform authentication to the site s authentication service and go to the home page. Change the language to English. <URL> Click on the UK United Kingdom link. <URL> <POST_PARAMETERS><![CDATA[action=selectFacet&quickSearchCriteria=&searchScope=LAST_EDITION]]></POST_PARAMETERS> Complete configuration source profile can be downloaded from here. Click on the Contract notice link. PAGE 20

21 MuseKnowledge Proxy Source Profiling