Installation Guide. Sitecore Federated Experience Manager. Installation & Configuration Guide

Transcription

1 Sitecore Federated Experience Manager Installation Guide Rev: 23 August 2014 Sitecore Federated Experience Manager Installation Guide Installation & Configuration Guide

2 Table of Contents Chapter 1 Overview Capabilities Tracking External Sites Personalizing External Sites Tracking External Applications Support for Future Sitecore Versions... 8 Chapter 2 Quick Start Guide Installation External Site Tracking Create Site Definition Item Add Beacon Script to External Site Generate Mirror Items Personalization Add Placeholder to External Page Add Rendering to External Page External Application Tracking Determine the Data to Push into Sitecore Define Page Event Configure External Application to Push Data into Sitecore Define External Application in Sitecore Configure Sitecore to Accept Data from External Application Working with HTTPS Working with Content Delivery Networks (CDN) Working with Multi-Server Environments Working with Sitecore Azure Troubleshooting Link is broken popup message Page Editor link navigates out of Page Editor Page Editor link navigates to different Sitecore site Chapter 3 External Page Tracking Tracking Process Beacon Script Beacon Script URL Beacon Script Generation jquery Cross-site Scripting (XSS) External Sites Defining an External Site Template: External Site Site Resolvers Site Resolution Process Mirror Site Provider Specifying a Site Resolver Defining a Site Resolver Template: Site Resolver Chapter 4 Dynamic Tracking Web Tracking Rules Site-wide Web Tracking Rules Page-specific Web Tracking Rules Conditions & Actions Tracking Settings Defining Tracking Settings Site-wide Tracking Settings Page-specific Tracking Settings Page 2 of 84

3 4.3 Content Tracking Rules Specifying Content Tracking Rules Conditions & Actions Chapter 5 Crawling Process Overview Website Crawling Process Crawler Manager Crawler Provider Pipeline: fxm.web.websitecrawling Crawlers Specifying a Crawler Defining a Crawler Page Crawling Process Crawler Manager Crawler Provider Mirror Manager Mirror Provider Pipeline: fxm.web.createmirroritem Pipeline: fxm.web.resolvemirrortemplate Mirror Item Creation Process Language Resolution Generate Mirror Item Path Filter Crawling Pages Run Crawling Rules Mirror Items Defining a Mirror Item Template: External Page Language Resolvers Specifying a Language Resolver Defining a Language Resolver Template: QueryString Resolver Template: URL Segment Resolver Crawling Rules Specifying Crawling Rules Conditions & Actions Tracking Configuration Rules Specifying Tracking Configuration Rules Conditions & Actions Page Crawling & Page Editor Page Editor Layout Link Rewriting Page Crawling & External Page Views Chapter 6 Running the Crawling Process Comparison of Options Bulk Mirror Item Creation Starting the Process Manually Monitoring the Process Stopping the Process Manually Starting the Process Programmatically Scheduling the Process Manual Mirror Item Creation Disabling Ad Hoc Mirror Item Creation Enabling Chapter 7 External Page Personalization Personalization Process Compatibility of Sitecore Renderings Page 3 of 84

4 7.1.2 Making Incompatible Renderings Compatible Placeholder Selectors Specifying Placeholder Selectors Defining a Placeholder Selector Template: Placeholder Selector Chapter 8 External Application Tracking Tracking Process Analytics Manager Analytics Provider Pipeline: fxm.analytics.trackoffsiteevent External Applications Defining an External Web Application Template: External Web Application How External Applications Are Used External Application Configurations Defining an External Application Configuration Template: External Application Configuration Parameter Readers Specifying a Parameter Reader Defining a Parameter Reader Template: Parameters Reader Parameter Reader: Form Reader Parameter Reader: Query String Reader Parameter Reader: Query String and Form Reader Configuration Rules Specifying Configuration Rules Conditions & Actions Event Parsers Specifying an Event Parser Single Event Parsers Configuring a Rule Using a Single Event Parser Multi-Event Parsers Javascript Triggered Tracking Requests Client.js Variable: _scfxmanalytics Tracking Instructions Preventing Event Tracking Requests from Being Cancelled Type: SitecoreFxmAnalytics Non-Javascript Triggered Tracking Requests ExternalEventsListener URL Submitting Tracking Requests Resolving the External Site Chapter 9 Conditions & Actions FXM Analytics Conditions Actions FXM Content Tracking Conditions Actions FXM Page Conditions Actions FXM Template Definition Conditions Actions FXM Tracking Conditions Page 4 of 84

5 9.5.2 Actions FXM Tracking Configuration Conditions Actions FXM Web Crawling Conditions Actions Page 5 of 84

6 Chapter 1 Overview Sitecore Federated Experience Manager allows activity in non-sitecore websites and applications to be tracked, and for content managed and personalized by Sitecore to be injected in non-sitecore websites. This chapter provides an overview of Sitecore Federated Experience Manager. Page 6 of 84

7 1.1 Capabilities This section provides an overview of the capabilities of Federated Experience Manager Tracking External Sites Sitecore Analytics allows you to better understand your customers' interests and activities. But if Sitecore is only able to track activity on channels managed by Sitecore, the understanding you are getting is not complete. Therefore, tracking activity on non-sitecore web sites is an important part of a comprehensive customer engagement strategy. Sitecore Federated Experience Manager allows you to use the tracking capabilities available on Sitecore web sites to non-sitecore web sites using a common approach used by analytics vendors. A small amount of Javascript is added to the non-sitecore pages that need to be tracked. This Javascript makes a call to Sitecore when a visitor views the non-sitecore page. Federated Experience Manager creates a Sitecore item that corresponds to the non-sitecore page. This item is called a "mirror item". The mirror item gives Sitecore a means by which to track activity. This approach allows Sitecore to use its standard request-handling process for requests to non- Sitecore pages. This means that all of Sitecore's tracking capabilities - profiling, goals, page events, engagement plans, etc. - can be used with non-sitecore pages Personalizing External Sites Tracking allows Sitecore to better understand your customers' interests and activities. But being able to collect this information is only a small part of the reason you would use Sitecore as a part of your customer engagement strategy. Being able to use this information in order to affect customer engagement is a bigger part of the reason. Federated Experience Manager allows you to inject personalized content from Sitecore into non- Sitecore web sites. This personalization is able to use the information collect across all channels in order to provide your customers with the most relevant content, even on non-sitecore web sites. As explained in the Tracking section, non-sitecore pages that are tracked have the Federated Experience Manager Javascript added to it. When a visitor views a non-sitecore page, a request is sent from the visitor's browser to Sitecore. By handling this request Sitecore is able to track the non- Sitecore page view. In addition to tracking the page view, handling this request also allows Sitecore to generate personalized content. The response that Sitecore returns contents any personalized content that should be injected in the non-sitecore page. The Javascript that was responsible for making the request to Sitecore is also responsible for injecting the personalized content from Sitecore into the non-sitecore page Tracking External Applications Tracking is not limited to external sites. Federated Experience Manager allows you to track activity on external applications as well. For example, when a visitor leaves a comment on a non-sitecore site you might want to record a page event in Sitecore. This is accomplished by providing a web-based API that allows you to push activity data into Sitecore. After the activity data is passed to Sitecore, Federated Experience Manager allows you to configure how that data should be handled. Page 7 of 84

8 1.2 Support for Future Sitecore Versions Federated Experience Manager was designed for the Sitecore 7 product line (7.1, 7.2 and 7.5). The functionality available with Federated Experience Manager will be incorporated into Sitecore 8 and later. However, this functionality will be implemented in a very different way. This is in order to take advantage of a number of new platform features that are available starting with Sitecore 8. Sitecore will provide information on how customers can migrate their Federated Experience Manager configuration to its equivalent in Sitecore 8. However, this will be a manual process. Due to the highly customized nature of a Federated Experience Manager implementation it is impossible to provide tools to automate the process. For more information contact Sitecore Product Support. Page 8 of 84

9 Chapter 2 Quick Start Guide This chapter describes the most basic configuration and usage of Federated Experience Manager. Page 9 of 84

10 2.1 Installation Sitecore Federated Experience Manager is installed using the Sitecore Installation Wizard. Note: The license on your Sitecore server must include Federated Experience Manager in order for the module to work correctly. If you are unsure whether you are licensed to use Federated Experience Manager check with your local Sitecore representative. 1. Use the Sitecore Installation Wizard to install the Federated Experience Manager package. 2. Publish the site using the "republish everything" option. Page 10 of 84

11 2.2 External Site Tracking This section describes the minimal steps needed in order to configure external site tracking Create Site Definition Item In order to track and personalize a non-sitecore web site, some information about the site must be available to Federated Experience Manager. A site definition item is the Sitecore item that represents a non-sitecore web site. 1. In Content Editor navigate to /sitecore/content/federated Experience Manager 2. Create a new item using the "External Site" template 3. In the "Protocol" field select the protocol used to access the external site you want to track 4. In the "Primary Host Name" field enter the host name of the external site you want to track 5. Save the item 6. Publish the item Add Beacon Script to External Site The beacon script must be added to the external site you want to track. Add the following link to the pages in the external site (be sure to replace [host name] with the proper host name for your Sitecore server): <script type="text/javascript" src="//[host name]/-/fxm/web/"></script> Generate Mirror Items Mirror items are the Sitecore items that correspond with a URL on a non-sitecore website. Federated Experience Manager requires a mirror item exist for each external URL you want to track. There are several different ways that mirror items can be generated. The mirror item creation strategy you will use will involve one or more of the options defined in this section. Determining the appropriate strategy must be considered on a case-by-case basis. Option 1. Bulk Creation Federated Experience Manager includes a component that can crawl a non-sitecore web site and create mirror items for the pages in the non-sitecore web site. The advantage of this component is that it creates mirror items before they are needed. This can reduce the load on the delivery server. For more information see section 6.2. Option 2. Manual Creation Page Editor can be used to navigate the non-sitecore site. When you click a link in Page Editor, a mirror item can be created. For more information see section 6.3. Option 3. Ad Hoc Creation When a visitor views a page on an external site, a mirror item can be created if one does not already exist. This is similar to the manual creation option. For more information see section 6.4. Page 11 of 84

12 2.3 Personalization This section describes the minimal steps needed in order to configure mirror items to support personalization Add Placeholder to External Page Sitecore renderings are used to display personalized content. Renderings are added to pages using placeholders. In order to add personalized content to a non-sitecore page, you add renderings to placeholders just like you would with a Sitecore-managed page. When using Federated Experience Manager to add personalization to an external page you must tell Sitecore where the placeholders belong. This is accomplished using placeholder selectors. 1. In Page Editor navigate to the external site item located under /sitecore/content/federated Experience Manager 2. Navigate to the page where you want to add a placeholder 3. In ribbon click View and check the box for FXM 4. Click anywhere on the page. The page element you click will be selected. You may not select the precise element you want with your first click. Click to select the parent 1 element. 5. Add a placeholder by clicking one of the following buttons: 1 A web page is a collection of nested elements. For example, a table has one or more rows. Each row has one or more columns. Each column has its own elements and so on. The terms "parent" and "child" refer to this relationship. In this example, the parent of a row is the table. The children of the table are its rows. Page 12 of 84

13 Inserts the new placeholder before the selected element Replaces the selected element with the new placeholder Inserts the new placeholder after the selected element 6. Enter a name for the placeholder and click Ok. The placeholder name serves two purposes. One, it describes the purpose of the placeholder. Two, it is used to configure placeholder settings. 7. The new placeholder appears on the page. Save the page Add Rendering to External Page Adding a rendering to an external page is done by adding renderings to the mirror item that corresponds to the external page. Simply open the mirror item in Page Editor and add renderings as you would with any other Sitecore item. Page 13 of 84

14 Note: FXM mode in Page Editor must be turned off before you are able to add renderings to placeholders. Page 14 of 84

15 2.4 External Application Tracking This section describes the minimal steps needed in order to configure external application tracking. This section uses a specific example to explain how to configure external application tracking. An external web application has a button. When the button is clicked a page event should be recorded in Sitecore. Note: Even though this example is capturing activity from a web application, Federated Experience Manager is not limited to activity from web applications. Activity from any application can be captured. For more information see Chapter Determine the Data to Push into Sitecore As described above, when a certain button on the external web application is clicked, a page event should be recorded in Sitecore. The page event should include the name of the button that was clicked Define Page Event In order to track a page event in Sitecore, a page event definition item is needed. 1. In Content Editor navigate to /sitecore/system/settings/analytics/page Events 2. Create a new item named "External Button Click" using the "Page Event" template. 3. Click the "Review" tab and click "Deploy". 4. Publish the item Configure External Application to Push Data into Sitecore The external application needs to be modified in order to include instructions on what data to push and how to push that data into Sitecore. 1. Add the following code to the external web page with the button to track (set the location of your Sitecore server for the "sitecoreurl" variable): <script type="text/javascript"> var sitecoreurl = ' host name]/'; var _scfxmanalytics = _scfxmanalytics []; _scfxmanalytics.push(['_setdomain', sitecoreurl]); (function () { var js = document.createelement('script'); js.type = 'text/javascript'; js.async = true; js.src = sitecoreurl + 'Fxm/Analytics/js/Client.js'; var firstscriptelement = document.getelementsbytagname('script')[0]; firstscriptelement.parentnode.insertbefore(js, firstscriptelement); })(); </script> 2. Add the following code to the button's "onclick" event handler: Page 15 of 84

16 _scfxmanalytics.push(['_trackevent', {buttonname: 'my external button'}]); Define External Application in Sitecore Federated Experience Manager will only accept data from external applications that are defined. This section describes how to define the external application. 1. In Content Editor navigate to /sitecore/system/modules/federated Experience Manager/Analytics/External Applications 2. Create a new item using the "External Web Application" template 3. In the "Host Name" field enter the host name of the web server that is hosting the external application 4. In the "Parameters Reader" field select "Form Reader" 5. Save the item 6. Publish the item Configure Sitecore to Accept Data from External Application After the external domain is defined the domain must be configured so Federated Experience Manager knows how to handle data from the external application. 1. In Content Editor navigate to /sitecore/system/modules/federated Experience Manager/Analytics/External Application Configurations 2. Create a new item using the "External Application Configuration" template 3. In the "External Applications" field select the external web application item added in section In the "Configuration" field click "Edit Rule" 5. Add the following condition a. Section: Fxm Analytics b. Condition: where the specific parameter names parameters are not empty 6. Set the "specific parameter names" variable to "buttonname" 7. Add the following action a. Section: Fxm Analytics b. Action: track the specific page event and set DataKey value from the specific parameter, Data value from the specific parameter and Text value from the specific parameter Page 16 of 84

17 8. Set the "specific page event" variable to the page event item created in section Set the "Data value parameter" variable to "buttonname" 10. Click OK (You will get a warning about the rule may not function correctly. This is OK. For this condition the other variables are optional.) 11. Save the item 12. Publish the item 13. When you click the button in the external application a page event will be recorded in Sitecore Page 17 of 84

18 2.5 Working with HTTPS Tracking when HTTPS is used is possible, but there are some limitations. The following table describes what is and is not supported when one or both of the systems uses HTTPS. External site protocol Sitecore protocol Website tracking Application tracking Website personalization Page Editor support http http Yes Yes Yes Yes Notes https http No Yes No No The browser will not allow a script from a secure source to send requests to a nonsecure destination. http https Yes Yes Yes No The browser will not allow a script from a secure source to send requests to a nonsecure destination. https https Yes Yes Yes Yes Page 18 of 84

19 2.6 Working with Content Delivery Networks (CDN) Federated Experience Manager supports tracking and personalization on websites that are delivered by a CDN (content delivery network). In order for tracking and personalization to work, the beacon script must be delivered by Sitecore. In other words, the beacon script cannot be cached on the CDN, nor can it be delivered by the CDN. Page 19 of 84

20 2.7 Working with Multi-Server Environments When installing Federated Experience Manager in a multi-server environment, delivery servers must be able to make outbound requests to any external sites that are tracked. This is required by the crawling process. Make sure that any firewalls that affect the delivery servers are configured appropriately. Page 20 of 84

21 2.8 Working with Sitecore Azure No special configuration is required in order for Federated Experience Manager to work with Sitecore Azure. Page 21 of 84

22 2.9 Troubleshooting This section describes some common issues you may encounter Link is broken popup message Issue: In Page Editor you click a link and a popup message appears with the message "link is broken". Fix: The crawling rules are specified in a way that the mirror item does not get created. Review the crawling rules Page Editor link navigates out of Page Editor Issue: Fix 1: Fix 2: Fix 3: In Page Editor you click a link and you are taken directly to the external page rather than remaining in Page Editor with the external page loaded. A mirror item does not exist for the link. The crawling rules might be configured in a way that prevents a mirror item for being created for this link. Change the crawling rules on the external site definition item so that mirror item is created. The link might use a host name that is different than the primary host name specified on the external site definition item. For example, the primary host name might be " but the link is for "something.com". Add the appropriate host name to the "Additional Host Names" field on the external site definition item. Federated Experience Manager is not recognizing the markup with the link and therefore is not rewriting the link. For more information see section Page 22 of 84

23 2.9.3 Page Editor link navigates to different Sitecore site Issue: Fix 1: Fix 2: In Page Editor you click a link and you are taken to a page on a different Sitecore site. The host name in the URL does not match any external site but the path on the URL can be resolved to an item in the default Sitecore site. The link might use a host name that is different than the primary host name specified on the external site definition item. For example, the primary host name might be " but the link is for "something.com". Add the appropriate host name to the "Additional Host Names" field on the external site definition item. The host name in the URL does not match any external site but the path on the URL can be resolved to an item in the default Sitecore site. Federated Experience Manager is not recognizing the markup with the link and therefore is not rewriting the link. For more information see section Page 23 of 84

24 Chapter 3 External Page Tracking This chapter covers how the tracking non-sitecore web pages works. Page 24 of 84

25 3.1 Tracking Process In order for Sitecore track a view on an external page a link to a Javascript file must be added to the external page. This Javascript file is called the "beacon script". The beacon script is responsible for notifying Sitecore that an external page has been viewed. When a browser makes a request for a page that contains a link to the beacon script, the browser makes a separate request for the beacon script. When the beacon script request is sent, the browser includes information in the request that identifies the URL of the page with the beacon script link. (This is standard browser functionality.) After the browser downloads the beacon script, the browser runs the beacon script. The beacon script makes an AJAX call back to Sitecore for the appropriate mirror item. Sitecore tracks this request just like any other Sitecore request. The following sequence diagram illustrates how the beacon script is generated and how it operates. IIS Sitecore HTTP module beacon script handler pipeline: clientrequest site definition item config standard handler beacon script request beacon script request run handler http context get site info site info beacon script beacon script beacon script beacon script tracking request tracking request run handler track request Page 25 of 84

26 3.2 Beacon Script The beacon script is responsible for sending the tracking request to Sitecore Beacon Script URL The URL for the beacon script is /-/fxm/web. In order to add a link to the beacon script, the following markup must be added to the external pages that need to be tracked: <script type="text/javascript" src="//[host]/-/fxm/web"></script> Beacon Script Generation The beacon script is not a static file. It is dynamically generated for each tracking request. This simplifies the process of adding the beacon script to the external site. You always add exactly the same code to the external site. The beacon script is generated by a custom System.Web.IHttpHandler, which calls the pipeline "fxm.web.clientrequest". Both the handler and the pipeline are defined in the file Sitecore.Fxm.Web.config jquery The beacon script requires jquery. If the beacon script does not detect jquery it will include a reference automatically. Federated Experience Manager uses the following settings to determine where to load the jquery library from when needed. These settings are defined in the file Sitecore.Fxm.Web.config. Setting name Fxm.Web.InternaljQuerySrc Fxm.Web.ExternaljQuerySrc Cross-site Scripting (XSS) Description Location of the jquery library the beacon script loads within Page Editor. Location of the jquery library the beacon script loads for external request. When Sitecore delivers the beacon script, Sitecore is delivering a block of Javascript that is injected into the external page. By definition this is cross-side scripting, though not the malicious kind. Federated Experience Manager uses CORS to enable cross-site scripting. Page 26 of 84

27 3.3 External Sites An external site is a website that is tracked and personalized using the beacon script Defining an External Site Within Sitecore each external site is represented by a Sitecore item defined under /sitecore/content/federated Experience Manager. External site items are also called site definition items or mirror site items. Site definition items are based on the template /sitecore/templates/federated Experience Manager/Web/External Site Template: External Site This section describes the fields on the template. Field name Path Protocol Primary Host Name Additional Host Names Default Documents Site Resolver Site Name Common Web Tracking Rules Common Tracking Settings Web Tracking Rules Tracking Settings Crawler Crawl Root Path Description Read-only field that indicates the path of the page on the external site that was crawled. The crawler uses this protocol when it crawls the external site. The crawler uses this host name when it crawls the external site. The mirror provider uses this value in order to resolve the site from a host name. In addition to the Primary Host Name, the mirror provider uses the values in this field in order to resolve the site from a host name. The mirror provider uses these values when resolving the mirror item. The site resolver that should be used for the external site. For more information on site resolvers see section 3.4. Name of the Sitecore site used for handling and tracking requests for the external site. This value is generated when the item is created and is read-only. Changing this value can result in tracking data being corrupted. The web tracking rules that apply to all mirror items in the site. For more information on web tracking rules see section 4.1. The tracking settings that apply to all mirror items in the site. For more information on tracking settings see section 4.2. The web tracking rules that apply only to the root mirror item. For more information on web tracking rules see section 4.1. The tracking settings that apply only to the root mirror item. For more information on tracking settings see section 4.2. The crawler that is used for the site. For more information crawlers see section 5.3. Path on the external site that the crawler starts from. If this field does not have a value the crawler will start from "/". Page 27 of 84

28 Field name Crawling Rules Template Definition Rules Tracking Configuration Rules Default Language Language Resolver Common Placeholder Selectors Placeholder Selectors Description The crawling rules that are used during the page crawling process. For more information on the crawling rules see section 5.8. The rules that determine which template is used for mirror items during the page crawling process. The rules that determine the tracking settings that are assigned to the mirror items during the page crawling process. The language that is used for mirror items if the language resolver does not match a language for the request. The language resolver that is used for the site. For more information on language resolvers see section 5.6. The placeholder selectors that apply to all mirror items in the site. For more information on placeholder selectors see section 7.2. The placeholder selectors for the root mirror item. For more information on placeholder selectors see section 7.2. Page 28 of 84

29 3.4 Site Resolvers Each site definition item is a separate Sitecore site. This allows tracking requests for Sitecore sites to be differentiated from external sites. A site resolver is responsible for setting properties on the Sitecore site Site Resolution Process The following sequence diagram illustrates how the site resolution process works: Site Manager Mirror Site Provider Mirror Manager Mirror Provider Site Definition Item Site Resolver site name site name get sites def items get site def items site def items site def items loop [each site def item] get site get site site site site site Note: This sequence diagram does not include the steps by which the mirror site provider resolves other (non-external) sites. The mirror provider configuration allows you to specify the site provider type to use for other sites. For more information on the mirror site provider configuration see section Mirror Site Provider The mirror site provider is a Sitecore Site Provider that exposes site definition items as Sitecore sites. The mirror site provider is configured in the config file Sitecore.Fxm.Web.config. The following values are specified via the config file: Page 29 of 84

30 Name Type Description Impl Constructor parameter The Sitecore.Sites.SiteProvider type that should be used to resolve non-fxm sites. Database Property The Sitecore database where the site definition items are defined for the delivery environment. This will usually be the "web" database Specifying a Site Resolver Each site definition item may have one site resolver assigned to it. A single site resolver may be assigned to multiple site definition items Defining a Site Resolver Each site resolver is represented by a Sitecore item defined under /sitecore/system/modules/federated Experience Manager/Web/Site Resolvers. Site resolver definition items are based on the template /sitecore/templates/federated Experience Manager/Web/SiteResolver Template: Site Resolver This section describes the fields on the template. Field name Type Site Parameters Description The fully-qualified name of the type that implements the site resolver. Parameters that are passed to the Sitecore.Sites.Site constructor. Page 30 of 84

31 Chapter 4 Dynamic Tracking With standard Sitecore tracking options (profiling, goals, campaigns, etc.) are assigned to individual Sitecore items. This is a very static means for assigning tracking settings. Federated Experience Manager provides a number of options for dynamically assigning tracking settings. This chapter covers the options that are available. Page 31 of 84

32 4.1 Web Tracking Rules Web tracking rules involve using the Sitecore rules engine to determine which tracking options are recorded when an external page is viewed. Web tracking rules are implemented additions to the pipeline "starttracking" that are included in the config file Sitecore.Fxm.Web.config Site-wide Web Tracking Rules Site-wide web tracking rules are applied to all tracking requests within a specific site. These tracking rules are defined using the field "Common Web Tracking Rules" on the external site definition item Page-specific Web Tracking Rules Page-specific web tracking rules are applied to all tracking requests for a specific mirror item. These tracking rules are defined using the field "Web Tracking Rules" on the mirror item definition item Conditions & Actions By default the following conditions and actions are available: Grouping FXM Page Section 9.3 FXM Tracking Section 9.5 More information available in Page 32 of 84

33 4.2 Tracking Settings Tracking settings allow you to separate the definition of tracking options from the application of tracking options. The advantage is it makes it easier to assign and manage tracking options that are used by multiple items. An example is the easiest way to explain how this works. Consider an environment where 3 external sites are tracked using the "New Mother Mary" persona. There are pages on each of the 3 external sites that need to be profiled for this persona. You can create a new Tracking Settings item and assign the tracking options for this persona on this item. Then you assign the Tracking Settings to the pages on the 3 external sites that you want to track using the settings you assigned. When a visitor views any of those pages on the 3 external sites the tracking options assigned to the Tracking Settings item will be recorded in Sitecore Defining Tracking Settings Tracking settings items are defined under the item /sitecore/system/modules/federated Experience Manager/Web/Tracking Settings. The items defined here can be selected as site-wide tracking settings (see section 4.2.2) and as page-specific tracking settings (see section 4.2.3) Site-wide Tracking Settings Tracking settings can be assigned to a specific site using the field "Common Tracking Settings" on the external site definition item. For information on how to define tracking settings items see section Page-specific Tracking Settings Tracking settings can be assigned to a specific page using the field "Tracking Settings" on the mirror item that corresponds to the external page. For information on how to define tracking settings items see section Page 33 of 84

34 4.3 Content Tracking Rules The site-wide and page-specific options allow you to select the items whose tracking options you want to use when a mirror item is viewed. But what if you need something more dynamic? For example, consider an external ecommerce site where you display thousands of products. You track each product detail page. The tracking options for each product detail page depend on the product's category. Content tracking rules are designed to support this sort of requirement. Content tracking rules are applied during the standard Sitecore tracking process. Note: Content tracking rules do not only apply to mirror items. They apply to all items that are tracked by Sitecore. This is because the global tracking rules are applied via an addition to the pipeline "starttracking" that is included in the config file Sitecore.Fxm.ContentTracker.config Specifying Content Tracking Rules Content tracking rules are defined under the item /sitecore/system/settings/rules/federated Experience Manager/Content Tracker/Content Tracking/Rules Conditions & Actions By default the following conditions and actions are available: Grouping FXM Content Tracking Section 9.2 More information available in Page 34 of 84

35 Chapter 5 Crawling Process This chapter covers how Federated Experience Manager manages the creation and maintenance of mirror items. It focuses on the logic that determines when and how mirror items are created and updated. Page 35 of 84

36 5.1 Overview In Federated Experience Manager each external site is represented by a site definition item. The site definition item has a crawler assigned to it. The crawler is responsible for making requests to the external site and using the response to create and update mirror items based on the site definition item's configuration. The crawling process can run in two modes. Website crawling mode includes the tasks of building an HTTP request, transmitting the request and handling the response. Page crawling mode assumes that the HTTP response is already available. It focuses on handling the response. Page 36 of 84

37 5.2 Website Crawling Process As its name suggests, the website crawling process involves crawling a website. This involves a lot of I/O operations. This is a potentially resource and time intensive task. As a result, the website crawling process runs as a job. The following sequence diagram illustrates how the website crawling process operates. Crawler Manager Crawler Provider Job Manager pipeline: websitecrawling Crawler external site pipeline: createmirroritem run crawler job for site run crawler job for site run website crawler pipeline run start crawling from site definition item request page handle page handle page handle page mirror item mirror item Crawler Manager The website crawling process is started using the "RunCrawlerJobFor" method on the Crawler Manager. This method calls the similarly-named method on the Crawler Provider Crawler Provider The "RunCrawlerJobFor" method on Crawler Provider creates and starts the job that runs the website crawling process. The method that is run when the job is executed is the "Crawl" method. This method runs the pipeline "fxm.web.websitecrawling" that is defined in the config file Sitecore.Fxm.Web.config Pipeline: fxm.web.websitecrawling This pipeline is responsible for determining which crawler to use, initializing the crawler and running the crawler. Processors for this pipeline must inherit from the class Sitecore.Fxm.Web.Pipelines.WebsiteCrawling.WebsiteCrawlingProcessor. Page 37 of 84

38 5.3 Crawlers Crawlers are used in the website crawling process. A crawler is responsible for making a request for an external page and passing the results to the page crawling process Specifying a Crawler Each site definition item may have one crawler assigned to it. A single crawler may be assigned to multiple site definition items Defining a Crawler Each crawler is represented by a Sitecore item defined under /sitecore/system/modules/federated Experience Manager/Web/Crawlers. Federated Experience Manager includes a crawler that uses the Abot C# Web Crawler. This is an open source,.net-based crawler. For more information on the Abot crawler see This crawler is configured using a definition item based on the template /sitecore/templates/federated Experience Manager/Web/Crawlers/AbotCrawler. Each field on this template represents a setting on the Abot crawler. This template has the following fields that can be configured: Field name Type Description The fully-qualified name for the type that implements the crawler. This value does not need to be changed. The follow template fields correspond to properties on the Abot.Poco.CrawlConfiguration instance that is used by the crawler: Field name Property name Default value Max Concurrent Threads MaxConcurrentThreads 2 Max Pages To Crawl MaxPagesToCrawl Max Pages To Crawl Per Domain MaxPagesToCrawlPerDomain 0 Max Page Size MaxPageSizeInBytes 0 Max Crawl Depth MaxCrawlDepth 100 Min Crawl Delay Per Domain MinCrawlDelayPerDomainMilliSeconds 0 Is Uri Recrawling Enabled IsHttpRequestAutoRedirectsEnabled true Is External Page Crawling Enabled HttpRequestMaxAutoRedirects 7 User Agent String HttpRequestTimeoutInSeconds 30 Downloadable Content Types CrawlTimeoutSeconds 0 Http Service Point Connection Limit Is External Page Links Crawling Enabled Is Http Request Automatic Recompression Enabled HttpServicePointConnectionLimit 0 IsExternalPageCrawlingEnabled IsExternalPageLinksCrawlingEnabled true true Page 38 of 84

39 Field name Property name Default value Is Http Request Auto Redirects Enabled Http Request Max Auto Redirects Http Request Timeout - in seconds IsHttpRequestAutomaticDecompressionEnabled IsRespectRobotsDotTextEnabled IsUriRecrawlingEnabled Crawl Timeout - in seconds MaxMemoryUsageCacheTimeInSeconds 0 Robots Dot Text User Agent String Max Robots Dot Text Crawl Delay In Seconds Is Respect Robots Dot Text Enabled Max Memory Usage Max Memory Usage Cache Time Min Available Memory Required MaxMemoryUsageInMb 0 MaxRobotsDotTextCrawlDelayInSeconds 0 MinAvailableMemoryRequiredInMb 10 DownloadableContentTypes RobotsDotTextUserAgentString UserAgentString true true true Page 39 of 84

40 5.4 Page Crawling Process The page crawling process is responsible for handling the data retrieved by the crawler in order to create or update a mirror item. The following sequence diagram illustrates how the page crawling process operates. Crawler Manager Crawler Provider pipeline: createmirroritem run crawler for page run crawler for page create mirror item The page crawling process runs under the following conditions: When a content author is viewing a mirror item in Page Editor and clicks a link. FXM includes a custom EditorLinkProvider which triggers the page crawling process. When crawler (meaning an instance of IWebCrawler) handles a URL. When a tracking request triggers the crawling Crawler Manager The page crawling process is started using the "CrawlPage" method on the Crawler Manager. This method calls the similarly-named method on the Crawler Provider Crawler Provider The "CrawlPage" method on Crawler Provider calls the "CreateMirrorItem" method on the Mirror Manager Mirror Manager The "CreateMirrorItem" method on the Mirror Manager calls the similarly-named method on the Mirror Provider Mirror Provider The "CreateMirrorItem" method on Mirror Provider runs the pipeline "fxm.web.createmirroritem" that is defined in the config file Sitecore.Fxm.Web.config Pipeline: fxm.web.createmirroritem This pipeline is responsible for controlling the mirror item creation process. The mirror item creation process is defined in detail in section 5.5. Processors for this pipeline must inherit from the class Sitecore.Fxm.Web.Pipelines.CreateMirrorItem.CreateMirrorItemBase. Page 40 of 84

41 5.4.6 Pipeline: fxm.web.resolvemirrortemplate This pipeline is responsible for determining which template should be used during the crawling process to create the mirror item. The mirror item creation process is defined in detail in section 5.5. Processors for this pipeline must inherit from the class Sitecore.Fxm.Web.Pipelines.ResolveMirrorTemplate.ResolveMirrorTemplateProcessor. Page 41 of 84

42 5.5 Mirror Item Creation Process The process of creating mirror items involves using information about an external page to create an appropriate representation of the external page in Sitecore. This includes things like determining the template to use to create the mirror item, the name for the mirror item and which tracking options should be assigned to the mirror item This process includes more than just creating new mirror items. It also handles updating mirror items that already exist. The mirror item creation process is handled by the pipeline "fxm.web.createmirroritem" that is defined in the config file Sitecore.Fxm.Web.config. This section describes the significant parts of this pipeline Language Resolution In order to be able to accurately track multi-language external sites, Federated Experience Manager must be able to recognize when multiple external URLs represent the same content, only in different languages. This logic is handled by the "ResolveLanguage" processor. This processor looks to see if the site definition item has a language resolver assigned to it. This is done by checking the value of the "Language Resolver" field. If a value is specified, the processor will use the corresponding language resolver. If no language resolver is specified, the value of the "Default Language" field on the site definition item is used. If no value is specified here, either, then Sitecore uses the context language. For information on how to configure language resolvers see section Generate Mirror Item Path The mirror item may or may not exist. This processor determines where the mirror item should exist by using the method "GetPagePaths" from the crawler provider Filter Crawling Pages The site definition item defines the root URL for the crawler. The crawler itself, however, may pick up URLs outside of the root. This processor ensures that mirror items are only created for URLs that are under the root URL Run Crawling Rules The creation of a mirror item is controlled by crawling rules. The processor "RunCrawlingRule" is responsible for executing the crawling rules. Information on how to configure crawling rules is available in section 5.8. Page 42 of 84

43 5.6 Mirror Items A mirror item represents an external web page within Sitecore. Tracking and personalization can be configured on this item Defining a Mirror Item Mirror items are created during the page crawling process. The template that is used depends on the template definition rules that are configured for the site. These rules are optional, however. If no template definition rules are applied, or if none are specified, the mirror item is based on the template /sitecore/templates/federated Experience Manager/Web/External Page Template: External Page This section describes the fields on the template. Field name Path Web Tracking Rules Tracking Settings Placeholder Selectors Description The path of the external page. This value is set during the crawling process. Web tracking rules that are applied when the tracking request is handled. For more information see section Tracking settings that are used when the tracking request is handled. For more information see section Placeholder selectors that are For more information see section 7.2. Page 43 of 84

44 5.7 Language Resolvers A language resolver parses a URL in order to determine which language a URL corresponds to. Language resolvers allow a single Sitecore item to correspond to multiple URLs in order to represent the same content in different languages Specifying a Language Resolver Each site definition item may have one language resolver assigned to it. A single language resolver may be assigned to multiple site definition items Defining a Language Resolver Each language resolver is represented by a Sitecore item defined under /sitecore/system/modules/federated Experience Manager/Web/Language Resolvers. Federated Experience Manager includes two language resolvers. Each language resolver has its own template in order to support its unique configuration options. These templates are defined under /sitecore/templates/federated Experience Manager/Web/Language Resolvers Template: QueryString Resolver This language resolver can determine the language from a value in a URL's query string. This type of language resolver is configured using the template /sitecore/templates/federated Experience Manager/Web/Language Resolvers/QueryStringResolver. This template has the following fields that can be configured: Field name Description Sample value Type Language Mapping Parameters The fully-qualified name for the type that implements the language resolver. This value does not need to be changed. A set of key-value pairs where the key is the value from the query string parameter and the value is the regional ISO code for the language item in Sitecore A comma-separate list of query string parameters whose values should be parsed Template: URL Segment Resolver fr de lang,language This language resolver can determine the language from a segment of a URL. fr-fr de-de This language resolver assumes that the language code is located in the first URL segment. For example, if you consider the URL the resolver will use "fr" for the language code. This type of language resolver is configured using the template /sitecore/templates/federated Experience Manager/Web/Language Resolvers/URL Segment Resolver. Page 44 of 84

45 This template has the following fields that can be configured: Field name Description Sample value Type Language Mapping The fully-qualified name for the type that implements the language resolver. This value does not need to be changed. A set of key-value pairs where the key is the value from the query string parameter and the value is the regional ISO code for the language item in Sitecore fr de fr-fr de-de Page 45 of 84

46 5.8 Crawling Rules The page crawling process is controlled by crawling rules. These rules allow you to configure the page crawling process to work the way you need it to work without requiring any custom code. Crawling rules are executed during the mirror item creation process, specifically in the pipeline "fxm.web.createmirroritem" that is defined in the config file Sitecore.Fxm.Web.config Specifying Crawling Rules Each site definition item has a "Crawling Rules" field. This field determines which crawling rules should be used for the site. Each site definition item may have multiple crawling rules assigned to it. Since crawling rules are configured on the site definition item a crawling rule may not be used by multiple sites Conditions & Actions By default the following conditions and actions are available: Grouping FXM Page Section 9.3 FXM Web Crawling Section 9.7 More information available in Page 46 of 84

47 5.9 Tracking Configuration Rules When a mirror item is created or updated you may be able to determine how the mirror item should be tracked. For example, if the external page has a meta-tag that identifies the page as a product page you may want to assign a specific profile card to the mirror item. Tracking configuration rules allow you to configure this sort of tracking. Tracking configuration rules are executed when a mirror item is created or updated. Specifically they are executed by the "create or update mirror item" action that can be configured in the crawling rules Specifying Tracking Configuration Rules Each site definition item has a "Tracking Configuration Rules" field. This field determines which tracking configuration rules should be used for the site. Each site definition item may have multiple tracking configuration rules assigned to it. Since tracking configuration rules are configured on the site definition item a tracking configuration rule may not be used by multiple sites Conditions & Actions By default the following conditions and actions are available: Grouping FXM Page Section 9.3 FXM Tracking Configuration Section 9.6 More information available in Page 47 of 84

48 5.10 Page Crawling & Page Editor A content author is able to navigate an external site using Page Editor for the purpose of configuring page-specific tracking options and personalization. Page Editor displays a Sitecore item. Therefore, in order for a content author to be able to view an external page within Page Editor, the mirror item for the external page must exist. This section describes how the page crawling process can be triggered from within Page Editor. By default this functionality is enabled. Instructions on how to disable it are included in section Page Editor Layout All mirror items have the layout /sitecore/layout/layouts/federated Experience Manager/Page Editor assigned for the default device. This layout controls the process needed in order to display the external page within Page Editor. The following sequence diagram describes the process: Page Editor layout OnLoad Mirror Item pipeline: rewritehtmldocument external page get rendered page get path field value path field value run HTTP request HTTP response rewrite links & insert placeholders controls rendered page Link Rewriting When the external page is displayed in Page Editor it is likely the external page will contain links to other pages on the external site. Those links must be rewritten so that that when the content author clicks the link he remains in Page Editor. Link rewriting is handled by the pipeline rewritehtmldocument, specifically the processor Sitecore.Fxm.Web.Pipelines.RewriteHtmlDocument.RewriteHyperlinks. An important part of the link rewriting logic is to inject jquery code that changes what happens when the link is clicked. A call to a custom HTTP handler is added. This handler, called the link handler, tries to ensure that a mirror item exists for the link. Page 48 of 84

49 The following sequence diagram describes the process: javascript IIS Sitecore HTTP module link handler Editor Link Manager Editor Link Provider external page page crawling process click event external URL external URL external URL get path field value path field value run HTTP request HTTP response crawl page request page crawl response JSON JSON JSON JSON JSON Page 49 of 84

50 5.11 Page Crawling & External Page Views It is possible to configure Federated Experience Manager so that an external page view will trigger the page crawling process. This is called ad hoc mirror item creation. By default this functionality is disabled. Instructions on how to enable it are included in section When ad hoc mirror item creation is enabled the page crawling process is triggered by the tracking request. As explained in section 3.1 the tracking request is a request that is handled by the standard Sitecore request handling process. Federated Experience Manager adds a step to the standard request handling process. The processor only runs if the request is a tracking request. In that case it sends a request to the external site and then uses the response to trigger the page crawling process. The following sequence diagram shows the request handling process for a tracking request when ad hoc mirror item creation is enabled: IIS Sitecore HTTP module Nexus module pipeline: httprequestbegin processor: CrawlPage external site page crawling process tracking request tracking request run handler http context http context request response page crawl request mirror item Page 50 of 84

51 Chapter 6 Running the Crawling Process This chapter covers the different options available for starting, stopping and monitoring the crawling process. Chapter 5 provides details on how the crawling process works and how it is configured. Page 51 of 84

52 6.1 Comparison of Options The following table provides a comparison of the different options for running the website crawling process. Requirement Only pages that are manually selected by a content author should be tracked Bulk Website Crawler Manual Page Editor Ad Hoc Page View Whether or not to track a page can be specified using rules Only pages viewed by a visitor should be tracked A large number of pages should be tracked Minimize processing on content delivery (CD) environment Automate the assignment of profiling/goals/page events to pages Manually assign profiling/goals/page events to pages Apply personalization to pages Activity on external site must not affect the content management (CM) environment The content delivery environment (CD) has no access to the content management (CM) server Page 52 of 84

53 6.2 Bulk Mirror Item Creation Bulk mirror item creation involves running the website crawling process. For information on how the website crawling process works see Chapter Starting the Process Manually To Start run the website crawling process: 1. In Content Editor navigate to /sitecore/content/federated Experience Manager 2. Select the "Web Crawler" tab 3. Find the row that represents the process you want to start 4. Click CRAWL NOW Monitoring the Process The website crawling process can be monitored using the "Web Crawler" tab on the item /sitecore/content/federated Experience Manager. Note: You should not expect the number of Crawled Pages to match the number of Saved Pages. Crawled Pages is the total number of requests the crawler made to the external site. Not all of those requests result in an item being saved. For example, the External Site item is also a mirror item. That item may not be changed during the crawling process, therefore the item will not be saved. This is one page that was crawled and not saved Stopping the Process Manually To stop run the website crawling process: 1. In Content Editor navigate to /sitecore/content/federated Experience Manager 2. Select the "Web Crawler" tab 3. Find the row that represents the process you want to stop 4. Click STOP CRAWL Page 53 of 84

54 6.2.4 Starting the Process Programmatically The Crawling Manager is used to start the website crawling process. The following code demonstrates how to use it: var item = //get the site definition item var runningfromscheduler = false; //this determines whether errors are displayed //false means errors are displayed via popup //true means errors are not displayed Sitecore.Fxm.Web.Crawler.CrawlerManager.RunCrawlerJobFor(item, runningfromscheduler); Scheduling the Process If you need to schedule the website crawling process you can use the code from section to create a task. The following is an example of the class that implements the task: using System; using System.Collections.Generic; using System.Linq; using System.Web; using Sitecore.Data; using Sitecore.Fxm.Web.Crawler; using Sitecore.SecurityModel; namespace Testing.Fxm { public class WebsiteCrawlTask { public WebsiteCrawlTask(string databasename, string sitedefitemid) { this.database = Database.GetDatabase(databaseName); this.id = ID.Parse(siteDefItemId); } public Database Database { get; private set; } public ID ID { get; private set; } } } public void Run() { using (new SecurityDisabler()) { var item = this.database.getitem(this.id); CrawlerManager.RunCrawlerJobFor(item, true); } } The following is an example of the configuration that is required in order for Sitecore to recognize the task: <configuration xmlns:patch=" <sitecore> <scheduling> <agent type="testing.fxm.websitecrawltask, Testing.Fxm" method="run" interval="00:01:00"> <param desc="database name">master</param> <param desc="site definition item ID">{6652CD1D ADDA- 261F32BDC69C}</param> </agent> </scheduling> </sitecore> </configuration> Page 54 of 84

55 6.3 Manual Mirror Item Creation Manual mirror item creation is performed using Page Editor. Provided this feature is enabled, the act of navigating an external site within Page Editor will trigger the mirror item creation process. This process is explained in section By default manual mirror item creation is enabled Disabling To disable manual mirror item creation you must change the configuration of the crawling rules on the site definition item. The following is an example of a rule that will disable manual mirror item creation: Page 55 of 84

56 6.4 Ad Hoc Mirror Item Creation Ad hoc mirror item creation is the result of an external page being viewed. Provided this feature is enabled, the act of viewing an external page will trigger the mirror item creation process. This process is explained in section By default ad hoc mirror item creation is disabled Enabling To enable ad hoc mirror item creation you must change the configuration of the crawling rules on the site definition item. The following is an example of a rule that will enable ad hoc mirror item creation. Page 56 of 84