Enesys IS Data Extension Manual 1.5

Transcription

1 1.5 Integration Services Data Flow Components for SharePoint ENESYS

2 Integration Services Data Flow Components for SharePoint 2

3 Summary ENESYS IS DATA EXTENSION MANUAL... 2 ABOUT ENESYS IS DATA EXTENSION... 5 INSTALLATION AND UPGRADE... 6 SYSTEM REQUIREMENT... 6 INSTALLATION... 6 OVERVIEW... 6 ENESYS IS DATA EXTENSION INSTALLATION... 6 UPGRADING A PACKAGE TO A HIGHER VERSION OF INTEGRATION SERVICES...15 WHAT S NEW...20 VERSION SHAREPOINT 2013 AND SQL SERVER 2012 COMPATIBILITY...20 LOOKUP-MULTI AND PROJECTION...20 SUPPORT FOR MULTI-VALUE PARAMETERS IN THE QUERY DESIGNER...22 VERSION MERGING LISTS ENUMERATED IN A RESULT SET...23 IMPROVED XMLQUERY STATEMENT...24 FILTERING BASED ON A RESULT SET...26 RETRIEVING LIST OF CHOICES...26 SUPPORT FOR INTEGRATION SERVICES QUERY DESIGNER IMPROVEMENTS...27 OTHER IMPROVEMENTS...33 VERSION NEW INFOPATH FEATURES...33 ITEM VERSIONS PERFORMANCE BOOST...35 REFERENCE LIST FOR ROLLUPLISTS STATEMENTS...35 QUERY DESIGNER IMPROVEMENTS...35 OTHER IMPROVEMENTS...43 VERSION RETRIEVING PARSED SURVEY DATA...43 QUERYING ORACLE DATABASES...44 QUERY DESIGNER - GREAT INTELLISENSE IMPROVEMENTS...46 USING PARAMETERS AND VARIABLES FOR SPECIFYING SHAREPOINT LIST URL AND TITLE...51 QUERY DESIGNER USER INTERFACE IMPROVEMENTS...53 OTHER IMPROVEMENTS...55 VERSION USING OBJECT MODEL FOR ROLLING UP DATA...55 SUPPORT FOR FORMS BASED AUTHENTICATION...56 SPECIFYING FIELDS FOR WHICH TO RETRIEVE ITEM VERSIONS...56 USING DATA SOURCE VARIABLES INSIDE THE SQLSERVERQUERY S CONNECTION STRING...56 ADVANCED PROPERTIES FORM...57 BUG CORRECTIONS...59 VERSION NEW CONNECTION STRING OPTIONS...59 NEW FORM FOR MANAGING CONNECTION MANAGERS

4 OTHER IMPROVEMENTS...61 VERSION 0.9 (RC2)...61 NEW SYNCHRONIZATION SOURCE COMPONENT...61 NEW MERGE DESTINATION COMPONENT...62 SUPPORT FOR INTEGRATED SECURITY...62 INTELLISENSE FOR SQLQUERY STATEMENTS...63 OTHER CHANGES...63 VERSION 0.9 (RC1)...63 QUICK START...64 CREATE A NEW INTEGRATION SERVICES PROJECT...64 ADD A DATA FLOW TASK...65 ADD AN ENESYS IS CONNECTION MANAGER...65 ADD AN ENESYS SHAREPOINT SOURCE COMPONENT...68 ASSOCIATE A CONNECTION MANAGER TO THE COMPONENT...69 DESIGN THE QUERY...70 MAP THE PARAMETERS...72 ADD A DESTINATION COMPONENT...74 EXECUTE THE DATA FLOW...79 SYNCHRONIZING SHAREPOINT LIST WITH SQL SERVER...80 ADD A DATA FLOW TASK...81 ADD AN ENESYS SYNCHRONIZATION SHAREPOINT SOURCE COMPONENT...81 SELECT THE CONNECTION MANAGERS...81 DESIGN THE QUERY...83 MAP THE PARAMETERS...85 ADD AN ENESYS MERGE DESTINATION COMPONENT...86 SELECT THE CONNECTION MANAGER...86 CREATE THE DESTINATION SQL TABLE...87 FINALIZE THE CONFIGURATION...89 EXECUTE THE PACKAGE...89 PARSING SYNCHRONIZED ITEMS...91 INFORMATION ABOUT THE SYNCHRONIZATION COMPONENT...91 THE COLUMNS...91 THE DATA...91 STEPS TO EXECUTE...92 INFORMATION...92 EXCEPTIONS...92 EXAMPLE WITH THE MERGE DESTINATION COMPONENT...93 FULL SYNCHRONIZATION TRIGGERS...94 USING THE OBJECT MODEL FOR ROLLING UP DATA...95 ENABLING THE OBJECT MODEL...95 THE ROLLUPLISTS OBJECT MODEL METHODS...95 METHOD SELECTION WORKFLOW...97 GENERAL GUIDELINES FOR CHOOSING A METHOD...98 FORMS BASED AUTHENTICATION

5 About Enesys IS Data Extension Enesys IS Data Extension is a product that provides Data Flow components for Microsoft SQL Server Integration Services that lets you build enterprise-level data integration and transformation from your SharePoint data. Enesys IS Data Extension lets you build sophisticated queries over any SharePoint list. It provides a Visual Query Designer that is completely integrated within Visual Studio. Enesys IS Data Extension uses the same query engine and designer as Enesys RS Data Extension. It is the ideal companion product when you need to deal with very large data sets and when you need to integrate SharePoint data to a common Data Warehouse. 5

6 Installation and upgrade System Requirement Software Microsoft SQL Server Integration Services Standard or Enterprise version 2005, 2008, 2008 R2 or MOSS 2007, WSS V3 and SharePoint Business Intelligence Development Studio Description Enesys IS Data Extension (EISDE) must be installed on an Integration Services server in order to run packages using EISDE Source Component. Please note that EISDE is not compatible with SQL Server Express or Workgroup editions. Enesys IS Data Extension is compatible with both 32bits and 64bits version of SQL Server 2005/2008/2008 R2 and Enesys IS Data Extension will let you build and run Integration Services packages over MOSS 2007, WSS V3 or SharePoint Packages are built using Microsoft Business Intelligence Development Studio which is a subset of Visual Studio 2005 (2008/2010). BIDS is provided as part of SQL Server installation. It can be installed along an existing Visual Studio 2005 (2008/2010) installation and will complement it with additional project types including Integration Services projects. With Integration Services 2012, the new name of BIDS is SQL Server Data Tools (SSDT), for simplicity, only the name BIDS will be used inside the documentation. Installation Overview In order to design and run packages from Business Intelligence Development Studio (BIDS - official Microsoft terminology for the package designer) and run packages from the Integration Services server, Enesys IS Data Extension will need to be installed and configured. The installation is done using the Enesys IS Data Extension installer that will install the necessary assemblies for designing and running packages using either BIDS or Integration Services. The Enesys IS Configuration tool allows you to configure the license key, the culture used inside the designer and the SPLogin credentials. Enesys IS Data Extension installation Enesys IS Data Extension is provided as an EXE package. You will need to run our installation program on your server in order to configure your Integration Services server. You will also need to run our installation program on any machine that will be used to build packages using Microsoft package designer (BIDS). 6

7 Please note that there are three versions of Enesys IS Data Extension, one is for SQL Server 2005, one is for SQL Server 2008 (and 2008 R2) and the last one is for SQL Server Download and launch the proper version depending on your SQL Server version. Here is a summary of the Installation steps: 1. Launch EnesysISDataExtension.exe To start the installation click Next. 7

8 2. License agreement Read the license agreement and check I accept the terms in the License Agreement if you accept the terms. Click Next to continue. 3. Select the features and installation location 8

9 All features are selected by default. In case you are installing the product on your server, you may want to uncheck the Designer feature, in any other case, we recommend to let them all checked. Feature Enesys IS Data Extension Runtime Designer Description Installs core Enesys IS Data Extension files to the specified location (including a configuration tool for setting the license key). Installs assemblies needed to run Enesys IS Data Extension (needed on both servers and workstations). Installs assemblies needed to design packages that use Enesys IS Data Extension (only needed on workstations). 4. Add shortcuts On the next page, you will be able to decide whether you want to add shortcuts in the current user program menu folder. We recommend adding shortcuts for easily accessing configuration tool. When checked, the following shortcuts will be installed: 5. Configure the License Key At the end of the installation, you will be proposed to launch the Enesys IS Configuration tool. 9

10 It is necessary to configure your installation using this program if this is the first time you install Enesys IS Data Extension, as you will need to configure the License Key. Enesys IS Configuration tool lets you set the License Key (needed to design and run packages), the language (used in the designer) and the SPLogin credentials (used with rolluplists statements). 10

11 You may change the following elements: Change product key Click on the Change License Key button. Enter your new license key and click on OK. Change the Query Designer language Choose the language you want to use inside the Query Designer and it will be directly saved. We currently support English, French and Russian languages. If you want to translate the product in another language, please contact us using our support address and we will give 11

12 you the data to translate. Once translated, we would be pleased to add it to the next version of our product. Configure SPLogin credentials If you want to configure or to change your SPLogin credentials, click on the Change SPLogin button. Enter the user s login and password that will be used to enumerate SharePoint sites when using rolluplists statement. Note: 6. Installing the component in the Toolbox This step is no longer required with SSIS 2012 as there is a new SSIS Toolbox tab that automatically lists all installed components. In design-mode, you will need to add the three data flow components of Enesys IS Data Extension to the toolbox inside your BIDS environment. 1) Open BIDS and create a new (or open an existing) Integration Services project 2) Drag a Data Flow Task from the Toolbox to the opened Package 3) Double click on the Data Flow Task to open the Data Flow tab 12

13 4) Right click in the General tab inside the toolbox and click on Choose Items 5) In the Choose Toolbox Items dialog window, select the SSIS Data Flow Items tab. Check the Enesys SharePoint Source, Enesys Synchronization SharePoint Source and Enesys Merge Destination lines and click on OK to confirm. 13

14 Note: The Enesys Merge Destination component is not available for SQL Server ) Move the two source components to the Data Flow Sources tab, and move the destination component to the Data Flow Destinations tab, as shown in the following screenshot: 14

15 Note: You will have to delete and add the components to the toolbox for every new version of the components. Upgrading a package to a higher version of Integration Services Before upgrading a package that uses Enesys IS Data Extension, we recommend you to follow these steps, in order to ensure that the package is properly upgraded. 1) Install the latest version of Enesys IS Data Extension for both versions of Integration Services (2005 and 2008, or 2005 and 2012, or 2008 and 2012). It is important that the two versions of Enesys IS Data Extension are the same. 2) Open the package(s) you want to upgrade using the previous version of BIDS (2005 or 2008), make a small modification and save the package. It will ensure that the correct version number is stored in the package XML for the Enesys IS Data Extension components. 15

16 Once you have done this, you can open the package using the newer version of BIDS (2008/2012). The Visual Studio Conversion Migration Wizard will convert the package to use the newer version of SQL Server and use the correct Enesys IS Data Extension version. Click on Next to launch the conversion of the project. 16

17 It will ask you to upgrade the packages: Set the Package password if needed and click on Next. 17

18 The next page allows you to change the options of the package upgrade: Choose your options and click on Next. After reviewing the selected options, click on Finish to launch the package upgrade. 18

19 Click on Close to finish the package upgrade and start using your package with BIDS

20 What s new Version 1.5 SharePoint 2013 and SQL Server 2012 compatibility Enesys IS Data Extension 1.5 is compatible with SharePoint 2013 and SQL Server Lookup-multi and projection Projected fields are now properly handled when expanding a multi-value lookup field. 20

21 When expanding a multi-value lookup column (see the screen shot below) meaning breaking down multiple values into single value by duplicating each item as if you were doing a join between the item and the lookup values -, Enesys IS Data Extension 1.4 and earlier will just duplicate projected fields value like with any other fields instead of retrieving the value that matches the single value in the target list. 21

22 Enesys IS Data Extension 1.5, provides support for projected fields and will automatically extract the single value that matches the lookup value item. Support for multi-value parameters in the query designer Even though Enesys IS Data Extension queries support multi-value parameters (using IN either at the list level or within a SqlQuery statement), it was only possible to pass a single parameter values when executing the query within the query designer for testing purposes. 22

23 To make it easier, it is now possible to pass multiple values directly within the query designer by using a comma to separate each value like shown in the following screen shot: Version 1.4 Merging lists enumerated in a result set Enesys IS Data Extension offers a brand new statement that allows you to merges the results of several SharePoint lists enumerated in a result set. This result set can be any other statement, as long as it returns a list of title/relativesiteurl couples (e.g.: the alistcollection statement). For example, this allows you to merge all the documents from a whole site, or even site collection. 23

24 For more information about the listsmerge statement, please see the Merging a variable number of lists chapter inside the Enesys SharePoint Query Designer manual. Improved XmlQuery statement Enesys IS Data Extension now has an improved version of the XmlQuery statement. With previous versions, you were bound to use an XPath formula to specify the data you wanted to retrieve from the file. While this is satisfying most of the time, you may need to retrieve repeating data along with some other data inside the file, as this is possible with the InfoPath statement. Starting with Enesys IS Data Extension 1.4, you can now choose between XPath and the custom ElementPath formula that is used in InfoPath statements. This greatly improves the possibilities of the XmlQuery statement. 24

25 To help you build your XPath formulas, we now provide you with the XML schema (generated from the file / inline data): And, when using an XPath formula, you can now select the fields to retrieve: 25

26 Filtering based on a result set Enesys IS Data Extension now supports filtering list statements based on the results of another statement. This filter is converted into a CAML query, which is applied before the data is returned from SharePoint. This results in great performance improvements when filtering large lists. For more information about the statement filters, please read the Filtering items based on another statement data chapter inside the Enesys SharePoint Query Designer manual. Retrieving list of choices Enesys IS Data Extension brings a brand new statement for retrieving a list of all the possible choices from any column of type Choice and derivated. This new columnchoices statement comes handy when you need to fill the possible values for a parameter. It may also help you retrieve a string representation for some integer columns (such as WorkflowStatus columns). 26

27 For additional information, please see the Retrieving column choices chapter inside the Enesys SharePoint Query Designer manual. Support for Integration Services 2012 Enesys IS Data Extension 1.4 is now fully compatible with Integration Services Query Designer improvements Performances This new version of the Query Designer brings significant performance improvements, especially in the initial loading, where you can expect up to 50% performance improvements. Renaming statements With previous versions of Enesys IS Data Extension, when renaming a statement, your query may become invalid because this statement name was used at other places (such as in a sqlquery statement), and its usages were not modified. Enesys SharePoint Query Designer now renames the statement usages all over the query, including inside the sqlquery statements. 27

28 For example, in the following query, the list statement WorkflowTests will be renamed to WorkflowList, its usages will be automatically updated inside the sqlquery statement: 28

29 Once renamed, its usages were updated: Advanced filter features The Query Designer now adds an improved version of its filter control. All filters now have a <Parameter > item for easily setting a parameter value: 29

30 By clicking on the Parameter item, it displays a form for either selecting an existing parameter or entering a new parameter name: Columns of type Choices or derivated now have a list of all their possible choices so that it s easier for you to filter on such columns. For example with the Status column from Issues lists: Or with WorkflowStatus columns: Note that such column type is Integer, the value that will be used for the comparison will be 0, 1, 2 Their string representation is only here for visual help. 30

31 Non-sortable & Non-filterable columns Some columns, such as Lookup Multi columns are not sortable, or others are non-filterable. There are no longer displayed, respectively, inside the Filter & Order By lists. Displaying fields inside the Query Statements panel Enesys SharePoint Query Designer 3.7 (EISDE 1.3) displayed the fields of selected fields for some statements, such as list and derivated. Inside this new version, we improved this, and the resulting schema of all statement types is now displayed inside the panel. Note that to avoid refreshing the fields all the time, they are refreshed when leaving the statement. New form for creating administrative statements When creating administrative statements, no data was asked during the creation, and you needed to go into the Properties to specify the source of the statements. This was not necessarily clear at first sight. 31

32 Enesys SharePoint Query Designer now displays a form when adding a new administrative statement to the query: It allows you to set the source as a previously created statement, if it has the necessary columns (RelativeSiteUrl in the case of an alistcollection statement), or to manually specify the values. 32

33 The Properties form of administrative statement was also updated to match those changes: Other improvements A new IncludesAll pseudo-caml element that allows filtering a multi-values column to ensure that returned items contain all the given values. A new distinct option was added to list statements to apply case sensitive or case insensitive distinct on the returned data. Improved <In> performances when using SharePoint 2010 by converting the pseudo- CAML In element into the new SharePoint 2010 <In> CAML element. Corrected a bug with the render of specific characters (such as < or >) when used in Note ( Several lines of text ) fields. Version 1.3 New InfoPath features Enesys IS Data Extension offers a brand new statement dedicated to InfoPath forms. This statement will let you retrieve data from both InfoPath promoted and non-promoted fields: 33

34 Enesys IS Data Extension returns the data using the data type specified in the InfoPath form. However, you can easily select a different data type: 34

35 You can find more information about the new InfoPath statement by reading the Retrieving data from InfoPath forms chapter located inside the Enesys SharePoint Query Designer manual. Item Versions Performance Boost If you have used Enesys IS Data Extension for building reports that deal with item version, you certainly know that this is a time consuming process. The main reason is the limitation of the SharePoint Web Services API in that specific area. By using the Object Model, Enesys IS Data Extension v1.3 now provides dramatically improved performances when retrieving item versions in SharePoint integrated mode. The following table provides performance comparisons between the version 1.2 and the version 1.3. Those results are based on a query that retrieves 500 items from a list, each item having 4 versions: VERSION 1.2 VERSION 1.3 ON A SHAREPOINT SERVER RATIO 500 items Retrieving version from 9 fields 500 items Retrieving version from 1 field 03:10 00:18 10x 00:38 00:07 5x Reference List for RollUpLists statements In previous versions of Enesys IS Data Extension, the SharePoint list used to build the target query schema was the first list available. In case all the lists you are aggregating are strictly identical, this is not an issue. But in case some of your lists have additional fields that you would like to be part of the target dataset, you will have to ensure that the first list (which is used to build the schema) has all the necessary fields. Depending on your scenario, this may not be practical to achieve. Enesys IS Data Extension 1.3 now allows you to specify a referencesite attribute that will contain the URL to the site containing the list used to build the schema. Similarly to the relativesiteurl, the referencesite is a URL relative to the data source URL. For more information on the specification of the referencesite attribute within the Query Designer, please read the Rolling up list items in an entire site collection chapter inside the Enesys SharePoint Query Designer manual. Query Designer Improvements Enesys IS Data Extension 1.3 brings various user interface improvements. 35

36 Intellisense Parameters for functions that have the same name as keywords (for example the REPLACE function is also a keyword) are now properly displayed. In order to simplify field selection, Intellisense for SQLite function is now disabled by default. You can re-enable it by clicking on the appropriate button: Corrected a bug which prevented Intellisense for some statements when joining 3 or more statements with JOIN tablename ON condition. New Query Statements toolbar A new toolbar was added into the Query Statements panel to allow you to quickly manipulate the statements. This toolbar allows you to: Add new statements Go to the Properties form of the selected statement Delete the selected statement Move up/down the selected statement inside the statement list 36

37 New Form for adding statements By clicking on the New menu item, a form is displayed that lists all the available statements, grouped by category: Removed the Columns/Filtering/Sorting tab For simplification, the Columns/Filtering/Sorting tab was removed and all its functionalities where moved in order to be available in the main design area. The Query columns panel was removed. 37

38 You can now reorder the fields inside the Query Statements panel (along with the possibility to delete them): The Columns sorting order panel was moved into a new Order By tab of the Columns tab: All the previous functionalities of the control are kept, and this new control offers an enhanced field selector: The Query conditions tab was moved into the new Filters tab of the Columns tab: 38

39 All the previous functionalities of the Query conditions panel were preserved, and some new features were added: You can read more about the new filtering control inside the following chapter. New advanced filtering control With previous versions of Enesys RS Data Extension, the Query Designer was using a third party filtering control. Even if this control was satisfying, it misses some important functions when writing a query. Functions that we tried to have in mind when developing our own filtering control: Support for moving conditions/groups (up/down and left/right): Support for adding parent groups to a specific group: 39

40 Improved field selector by using an autocomplete drop down list: Changing the operator no longer changes the value: 40

41 Better support for LookupId and IncludeTimeValue options: For more information on the new filtering control, please read the Filtering list items chapter inside the Enesys SharePoint Query Designer manual. 41

42 Showing/Hiding panels By default, the Results panel is now hidden. But you can quickly show it by clicking on the appropriate button: 42

43 Similarly, you can quickly show/hide the statement panel by clicking on the appropriate button: Other improvements Added two new SQLite collating sequences: UTF8 and UTF8CI. They bring support for UTF-8 comparisons when sorting/grouping data. The UTF8 collating function is case sensitive while the UTF8CI one is case insensitive. Usage: SELECT foo FROM bar ORDER BY foo COLLATE UTF8 or SELECT foo FROM bar GROUP BY foo COLLATE UTF8CI This is useful when sorting/grouping data that has diacritics or non-latin characters. Corrected a bug which prevented to roll up data starting from a site having spaces in its URL. Version 1.2 Retrieving parsed Survey data Enesys IS Data Extension offers a new statement that lets you retrieve transformed Survey lists data. 43

44 Though this transformed data is mostly intended to creating reports on Survey data using Enesys RS Data Extension, you can still use the new survey statement as part of your Integration Services packages. Querying Oracle databases Enesys IS Data Extension, using the sqlserverquery statement allowed you to retrieve data from any SQL Server database. In this new version, the sqlserverquery statement is renamed to sqldata and is extended by adding support for Oracle databases. The behavior is similar, you just have to select between SQL Server and Oracle when you create the statement: 44

45 Then, enter the connection string properties: You can now query the database: For more information, see the Joining SharePoint lists and external SQL data chapter inside the Enesys SharePoint Query Designer manual, and the sqldata statement chapter inside the Enesys SharePoint Query manual. 45

46 Note: Please note that the existing sqlserverquery statements are still valid and are converted under the hood into sqldata statements, so that you don t have to change anything when upgrading to v1.2. Query Designer - Great Intellisense improvements SqlQuery statement is central to Enesys IS Data Extension innovative approach to build sophisticated queries. In this version, we have made several improvements regarding the Intellisense features. Extended Statement Support With the previous versions, only the statement name was provided for sqldata, xmlquery and sqlquery statements, you still had to write the field names without any support from intellisense. In this version, Intellisense is now provided for fields resulting from any statement: For example, a database queried using the sqldata statement: Or a RSS feed queried using the xmlquery statement: 46

47 After having joined SQL Server and SharePoint data using a sqlquery statement, you can use Intellisense inside a second sqlquery statement: SQL Functions support SQLite functions are listed along with their associated parameters and a brief description: Parameter description is available when editing a function: 47

48 Better Contextual Support Real time query parsing has been much improved in order to provide contextual intellisense support: No statement is selected, the fields are not available: Only the Products statement is selected, only its fields are available: Both statements are joined, all the fields are available: Support for table aliases: The p name is detected as an alias of the Products table; therefore, fields from the Products table are available. 48

49 pc is recognized as an alias of the Product categories table, its fields are available. SELECT and JOIN Snippets Two code snippets were added for quickly adding SQL clauses: A SELECT code snippet for inserting a SELECT clause with a FROM clause that selects a single statement: By validating (with the Tab key), the snippet is inserted: Available statements are proposed for selection: Enter/select the desired statement and click on Enter to validate the snippet modification. The caret is then moved just after the SELECT so that you can select your fields: 49

50 A JOIN snippet for inserting a JOIN clause: By clicking on the TAB key, you insert the snippet: Three placeholders are available for specifying the joined table name and the two columns used as part of the join: For more information on Intellisense usage, please see the Using Intellisense inside sqlquery statements chapter inside the Enesys SharePoint Query Designer manual. 50

51 Using parameters and variables for specifying SharePoint List URL and title The Properties form of list, survey, mergelists and rolluplists statements now has a Url and Title tab that allows you to easily change the way you specify the list title and its relative site URL: 51

52 When specifying a parameter, the value is specified into the DefaultParameterValues query option: The query now has an options statement with the value specified: When Integration Services will retrieve the list schema, it will use this default value, so that you don t get prompted for specifying the parameter values each time you edit the query. 52

53 For more information, see the Using data source variables chapter inside the Enesys SharePoint Query Designer manual. And for more information about the options node, see the options statement chapter inside the Enesys SharePoint Query manual. Query Designer User Interface Improvements The version 3.6 includes some improvements regarding the User Interface that will enhance your query design experience. Adding a list statement The dialog box for selecting a list has been completely revamped as shown in the following screenshot: Key improvements: Using a data source variable for specifying the relative site URL Filtering lists by name, type and visibility 53

54 Adding a rolluplists statement Similarly to the Add list statement dialog box, the Add rolluplists statement dialog box was revamped: Key improvements: Improved performances by limiting the data retrieved from SharePoint On-demand retrieval of the sub-sites lists Using a data source variable for specifying the relative site URL Filtering lists by name, type and visibility 54

55 Selecting columns SharePoint lists may have numerous columns and it may not be easy to go through every column to look for a specific column. We have added some options to the Columns tab, as you can see on the following screenshot: Key improvements: Dynamic filter of the columns based on their internal and display name. Show/hide the hidden columns with a single click Select/unselect all displayed columns with a single click Other improvements The alistpermissions and asitepermissions statements now use the correct enumeration for the permissions. Improved Null parameter value handling within CAML queries: o Eq operator is transformed to IsNull if the parameter value is null. o Neq operator is transformed to IsNotNull if the parameter value is null. Null parameter values are properly handled inside sqldata (previously named sqlserverquery) statements. Parameters may now be used inside the title attribute of the list, mergelists and rolluplists statements. This allows you to dynamically change the selected list at execution-time, granted that lists have the same selected fields. Corrected a bug with SharePoint 2010 that caused an error to be thrown when retrieving Item Versions for the Editor column. Version 1.1 Using Object Model for rolling up data In this version, Enesys IS Data Extension can take advantage of the SharePoint Object Model when your packaged is launched on a SharePoint server. This can lead to tremendous performance improvement in some scenarios. 55

56 For more information, see the Using the Object Model for rolling up data chapter, page 95. Support for Forms Based Authentication Enesys IS Data Extension is now able to query a SharePoint server using Forms Based Authentication by simply setting the authenticationmode Connection String option to Forms. For more information, see the Forms Based Authentication chapter, page 100. Specifying fields for which to retrieve item versions In previous versions, retrieving item versions was an all or nothing option. Enesys IS Data Extension would retrieve version information for all the selected fields. Retrieving item versions is a time consuming process. In this version, we added the possibility to specify the fields for which you need version information. For more information, please see the itemversionfields element chapter inside the Enesys SharePoint Query manual, and Specifying the item version fields chapter inside the Enesys SharePoint Query Designer manual. Using Data Source variables inside the SqlServerQuery s connection string In this version, you can now specify data source variables inside the sqlserverquery statement s Connection String. This additional feature makes it easier for moving packages from a development environment to a production environment by simply modifying the data source. 56

57 Example: Using this data source connection string: server= You can use a sqlserverquery statement like this: <sqlserverquery tablename="sqlserverquery1"> <connectionstring> Data Source=$source!;User ID=$user!;Password=$pass!;... </connectionstring> <commandtext>select...</commandtext> </sqlserverquery> For more information, see the Using variables inside the sqlserverquery s Connection String chapter inside the Enesys SharePoint Query Designer manual, and the SqlServerQuery statement chapter inside the Enesys SharePoint Query manual. Advanced Properties Form We added a new Properties form that you can access by right clicking on a statement or by clicking on the appropriate button inside the Properties tab: 57

58 This new Properties form is available for all the statements and offers a better user experience for editing the most common statement properties. 58

59 It automatically validates the data, and displays a small icon with information about the error if the data is invalid: Bug corrections Corrected a bug with itemversions and Document Library. Corrected a bug that prevented the use of a quote character inside the tablename attribute when using the rolluplists statement. Corrected a bug inside the Query Designer that displayed Intellisense for invalid lookup ID column. Corrected a bug inside the Add list statement form that could lead to an error when trying to add a list. Version 1.0 New Connection String options Three connection string options are available to help you configure the Web Services calls: timeout: The time (in milliseconds) to wait for the Web Services queries to execute. Default value is (5 minutes). retrytime: The time (in milliseconds) to wait before retrying to call the Web Services when a SharePoint timeout occurred. 59

60 sleeptime: The time (in milliseconds) to sleep between two calls to the Web Services. When full synchronizing a very large list, it may be interesting to play with those options to put less pressure on your SharePoint server. Those options can directly be edited using the Connection Manager Editor: New form for managing Connection Managers With Integration Services, it is not possible to share connection managers with two different packages. So you end up creating a lot of similar connection managers. 60

61 To help you create similar Connection Managers, a new form lets you create Connection Managers based on existing ones: Its concept is similar to the one available for ADO.Net and OleDB Connection Managers: Click on New to create a new Connection Manager. Click on a Connection Manager to display its properties. Double click on a previously created Connection Manager to recreate it. The Connection Manager names are generated based on the data source property and the credentials. Other improvements Fixed a versioning problem with the Merge Component. Set the default value of the CommandTimeout property to 300 seconds instead of 30. Version 0.9 (RC2) New Synchronization Source Component The second Release Candidate version of Enesys IS Data Extension contains a new Synchronization source component that allows synchronizing any SharePoint list with an SQL Server table (one-way synchronization). 61

62 With the Enesys SharePoint Source component, you will retrieve all the items returned by the query each time you execute your package. Using Enesys Synchronization SharePoint Source, only items that have changed since the last synchronization are retrieved from SharePoint. It allows a very fast update (even with lists that contain more than items), without the need to clean the SQL table each time you execute the package. The only statement available inside the Enesys Synchronization SharePoint Source is the synclist statement that allows you to only retrieve the last changes from any SharePoint list. For more information on the synclist statement, please read the Enesys SharePoint Query manual. For more information on how to synchronize a SharePoint list with an SQL Server table, see the Synchronizing SharePoint list with SQL Server chapter, page 80. New Merge Destination Component The data retrieved from the Synchronization component may be tricky to use, you would need to insert the data into a temporary SQL table using an ADO.Net Destination component, then execute some T-SQL statements (including a MERGE statement between the temporary table and the final table) using an Execute SQL Task. To simplify this process, we have added a custom Merge Destination component that you can connect to the Synchronization component, and most of the configuration will be made automatically by the component so you don t have to spend time configuring the synchronization. Support for Integrated Security The previous version only allowed specifying credentials to connect to the SharePoint server. This new version fully supports the Integrated Security credentials. 62

63 Intellisense for SqlQuery statements Basic intellisense features are now provided when editing SqlQuery statements. Other changes Corrected a bug that prevented the use of SQLite functions within the sqlquery statement and resulted in an Unexpected DataReader column type error. Renamed the Enesys IS Data Extension component to Enesys SharePoint Source. Version 0.9 (RC1) This version contains a source component for retrieving data from SharePoint, it includes the same query engine as Enesys RS Data Extension allowing you to use any of the statements (list, mergelists, rolluplists, sqlquery, xmlquery and administrative statements), for integration into your data flow. A visual Query Designer is available within the product, allowing you to easily design queries. Parameters are supported inside the components, by mapping them to SSIS Variables. 63

64 Quick start This chapter shows how to quickly create a package using data from a SharePoint list. If you have already built packages with Business Intelligence Development Studio, you may want to skip this section. Create a new Integration Services project Open Business Intelligence Development Studio or Visual Studio 2005/2008/2010 and create a new Integration Services Project. A package named Package.dtsx is already added to the project and opened in the visual editor. 64

65 Add a Data Flow Task Drag/drop the Data Flow Task item from the Control Flow Items toolbox to the Control Flow tab of the Visual Editor. Add an Enesys IS Connection Manager Right click inside the Connection Managers tab and select New Connection. 65

66 Select the EISCM Connection Manager type and click on Add. 66

67 A form that allows you to recreate previously created connection managers is available. If this is not the first time you created a Connection Manager, you can choose to create one from any previously created one. If this is the first time, click on New to create a new one. 67

68 The Enesys IS Connection Manager Editor allows you to specify the properties of the connection (Connection String and Credentials). Once you have filled in the various options, click OK to validate the creation of the new Connection Manager. Note: With Integration Services 2012, you can now create Connection Managers that are linked to a Project instead of being linked to a single package, similarly to the principle of Shared Data Source with Reporting Services. We recommend using this approach as this allows you to easily use the same connection managers for all your packages. Add an Enesys SharePoint Source component Double click on the Data Flow Task to go to the Data Flow tab. Drag/drop an Enesys SharePoint Source source component from the Data Flow Sources toolbox to the Visual Editor. 68

69 Note: If the Enesys SharePoint Source component is not available in the Toolbox, please review the installation steps. Note: With Integration Services 2012, the components are no longer inside the Toolbox, but inside the SSIS Toolbox, you can show it by clicking on the appropriate button: Associate a Connection Manager to the component Double click on the Enesys SharePoint Source component that was just created. In the Connection Manager tab of the editor, choose the Connection Manager you created in the previous step. 69

70 If you did not create a Connection Manager, you can still create a new one by clicking on the New button. Design the query Enesys SharePoint Source component provides a Visual Query Designer that allows you to build queries without the need to know the details of the query language. Switch to the Query tab in order to design your query. Add a new list statement Right click on the Query Statements panel Click on Add list statement to add a new list statement 70

71 Enter the relative site URL and click on Check Available Lists to get all the lists in this site. Choose the desired lists and click on the Add button. 71

72 Previewing/Executing the query When executing a query within the Query Designer, it will return 200 items by default. Though you may certainly not want to retrieve all items when dealing with large lists, you may change the Preview Limit value. When set to 0, all items will be retrieved. Click on the Preview/Execute button to retrieve the items. You should get the 200 first items from your SharePoint list in the Results panel, as shown in the following screenshot: For more information on how to design a query using the visual Query Designer, please see the Enesys SharePoint Query Designer manual. Map the parameters If your query uses parameters, you have to map those parameters to SSIS Variables. 72

73 Click on the Parameters tab and for each parameter, select an existing SSIS variable, or create a new one by choosing <New Variable > : In case you selected the New Variable option, you have to fill in the Add Variable form: Once each parameter is mapped to an SSIS variable, you can click OK to validate the component. 73

74 Add a Destination component You may at least use a Data Flow Destination component to store the data retrieved from SharePoint. Drag the Data Flow Destination component you want to use into the Data Flow tab, for example an ADO NET Destination component. Link the Enesys SharePoint Source component to the ADO NET Destination component. 74

75 Double click on the ADO NET Destination component to edit its properties. Create a new ADO NET Connection Manager by clicking on the New button. In the Configure ADO.NET Connection Manager form, click on the New button. 75

76 A Connection Manager form will be displayed: Click on the Test Connection button to ensure that the connection is possible and validate with OK. 76

77 In the ADO.NET Destination Editor, click on the New button to create a new table: 77

78 A table script generation will be automatically generated based on the columns provided by the Enesys SharePoint Source component: 78

79 Click on the Mapping tab to generate the mapping between the Input and the Destination columns: And click OK to validate the modification of the Destination component. Execute the Data Flow Click the Debug button inside BIDS to execute the data flow. 79

80 Synchronizing SharePoint list with SQL Server This chapter describes how to use the Enesys Synchronization SharePoint Source component to synchronize a SharePoint list with an SQL Server table (one-way synchronization). Enesys Synchronization SharePoint Source component is a specialized component that will let you synchronize a SharePoint list with a SQL table. After an initial full synchronization, the component will update the table with the changes that occurred in the SharePoint list since the last synchronization. Therefore, a list with items can be updated in a matter of seconds or minutes. This section requires that you have already created a package. Refer to the Quick start chapter, page 20 for more information about the package creation. For synchronizing a SharePoint list with a SQL Server table we will need a Data Flow Task and two Data Flow Components: An Enesys Synchronization SharePoint Source component that retrieves synchronized data from SharePoint An Enesys Merge Destination component that merges the synchronized data with a SQL table. Important notice Some of the T-SQL statements used in the destination component (such as MERGE) need at least SQL Server Therefore, the destination component is not available on SQL Server You can still use the Enesys Synchronization SharePoint Source component but you would have to write your own logic to merge the data returned by the component (items updated, added or deleted since the last synchronization) into a SQL data table. More information can be found inside the Parsing Synchronized items chapter. 80

81 Add a Data Flow Task Drag/drop the Data Flow Task item from the Control Flow Items toolbox to the Control Flow tab of the Visual Editor. Add an Enesys Synchronization SharePoint Source component Double click on the Data Flow Task you just created, to switch to the Data Flow tab. Drag/drop an Enesys Synchronization SharePoint Source component from the Data Flow Sources toolbox to the Visual Editor. Note: If the Enesys Synchronization SharePoint Source component is not available in the Toolbox, please review the installation steps. Select the Connection Managers Double click on the Enesys Synchronization SharePoint Source component that was just created. In the Connection Managers tab of the editor, you have to set up two Connection Managers that are used for Synchronization. 81

82 The first Connection Manager is the connection to the SharePoint server using an Enesys IS Connection Manager. You can choose the Connection Manager you created in the previous steps if you followed the Quick Start, or you can create a new Connection Manager by clicking on the New button. The second Connection Manager is the ADO.Net connection used to connect to a SQL Server table (named SharePointTokens ) that will be used to store the necessary tokens used for synchronizing the data. Note that it is necessary to select a.net SqlClient Data Provider (an OleDb provider will not work): You can create new Connection Managers by clicking on the New button for each Connection Manager. Important Notice: The list token is stored in the SQL Catalog specified in the Synchronization Connection Manager. If you would like to run two synchronization tasks on the same list (which should not be a very 82

83 common scenario anyway), you need to use two different catalogs for storing the tokens. A token is associated with the SharePoint list name in a table named SharePointTokens. Therefore, the respective tasks will overwrite the same token when using the same catalog. This is a known limitation of the current version. Design the query Enesys Synchronization SharePoint Source component provides a Visual Query Designer that allows you to build queries without the need to know the details of the query language. Switch to the Query tab in order to design your query. Add a new synclist statement Right click on the Query Statements panel Click on Add synclist statement to add a new synclist statement 83

84 Enter the relative site URL and click on Check Available Lists to get all the lists in this site. Choose the desired list and click on the Add button. Select the necessary fields: Execute the query Click on the Preview/Execute button to retrieve the items that have changed since the last synchronization (or all the items if this is the first time you synchronize this list). 84

85 You should get the items from your SharePoint list in the Results panel, as shown in the following screenshot: As you can see, some specific columns used for synchronization are added to the result set. You may find additional information regarding those columns in the Parsing Synchronized items chapter, page 91. Map the parameters If your query uses parameters, you have to map those parameters to SSIS Variables. Click on the Parameters tab and for each parameter, select an existing SSIS variable, or create a new one by choosing the <New Variable > option: 85

86 Once each parameter is mapped to an SSIS variable, you can click OK to validate the component modification. Add an Enesys Merge Destination component Once the Enesys Synchronization SharePoint Source component has been configured, we will add an Enesys Merge Destination component that merges the data with some SQL table. Drag/drop an Enesys Merge Destination component from the Data Flow Destinations toolbox to the Visual Editor. Then, link the Enesys Synchronization SharePoint Source component to the Enesys Merge Destination component: Select the Connection Manager Double click on the Enesys Merge Destination component to edit its properties. 86

87 Create a new ADO NET Connection Manager by clicking the New button or re use an existing ADO Net Connection Manager if it makes sense: Create the destination SQL table Click on the New button located next to the Use a table or view textbox. 87

88 The Create Table dialog box will display the necessary query to create a table with all the columns retrieved from the Enesys Synchronization SharePoint Source component. Replace the table name by the destination table s name, then click on OK to create the SQL table. 88

89 Finalize the configuration Switch to the Mappings tab to automatically map the input and destination columns: The mapping is done automatically if the column names matches with the input column names, you may need to map the columns if this is not the case. Click OK to validate the configuration of the Enesys Merge Destination component. Execute the package Click on the Debug button inside BIDS to execute the package. It will synchronize your list with the specified SQL server table. 89

90 Note: Please note that, depending on your SharePoint list size, the first synchronization may take a long time to execute. 90