MeasureUp Notes. Contents

Transcription

1 MeasureUp Notes Contents Misc... 2 SSIS Catalog... 4 Options to run packages... 5 MDS... 6 CDC... 9 Compare Project Deployment and Package Deployment Features of Project Deployment Model SCHEMAS: Data Taps SQL Task Execute SQL tasks depend upon connection manager type Tasks SCD types Type I Type II Type III... 14

2 Misc Expression task creates and evaluates expressions that set variable values at runtime. Data steward is the person that is responsible for maintaining the dimension data. DTSWizard launches the SQL Import Export Wizard OLE DB Simple Provider is intended for simple OLE DB access of in memory arrays or XML documents Error output includes 2 additional columns ErrorCode (numeric code for the error) and ErrorColumn (contains the original number of the column that caused the error) Encryption is determined in the Sensitive property MaxConcurrentExecutables specifies the number of tasks that can run in parallel in a package. Default 1 Package including Execute Package tasks with needed variables Default ScriptMain class is the required location to specify your method SSIS calls for the entry point into the code of the Script Task. InitialCatalog property set to use the database parameter ServerName property to use the Instance parameter Gacutil.exe is used to deploy custom objects (dll s) for SSIS Global Assembly Cache You need to configure SSIS on Development1 server to pass the environment variable to the pakage. Question of Package A & B using a folder path:

3 Add environment to the project: : Mark a batch as successful: Edit the package to log specific errors: 12 tasks, no order send mail

4 SSIS Catalog must be created initially in SSMS enable CLR integration on the server right click Integration Services in SSMS and create the catalog define how logs are cleared, max time to allow project validation, etc. The SSIDB must be created before projects can be stored Packages can be still stored on the file system or in MSDB database IsDeploymentWizard is a command line utility and deploys SSIS 2012 packages to Integration Services Catalog on SQL 2012 instances Catalog Object Identifiers When you create a new object in the catalog, assign a name to the object. The object name is an identifier. SQL Server defines rules for which characters can be used in an identifier. Names for the following objects must follow identifier rules. Folder Project Environment Parameter Environment Variable Folders catalog.create_folder (SSISDB Database) catalog.delete_folder (SSISDB Database) catalog.rename_folder (SSISDB Database) catalog.set_folder_description (SSISDB Database) Projects and Packages You can carry out other project tasks by calling the following stored procedures. catalog.delete_project (SSISDB Database) catalog.deploy_project (SSISDB Database) catalog.get_project (SSISDB Database) catalog.move_project ((SSISDB Database) catalog.restore_project (SSISDB Database) These views provide details about packages, projects, and project versions. catalog.projects (SSISDB Database) catalog.packages (SSISDB Database) catalog.object_versions (SSISDB Database) These views show the parameters for all packages and projects, and parameter values that are used for an instance of execution. catalog.object_parameters (SSISDB Database) catalog.execution_parameter_values (SSISDB Database) The following stored procedures enable you to perform many other management tasks for environments and variables. catalog.create_environment (SSISDB Database)

5 catalog.delete_environment (SSISDB Database) catalog.move_environment (SSISDB Database) catalog.rename_environment (SSISDB Database) catalog.set_environment_property (SSISDB Database) catalog.create_environment_variable (SSISDB Database) catalog.delete_environment_variable (SSISDB Database) catalog.set_environment_variable_property (SSISDB Database) catalog.set_environment_variable_value (SSISDB Database) Server variables catalog.create_environment_reference (SSISDB Database) catalog.delete_environment_reference (SSISDB Database) catalog.set_environment_reference_type (SSISDB Database) For more details about environments and variables, query these views. catalog.environments (SSISDB Database) catalog.environment_variables (SSISDB Database) catalog.environment_references (SSISDB Database) Executions and Validations An execution is an instance of a package execution. Call catalog.create_execution (SSISDB Database) and catalog.start_execution (SSISDB Database) to create and start an execution. To stop an execution or a package/project validation, call catalog.stop_operation (SSISDB Database). To cause a running package to pause and create a dump file, call the catalog.create_execution_dump stored procedure For details about executions, validations, messages that are logged during operations, and contextual information related to errors, query these views. catalog.executions (SSISDB Database) catalog.operations (SSISDB Database) catalog.operation_messages (SSISDB Database) catalog.extended_operation_info (SSISDB Database) catalog.event_messages catalog.event_message_context Options to run packages SSMS DTExec or DTExecui Powershell SQL Server Agent Managed API can only execute packages stored on the SSIS server SQL Server Data Tools can only execute packages stored on the file system DTExec is used to configure and execute SSIS packages DTExecui, sames as DTExec but it opens a user interface, uses SQL log provider Dtexec (SSIS Tool): Exit codes returned, the exit code is used to populate the ERRORLEVEL variable Value Description 0 The package executed successfully. 1 The package failed. 3 The package was canceled by the user.

6 4 The utility was unable to locate the requested package. The package could not be found. 5 The utility was unable to load the requested package. The package could not be loaded. 6 The utility encountered an internal error of syntactic or semantic errors in the command line. DTUtil allows you to script the deployment of the legacy packages to the legacy instances of SQL 08 R2. i.e. DTUtil /FILE c:\sourcepkg.dtsx /DESTServer <server name> /COPY SQL;destpkgname can copy, move, delete and verify SSIS packages but cannot execute packages. MDS Data Quality Services and Master Data Services are used to manage data in a data warehouse Master Data Manager configures business rules Master Data Service Configuration Manager is used to configure MDS database and web apps not rules MDSModelDeploy.exe creates deployable packages Mdsmodeldeploy: file (product.pkg) stored in the configuration folder of the MDS installation Mdsmodeldeploy deploynew model Product service MDS4 package Product.pkg Master Data Server Web service is used by developers to customize MDS

7 Matching feature determines matching data and to perform de duplication DQS_Main contains stored procedures, DQS engine and published Knowledge Bases DQS Roles:

8 Staging Tables: The Master Data Services database includes three staging tables that you can populate with your organization s data. After you populate these tables, initiate the staging process to load the data from the staging tables into the appropriate MDS tables. The name of each table is specified when an entity is created and cannot be changed. If the staging table name contains a _1 or other number, another table of that name already existed when the entity was created. Table Name Description stg.name_leaf Create, update, and delete leaf members and their attributes. stg.name_consolidated Create, update, and delete consolidated members and their attributes. stg.name_relationship Move members in an explicit hierarchy. ImportType Determines what to do when staged data matches data that already exists in the Master Data Services database. Required 0: Create new members. Replace existing MDS data with staged data, but only if the staged data is not NULL. NULL values are ignored. To change a string attribute value to NULL, set it ~NULL~. To change a number attribute value to NULL, set it to To change a datetime attribute value to NULL, set it to T12:34:56. 1: Create new members only. Any updates to existing MDS data fail. 2: Create new members. Replace existing MDS data with staged data. If you import NULL values, they will overwrite existing MDS values. 3: Deactivate the member, based on the Code value. All attributes, hierarchy and collection memberships, and transactions are maintained but no longer available in the UI. If the member is used as a domain-based attribute value of another member, the deactivation will fail. See ImportType 5 for an alternative. 4: Permanently delete the member, based on the Code value. All attributes, hierarchy and collection memberships, and transactions are permanently deleted. If the member is used as a domain-based attribute value of another member, the deletion will fail. See ImportType 6 for an alternative. 5: Deactivate the member, based on the Code value. All attributes, hierarchy and collection memberships, and transactions are maintained but no longer available in the UI. If the member is used as a domain-based attribute value of other members, the related values will be set to NULL. ImportType 5 is for leaf members only. 6: Permanently delete the member, based on the Code value. All attributes, hierarchy and collection memberships, and transactions are permanently deleted. If the member is used as a domain-based attribute value of other members, the related values will be set to NULL. ImportType 6 is for leaf members only. DQS_Projects contains the data quality project information

9 DQS_Staging_Data used as a staging area where you can copy source data to perform DQS operations CDC CDC components in SSIS require an ADO.NET connection manager CDC Change Data Capture; uses an LSN, Log Sequence Number, to track all insert, update and deletes that occur within a specific time. CDC Splitter: Value of Operation Output Description 1 Delete Deleted Row 2 Insert Inserted row (not available when using Net with Merge CDC mode) 3 Update Before-update row (available only when using All with Old Values CDC mode) 4 Update After-update row (follows the Before-update) 5 Update Merge row (available only when using Net with Merge CDC mode) Other Error Change Data Capture Functions (Transact-SQL) cdc.fn_cdc_get_all_changes_<capture_instance> (Transact-SQL) cdc.fn_cdc_get_net_changes_<capture_instance> (Transact- SQL) sys.fn_cdc_decrement_lsn (Transact-SQL) sys.fn_cdc_get_column_ordinal (Transact-SQL) sys.fn_cdc_get_max_lsn (Transact-SQL) sys.fn_cdc_get_min_lsn (Transact-SQL) sys.fn_cdc_has_column_changed (Transact- SQL) sys.fn_cdc_increment_lsn (Transact-SQL) syss.fn_cdc_is_bit_set (Transact-SQL) sys.fn_cdc_map_lsn_to_time (Transact-SQL) sys.fn_cdc_map_time_to_lsn (Transact-SQL) Change Data Capture Stored Procedures (Transact-SQL) sys.sp_cdc_add_job (Transact-SQL) sys.sp_cdc_generate_wrapper_function (Transact-SQL) sys.sp_cdc_change_job (Transact-SQL) sys.sp_cdc_get_captured_columns (Transact-SQL) sys.sp_cdc_cleanup_change_table (Transact-SQL) sys.sp_cdc_get_ddl_history (Transact-SQL) sys.sp_cdc_disable_db (Transact-SQL) sys.sp_cdc_help_change_data_capture (Transact-SQL) sys.sp_cdc_disable_table (Transact-SQL) sys.sp_cdc_help_jobs (Transact-SQL) sys.sp_cdc_drop_job (Transact-SQL) sys.sp_cdc_scan (Transact-SQL) sys.sp_cdc_enable_db (Transact-SQL) sys.sp_cdc_start_job (Transact-SQL) sys.sp_cdc_enable_table (Transact-SQL) sys.sp_cdc_stop_job (Transact-SQL)

10 Change Data Capture Tables (Transact-SQL) cdc.<capture_instance>_ct Returns one row for each change made to a captured column in the associated source table. cdc.captured_columns Returns one row for each column tracked in a capture instance. cdc.change_tables Returns one row for each change table in the database. cdc.ddl_history Returns one row for each data definition language (DDL) change made to tables that are enabled for change data capture. cdc.lsn_time_mapping Returns one row for each transaction having rows in a change table. This table is used to map between log sequence number (LSN) commit values and the time the transaction committed. cdc.index_columns Returns one row for each index column associated with a change table. dbo.cdc_jobs (Transact-SQL) Returns the configuration parameters for change data capture agent jobs. Compare Project Deployment and Package Deployment The type of deployment model that you choose for a project determines which development and administrative options are available for that project. The following table shows the differences and similarities between using the project deployment model and using the package deployment model. When Using the Project Deployment Model A project is the unit of deployment. Parameters are used to assign values to package properties. A project, containing packages and parameters, is built to a project deployment file (.ispac extension). A project, containing packages and parameters, is deployed to the SSISDB catalog on an instance of SQL Server. CLR integration is required on the database engine. Environment-specific parameter values are stored in environment variables. Projects and packages in the catalog can be validated on the server before execution. You can use SQL Server Management Studio, stored procedures, or managed code to perform the validation. Packages are executed by starting an execution on the database engine. A project identifier, explicit parameter values (optional), and environment references (optional) are assigned When Using the Package Deployment Model A package is the unit of deployment. Configurations are used to assign values to package properties. Packages (.dtsx extension) and configurations (.dtsconfig extension) are saved individually to the file system. Packages and configurations are copied to the file system on another computer. Packages can also be saved to the MSDB database on an instance of SQL Server. CLR integration is not required on the database engine. Environment-specific configuration values are stored in configuration files. Packages are validated just before execution. You can also validate a package with dtexec or managed code. Packages are executed using the dtexec and DTExecUI execution utilities. Applicable configurations are identified by command-prompt arguments (optional).

11 to an execution before it is started. You can also execute packages using dtexec. During execution, events that are produced by the package are captured automatically and saved to the catalog. You can query these events with Transact-SQL views. Packages are run in a separate Windows process. SQL Server Agent is used to schedule package execution. During execution, events that are produced by a package are not captured automatically. A log provider must be added to the package to capture events. Packages are run in a separate Windows process. SQL Server Agent is used to schedule package execution. Features of Project Deployment Model The following table lists the features that are available to projects developed only for the project deployment model. Feature Parameters Environments Environment variables SSISDB catalog Catalog stored procedures and views Description A parameter specifies the data that will be used by a package. You can scope parameters to the package level or project level with package parameters and project parameters, respectively. Parameters can be used in expressions or tasks. When the project is deployed to the catalog, you can assign a literal value for each parameter or use the default value that was assigned at design time. In place of a literal value, you can also reference an environment variable. Environment variable values are resolved at the time of package execution. An environment is a container of variables that can be referenced by Integration Services projects. Each project can have multiple environment references, but a single instance of package execution can only reference variables from a single environment. Environments allow you to organize the values that you assign to a package. For example, you might have environments named "Dev", "test", and "Production". An environment variable defines a literal value that can be assigned to a parameter during package execution. To use an environment variable, create an environment reference (in the project that corresponds to the environment having the parameter), assign a parameter value to the name of the environment variable, and specify the corresponding environment reference when you configure an instance of execution. All Integration Services objects are stored and managed on an instance of SQL Server in a database referred to as the SSISDB catalog. The catalog allows you to use folders to organize your projects and environments. Each instance of SQL Server can have one catalog. Each catalog can have zero or more folders. Each folder can have zero or more projects and zero or more environments. A folder in the catalog can also be used as a boundary for permissions to Integration Services objects. A large number of stored procedures and views can be used to manage Integration Services objects in the catalog. For example, you can specify values to parameters and environment variables, create and start executions, and monitor catalog operations. You can even see exactly which values will be used by a package before execution starts.

12 SCHEMAS: Star schema is formed using de normalized dimensions as the main dimensional structure Snowflake schema is formed using normalized dimensions as the main dimensional structure Dimensional Structures: Granular, Hierarchal and Conformed Fact tables contain the data to be aggregated and do not normally contain dimension data, it can but not as a main purpose. A Fact table that relates to more than 1 Dimension table creates a Many to Many relationship DD (degenerate dimension) serves as a dimension key in a fact table Non aggregatable measures support values that cannot be summed across a dimension. Semi additive measures apply to measures can be summed across some dimensions but not others. (bank balance of account types but not over multiple years) There are three kinds of Attributes: Fixed Attribute - For which value is not changing Changing Attribute - For which value will be over-written (Type 1) Historical Attribute- For which value is saved as a new record. Previous values are saved in the records marked as outdated (Type 2)

13 Data Taps server/2904/ssis 2012 data tap A Data Tap is an output of the data from one of the data flow paths in your package. Data Taps are a lot like Data Viewers. You can add Data Viewers in design mode while working with the package inside Visual Studio (VS*) as a debugging tool. The differences are that a Data Tap exports the data to a file from an already deployed package. Implement data taps in the SSISDB catalog SQL Task Execute SQL tasks depend upon connection manager type Connection type SELECT syntax EXCEL, ODBC, and SELECT* FROM Production.Product WHERE ProductId >? AND ProductID <? OLEDB ADO SELECT* FROM Production.Product WHERE ProductId >? AND ProductID <? ADO.NET SELECT* FROM Production.Product WHERE ProductId AND ProductID The examples would require parameters that have the following names: The EXCEL and OLED DB connection managers use the parameter names 0 and 1. The ODBC connection type uses 1 and 2. The ADO connection type could use any two parameter names, such as Param1 and Param2, but the parameters must be mapped by their ordinal position in the parameter list. The ADO.NET connection type uses the parameter What is the? used for in WHERE LastModified >?

14 Tasks SCD types Slowly Changing Dimensions Type I This methodology overwrites old with new data, and therefore does not track historical data. Example of a supplier table: Supplier_Key Supplier_Code Supplier_Name Supplier_State 123 ABC Acme Supply Co CA Type II This method tracks historical data by creating multiple records for a given natural key in the dimensional tables with separate surrogate keys and/or different version numbers. Unlimited history is preserved for each insert. For example, if the supplier relocates to Illinois the version numbers will be incremented sequentially: Supplier_Key Supplier_Code Supplier_Name Supplier_State Version. 123 ABC Acme Supply Co CA ABC Acme Supply Co IL 1 Type III This method tracks changes using separate columns and preserves limited history. The Type III preserves limited history as it's limited to the number of columns designated for storing historical data. The original table structure in Type I and Type II is the same but Type III adds additional columns. In the following example, an additional column has been added to the table to record the supplier's original state only the previous history is stored. Supplier_Key Supplier_Code Supplier_Name Original_Supplier_State Effective_Date Current_Supplier_State 123 ABC Acme Supply Co CA 22 Dec 2004 IL

15

16 Acronyms SSDT SQL Server Data Tools used to be BIDS Cells (columns) MDX Multidimension Data Expression DAX Direct Analysis Expression GAC Global Assembly Cache MDS Master Data Services SSDT SQL Server Data Tools CDC Change Data Capture DQS Data Quality Services SCD Slowly Changing Dimensions