Informatica Data Quality Upgrade. Marlene Simon, Practice Manager IPS Data Quality Vertical Informatica

Transcription

1

2 Informatica Data Quality Upgrade Marlene Simon, Practice Manager IPS Data Quality Vertical Informatica 2

3 Biography Marlene Simon Practice Manager IPS Data Quality Vertical Based in Colorado 5+ years with Informatica Professional Services 20+ years experience in delivering and managing data quality projects 3

4 Agenda Informatica Data Explorer (IDE) Data Quality v9.1 Functionality IDE Migration/Upgrade Informatica Data Quality (IDQ) Overview of version differences IDQ Migration/Upgrade Questions 4

5 Informatica Data Explorer 5

6 Data Quality v9.1 Functionality Feature Supported By: Both Analyst and Developer Analyst Only Developer Only Common to BOTH IDQ & IDE IDQ 9.1 IDE DQ Transformations NOT entitled to Structural Profiling features Informatica DQ Analyst - 5 named users (additional users available on tiered pricing model by seat) Informatica Analyst Viewer - Unlimited ( read only Analyst user) Informatica Developer - Unlimited - Entitled to core transforms Server -CPU Cores Base Profiling Features - Column Profiling - Rule Profiling - Scorecarding - RTM - Midstream - Comparative Profiling - Mapping generation from Profile - Join Validation Structural Profiling - Table Profiling - Primary Key Inference - Functional Dependency - Cross Table Profiling - Foreign Key Inference - Overlap Discovery - Profile Model NOT entitled to DQ Transformations Note: Effective in 9.1.0, Informatica Data Explorer Standard Edition is referred to as Informatica Data Explorer Legacy and Informatica Data Explorer Advanced Edition is renamed to Informatica Data Explorer. 6

7 IDE Migration/Upgrade Overview Supported versions Informatica Data Explorer Legacy 8.6.x 9.1 Informatica Data Explorer 9.1 You can transfer the following objects to the Informatica Data Explorer platform as part of the migration: Database connections Data sources Column profiling Schemas Projects 7

8 IDE Migration/Upgrade Overview IDE Migration Object Conversion Informatica Data Explorer Legacy Data Source Connections Tables Rules Action Items Notes Schemas Projects Informatica Data Explorer Data Source Connections Data Objects Expression Rules Comments Comments Folders Projects 8

9 IDE Migration path to current release Informatica V9 Mercury Platform IDE 9.1 IDE 9.5 IDE3.0 Legacy IDE Migration Utility Install IDE Legacy Migration Server Legacy IDE 9.0 Run migration utility to upgrade to IDE 9.1 Upgrade to 9.5 Legacy IDE 9.1 Migration Server is part of IDE Legacy 9.1 Installation run utility to upgrade 9

10 Migration Reference Documents Documentation is available on the Informatica website 0281_MigratingObjectsFromIDELegacyToIDE.pdf Outlines the migration process from Informatica Data Explorer Legacy (V8.6.2) to Informatica Data Explorer (V9.1) 10

11 Informatica Data Quality 11

12 IDQ Overview Version Differences Different Architecture Differences between Components and V9 transformations Change in terminology between Versions 12

13 IDQ Architecture Common Configuration One repository per user Reference data on local file system Data quality metadata contained in IDQ plan Connection details embedded within IDQ plan IDQ Client with local repository IDQ Server and shared repository 13

14 Informatica 9 Architecture 9.1 Common Configuration Central repository shared by all users Analyst and Developer Tool Reference data in Reference Table Manager Data Quality metadata in models Connection details stored and managed centrally Informatica Analyst Informatica Server Environment Data Quality Server File System Address Doctor 5 Reference Data Informatica Developer Informatica Repository MRS Address Doctor 5 14

15 8.6.2 Component vs. V9 Transformation Direct Object Conversion Multiple Objects in Supported Enhanced Functionality Not Supported Component converts to V9 Transformation Source, Target or component convert directly to transformation Merge Merge :: To Upper Case Component converts to multiple V9 Transformations Source, Target or component convert to multiple transform or mapplet Context Parser = Labeller + Parser (set to pattern base) V9 Transformation provides Enhanced functionality Provide equivalent functionality however have evolved from Jaro, Bigram, Edit and Hamming distance convert to Comparison Component does not Migrate to V9 Component not supported in repository Scripting component :: will convert to expression 15

16 Example Migration Exclusions Report Targets (flat file and relational) Scorecards now part of Analyst tool and based on profiling results (of columns or rules) Scripting (TCL) Version uses Java for scripting requirements TCL script can also be in the RBA will be converted to a decision transformation Mixed Field Matcher Component no longer supported NOTE: Full set of component to transformation conversions provided in Appendix A of this presentation 16

17 Changes in terminology IDQ IDQ 9 Plan Component Dictionary Mapping Transform / Transformation Reference Table 17

18 IDQ Migration/Upgrade Migration/Upgrade Path Migration/Upgrade Considerations Migration Pre-requisites Migration Process Post Migration Tasks Upgrade Process Post Upgrade Tasks 18

19 Migration/Upgrade Path IDQ 3.0 IDQ 3.1 IDQ 8.5 IDQ 8.6.x prior to Informatica V9 Mercury Platform IDQ IDQ IDQ 9.1 IDQ 9.5 IDQ 3.0 IDQ 3.1 IDQ 8.5 Upgrade 3.0 to 3.1 Upgrade 3.1 to Upgrade 3.1 to Upgrade 8.5 to Migration Utility -- Only works between and versions Export plans and dictionaries from Migrate to Upgrade to 9.1 or 9.5 IDQ 8.6.x prior to Upgrade to Validate mappings and mapplets IDQ

20 Migration/Upgrade Considerations Identify Level of effort for Migration/Upgrade Stand Alone Data Quality Migration IDQ Server Only (shared repository) IDQ Clients (local repositories) Data Quality Batch scripts PowerCenter Integration Verify all integrated DQ objects exist in the IDQ repository for migration/upgrade How many PC mappings require upgrade/modification Contain GAV component Contain deprecated components 20

21 Migration/Upgrade Considerations Review plans and dictionaries slated for migration. Remove or delete those that do not need to be migrated Identify any duplicate plans and determine which need be migrated Review batch scripts and remove or delete any which are obsolete 21

22 Migration/Upgrade Considerations Migrate plans or re-develop mappings Less than 4 plans and low to medium complexity Recommended to redevelop in IDQ Developer tool than perform migration and upgrade Ensure you plan adequate time for testing and QA More than 4 plans or extremely complex plans Typically more efficient to perform migration/upgrade Ensure you plan adequate time for testing and QA 22

23 Migration/Upgrade Considerations IDQ/PC Integration and IDQ repository no longer exists If the integrated DQ plans do not contain any GAV or deprecated components run the CRS utility in the PowerCenter environment, the plans continue running in the 9.x PowerCenter mappings If the integrated plans contain GAV or deprecated components You CAN NOT export IDQ plans from PowerCenter to IDQ Workbench or Developer tools you will have recreate the DQ mappings in the Developer tool and export to PowerCenter 23

24 Migration/Upgrade Considerations Stand Alone Data Quality Migration/Upgrade IDQ Local repositories IDQ Server repository Migration ClientPackage ServerPackage XML Import Upgrade Informatica 9.1 or 9.5 Domain MRS MRS Informatica Domain 24

25 Migration/Upgrade Considerations Data Quality Migration/Upgrade with PowerCenter Integration IDQ local repositories IDQ Server repository Single Informatica 9.1 or 9.5 Domain Migration ClientPackage ServerPackage XML Import Informatica Domain MRS * Requires separate domain for upgrade or VMware Upgrade MRS Informatica 9.1 or 9.5 Domain Informatica 9.1 or 9.5 Domain PowerCenter Repository MRS Export upgraded MRS and Content Run CRS utility - allows DQ plans to run in 9.x environment Export DQ mappings requiring upgrade AV transformation deprecated components 25

26 Migration/Upgrade Considerations Data Quality Migration/Upgrade with PowerCenter Integration Separate DQ 9.x Domain and PowerCenter 9.x Domain IDQ local repositories IDQ Server repository Migration ClientPackage ServerPackage XML Import Informatica 9.1 or 9.5 Domain PowerCenter Repository Run CRS utility - allows DQ plans to run in 9.x environment MRS Informatica Domain MRS Export DQ mappings requiring upgrade AV transformation deprecated components Informatica DQ 9.1 or 9.5 Domain 26

27 Migration/Upgrade Considerations Create Migration/Upgrade project plan In order to manage the IDQ migration/upgrade project it is crucial to identify and understand all task which will be executed in order to complete the project. Clearly define and list all steps required for the multi-step migration and multi-version upgrade of IDQ. Allocate adequate time for IDQ re-development and testing after upgrade. Allocate adequate time for PowerCenter/DQ integration testing Include V9 product training as part of the plan 27

28 Migration Pre-requisites Informatica needs to be properly installed, configured, and running before starting the migration process IDQ is at version Earlier versions cannot be directly migrated to Informatica Ensure Java version is current Gather all required login and connection information Review the migration steps and create a project plan Download and extract IDQMigration.zip 28

29 3 Step Migration Process IDQ Client Client Package Export IDQ plans from IDQ repository Identify connection details Gather local directories Package data for next step Server Server Import Unpack data from Client Package Create connections Import dictionary data into Reference Table Manager Convert plans to XML 9.01 Client XML Import Import XML from Server Import into Repository 29

30 Client Package Executing the ClientPackage will create the Package and Stage folders along with the corresponding logs 30

31 ClientPackage - Report <MigrationPackageLocation>/Package/PackageReport.html Identify Dictionaries used by plans and dictionaries that exist but are not used by any plan Database Connections used by plans. One entry for every DSN/Username/Password combination 31

32 ServerImport Pre-requisites (perform in Developer Tool) Create new blank project for mappings to be imported to Create new folder for imported reference tables Install Informatica Content packages in shared project Server Import Unpacks data from ClientPackage (MigrationPackage.zip) Creates connections Import dictionary data into Reference Table Manager Converts Plans to Mapping XML 32

33 ServerImport Summary/Overview Report Overall status of conversion Links to detail / individual reports Default location <MigrationPackageLocation>/migration_reports 33

34 ServerImport Detail Reports One Detail report per plan/9.0.1 mapping Component / Port level detail Includes warnings / errors Default location <MigrationPackageLocation>/migration_reports 34

35 Client XML Import Using Developer tool, import the MigratedMappings.xml file produced during the ServerImport process. 35

36 Post Migration Tasks Validate all DQ mappings Make any required changes to get mappings to validate do not start re-development effort at this time 36

37 Upgrade Process Stand Alone Data Quality Upgrade repository to most current version of Informatica 9.1 or 9.5 PowerCenter and Data Quality Integration Single Domain for PowerCenter and Data Quality Upgrade 9.0.1repository to current domain version Separate Domain for PowerCenter and Data Quality Upgrade repository to most current version of Informatica Run CRS Utility to enable plans to continue running in PowerCenter 9.1 or 9.5 environments If new PowerCenter install and repo upgrade, the 8.6 DQ binaries need to be accessible to new environment (either moved or referenced) 37

38 Post Upgrade Tasks Re-develop DQ mappings with deprecated components During migration deprecated components convert to expression, modify mappings to convert to appropriate logic Address Validation some output fields are not available in the 9.x releases. Status codes in 9.x are different from codes. Validate output for all mappings in Developer tool to ensure desired results are achieved 38

39 Post Upgrade Tasks Export modified DQ mappings to PowerCenter Only mappings which contain deprecated components or Address Validation transformation must be re-exported Verify PowerCenter mappings which utilize Address Validation match status codes are producing desired results Perform any necessary code changes to PowerCenter mappings utilizing updated DQ mappings QA results from PowerCenter mappings to ensure desired output is being achieved with updated DQ logic 39

40 IDQ Migration Reference Documents Documentation is available on the Informatica website IN_901_DQ_Repository_Migration_Guide_en Installation and configuration documentation for current version is also available at product documentation 40

41 Informatica Velocity Methodology Velocity is the blueprint for delivering efficient and successful Informatica solutions that solve business problems. New Website New search capability Filtering/viewing content by project type project phase or other tags New accelerator tools Hot links between the articles Access at: mysupport.informatica.com Visit the Informatica Pavilion at the Technology and Solutions Fair for more details. Check out more than 100 new articles! 41

42 Questions? 42

43 Appendix A Component to Transformation Conversion 43

44 Component to Transformation conversion Component Component Aggregation Aggregator transformation Association [for PowerCenter] Association transformation Bigram Comparison transformation Character Labeler Labeler transformations Consolidation [for PowerCenter] Consolidation transformation Context Parser Count CSV Dual Match Source CSV Identity Group Source CSV Match Sink CSV Match Source CSV Merge Sink CSV Sink CSV Source DB Identity Group Source DB Match Source DB Report Sink DB Sink Labeler and Parser transformations. The Parser is set to pattern-based parsing mode. Mapplet containing Aggregator, Union, Expression, Joiner, Sorter, and Filter transformations Two file-based data sources and a Match transformation File-based data source and Match transformation. May convert to a mapplet. File-based data target File-based data target and Match transformation File-based data target File-based data target File-based data source Relational data source and Match transformation. May convert to a mapplet. Relational data source and Match transformation File-based data target SQL transformation and relational data target 44

45 Component to Transformation conversion Component Component DB Source Dual Group Source Edit Distance Fixed Width Sink Fixed Width Source Global AV [Address Doctor engine] Global AV [Melissa Data engine] Global AV [QAS engine] Global AV [SDK] Group Sink Group Source Hamming Distance Identity Group Target Identity Match Jaro Distance Match Key Sink Relational data source Multiple file-based data sources and Union transformation if required. May convert to a mapplet Comparison transformation File-based data target File-based data source Address Validator transformation. This transformation needs additional configuration following import to the Model repository. Address Validator transformation. This transformation needs additional configuration following import to the Model repository. Address Validator transformation. This transformation needs additional configuration following import to the Model repository. Not supported Flat-file data target and Sorter and Expression transformations Multiple file-based data sources and Union transformation if required. May convert to a mapplet. Comparison transformation Mapplet output transformation Match transformation Comparison transformation Relational data target 45

46 Component to Transformation conversion Component Component Merge MinAvgMax Merge transformation Mapplet containing Aggregator, Union, Expression, Joiner, and Router transformations Missing Values Mixed Field Matcher Normalization [SDK] NYSIIS Parsing [SDK] Profile Standardizer Range Counter Realtime Sink Realtime Source Report Sink Rule Based Analyzer SAP Sink SAP Source Scripting Search Replace Similarity [SDK] Soundex Mapplet containing Aggregator, Expression, Joiner transformations Not supported Not supported Key Generator transformation Not supported Parser transformation. The Parser is set to pattern-based parsing mode. Linear Range: Aggregator, Expression, Joiner, and Sorter transformations Variable Range: Aggregator, Expression, Union transformations Mapplet containing data target Mapplet containing data source Flat-file data target Decision transformation Not supported Not supported Not supported Standardizer transformation Not supported Key Generator transformation 46

47 Component to Transformation conversion Component Component Splitter Sum To Upper Token Labeler Token Parser Weight Based Analyzer Word Manager Labeler, Parser, and Expression transformations. The Parser is set to patternbased parsing mode. Mapplet containing Aggregator, Expression, Joiner, Sorter, and Union transformations Case Converter transformation Labeler transformation Parser transformation Weighted Average transformation Standardizer transformation 47