1 Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Size: px

Start display at page:

Emory Pearson
6 years ago
Views:

2 Subsetting and Masking: Advanced Techniques for Test Management Jagan R. Athreya, Director, base and Exadata Manageability Derek Messie, Senior DBA, Cornell University 2 Copyright 2011, Oracle and/or its affiliates. All rights reserved. Presenting with

3 Program Agenda Costs of production data in test environments Application data model and data discovery subsetting masking PeopleSoft data masking: a case study by Cornell U. 3 Copyright 2011, Oracle and/or its affiliates. All rights reserved.

4 Program Agenda Costs of production data in test environments Application data model and data discovery subsetting masking PeopleSoft data masking: a case study by Cornell U. 4 Copyright 2011, Oracle and/or its affiliates. All rights reserved.

2/3 rd of Sensitive and Regulated Resides in bases and growing exponentially BUT IS IT SECURE?

Challenges in Setting up Test Systems Cannot use sensitive data in test without obfuscation Producing relationally intact subset is hard but necessary Full

6 Challenges in Setting up Test Systems Cannot use sensitive data in test without obfuscation Producing relationally intact subset is hard but necessary Full production copies for test systems not cost effective Test System Setup Error-prone, manual process 6 Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Complexity of Enterprise Applications Financials

Customer Relationship Management z What is the

used across the Where are these sensitive data

7 Complexity of Enterprise Applications Financials Supply Chain Management Human Capital Management Customer Relationship Management z What is the relationship between tables across the various applications? What are the data extraction rules that produces referentially intact subsets? Will the subsetted data fit into the available storage for test systems? What are the various types of sensitive data used across the applications? Where are these sensitive data stored and how are they related to each other? 7 Copyright 2011, Oracle and/or its affiliates. All rights reserved.

9 Program Agenda Costs of production data in test environments Application data models and data discovery subsetting masking PeopleSoft data masking: a case study by Cornell U. 9 Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Discovery and Modeling Application Models Scans application schemas to model

Applications meta-data Oracle ebusiness Suite Oracle Fusion Applications Store

as data subsetting, masking Relationship Modeling Sensitive Identification Test

10 Discovery and Modeling Application Models Scans application schemas to model relationships between tables and columns Extract data relationships from Oracle Applications meta-data Oracle ebusiness Suite Oracle Fusion Applications Store referential relationships stored in repository Enables test data operations such as data subsetting, masking Relationship Modeling Sensitive Identification Test System Setup Subsetting Masking 10 Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Discovery and Modeling Sensitive Identification Sensitive

pre-built mask templates Masking Templates for Oracle

Sensitive Identification Test System Setup Subsetting

11 Discovery and Modeling Sensitive Identification Sensitive data discovery Pattern-based database scanning Import from pre-built mask templates Masking Templates for Oracle Applications Fusion Applications Relationship Modeling Sensitive Identification Test System Setup Subsetting Masking 11 Copyright 2011, Oracle and/or its affiliates. All rights reserved.

12 Sensitive Column Discovery Discover the sensitive column using patterns Create Sensitive Column Types using Regular Expression to define the pattern for column name, comment, or data Run a discovery job to scan the database and discover the sensitive columns Import from masking templates Import the masking templates for Oracle Applications such as Fusion Applications Sensitive columns will be automatically tagged Set columns as sensitive Manually Add columns directly in the sensitive column list 12 Copyright 2011, Oracle and/or its affiliates. All rights reserved. 12

13 Program Agenda Costs of production data in test environments Application data models and data discovery subsetting masking PeopleSoft data masking: a case study by Cornell U. 13 Copyright 2011, Oracle and/or its affiliates. All rights reserved.

14 Subsetting What? A relationally intact and yet fractional representation of production data for test and development purposes Why? Reduce the storage overhead created by production data copies in various application z environments Allow developers to perform real world application development by using production-class data Production Application data Test Application data Relationship Modeling Sensitive Identification Test System Setup Subsetting Application metadata Subset criteria: REGION = NORTH AMERICA AND FISCAL_YEAR = 2009 Application metadata Masking 14 Copyright 2011, Oracle and/or its affiliates. All rights reserved.

15 Select Applications for Subset Relationship Modeling Sensitive Identification Test System Setup Subsetting Masking Financials Supply Chain Management Human Capital Management Customer Relationship Management z Identifies the tables associated with the application(s) Transfers relationships between tables from ADM into Subset Definition Transfer additional metadata from the ADM into Subset Definitions Constraints, Indexes 15 Copyright 2011, Oracle and/or its affiliates. All rights reserved.

16 Define Subset Parameters Relationship Modeling Sensitive Identification Test System Setup Subsetting Masking Time (FY:2011) Dimension (Region:Asia) Space (Size:10%) Analyzes dependencies between tables to select the subset driver tables by traversing down and across the ADM Defines extraction (WHERE) clause to select the rows for each of the tables that will included in the subset Analyzes table statistics to estimate size of the database generated from the specified subset parameter 16 Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Terabyte 200G (20%) 1 hour 8 minutes Clone and delete 1 Terabyte 200G (20%) 5 hours 49 minutes 17 Copyright 2011,

17 Subsetting - High Performance Execution Production Export = Writing subset data via Pump Import Test pump Export file Production Clone In-Place subset = Deleting data in the same database Test base size Subset size Time* Pump method 1 Terabyte 200G (20%) 1 hour 8 minutes Clone and delete 1 Terabyte 200G (20%) 5 hours 49 minutes 17 Copyright 2011, Oracle and/or its affiliates. All rights reserved. *2-nodes Intel Xeon 6-core X5675 Processor w/ 216G memory running OEL 5.5

Subsetting: End to End Process Production HR.

JOBS JOB_ID JOB Min_SAL Table rule: Min_Sal < 20,000

EMPLOYEES NAME JOB_ID SALARY AGUILAR SA_MAN 40000 HR.

10000 SA_REP Sales Repres 20000 Create Application Model Create

18 Subsetting: End to End Process Production HR.EMPLOYEES NAME JOB_ID SALARY AGUILAR SA_MAN BENSON SA_REP HR.JOBS JOB_ID JOB Min_SAL HR.EMPLOYEES NAME JOB_ID SALARY Table rule: Salary< 60,000 HR.JOBS JOB_ID JOB Min_SAL Table rule: Min_Sal < 20,000 Test/Staging HR.EMPLOYEES NAME JOB_ID SALARY AGUILAR SA_MAN HR.JOBS JOB_ID JOB Min_SAL SA_MAN Sales Mgr SA_MAN Sales Mgr SA_REP Sales Repres Create Application Model Create Subset Definition Create Test base Extract Subset: 2 methods Extract and import Clone and delete Schemas Tables Relationships collected EM Schemas Tables Relationships retrieved

19 Program Agenda Costs of production data in test environments Application data models and data discovery subsetting masking PeopleSoft data masking: a case study by Cornell U. 19 Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Secure Test System Deployment Masking Production LAST_NAME SSN SALARY AGUILAR

Subsetting LAST_NAME SSN SALARY SMITH 111-23-1111 60,000 Masking BENSON 323-22-2943

Extensible template library and policies for automation Sophisticated masking:

Heterogeneous Masking NEW in EM 11g: Pre- and Post-mask commands and command line

20 Secure Test System Deployment Masking Production LAST_NAME SSN SALARY AGUILAR ,000 Relationship Modeling Test Sensitive Identification Test System Setup Subsetting LAST_NAME SSN SALARY SMITH ,000 Masking BENSON ,000 MILLER ,000 Deploy secure test system by masking sensitive data Extensible template library and policies for automation Sophisticated masking: Condition-based, compound, deterministic Integrated masking and cloning NEW in EM 11g: Heterogeneous Masking NEW in EM 11g: Pre- and Post-mask commands and command line (EMCLI) support NEW in EM 12c: Masking integration with Real Application Testing NEW in EM 12c: Key-based reversible masking

21 Oracle Masking Comprehensive and Extensible Mask Library Mask formats for common sensitive data Accelerates solution deployment of masking Extensible mask routines Enables customization of business rules Define once, apply everywhere Ensures consistent enforcement of policies 21 Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Oracle Masking Sophisticated Masking Techniques Conditionbased Masking

g. Address, City, State, Zip, Phone Condition-based Masking Specify

driver s license format for each state SQL-expression based masking Use

22 Oracle Masking Sophisticated Masking Techniques Conditionbased Masking Compound Masking Compound Mask Sets of related columns masked together e.g. Address, City, State, Zip, Phone Condition-based Masking Specify separate mask format for each condition, e.g. driver s license format for each state SQL-expression based masking Use SQL functions, e.g. UPPER, SUBSTR, TO_CHAR, to generate mask values, e.g. SUBSTR(%ORIG_VALUE%,1,3)

23 Heterogeneous Masking Enterprise Manager Cloujd Control with Masking Oracle bases manage manage manage Production (Oracle) Staging (Oracle) Test (Oracle) Production (non-oracle) Staging (Oracle) Test (non-oracle) Non-Oracle bases base Gateway base Gateway monitor manage monitor Relationship Modeling Sensitive Identification Test System Setup Enterprise Manager Cloud Control with Masking Subsetting Masking Available for IBM DB2, Microsoft SQLServer, Sybase

Integrating ODI using EMCLI & pre- post- mask XML masking with Oracle Integrator Flat file masking with Oracle Integrator Incremental masking using Oracle

Masking Mask using Oracle Masking Mask Using Oracle Masking Post-Mask Write masked data back to XML from tables using DTD Write masked data back to flat file

24 Integrating ODI using EMCLI & pre- post- mask XML masking with Oracle Integrator Flat file masking with Oracle Integrator Incremental masking using Oracle Integrator Pre-Mask Read XML and DTD into tables Reverse engineer flat file into tables Listen for journaled records and write to tables Mask Mask using Oracle Masking Mask using Oracle Masking Mask Using Oracle Masking Post-Mask Write masked data back to XML from tables using DTD Write masked data back to flat file from tables using reverseengineered structure Write masked incremental transactions to Test DB 24 Copyright 2011, Oracle and/or its affiliates. All rights reserved. 24

Key-based Unmask Reverse the masked data back to its original value with

25 Key-based Reversible Masking New in EM 12c Mask format is preserved Any pattern and any length Numeric, alphanumeric or mixed Deterministic Unique Key-based Unmask Reverse the masked data back to its original value with the same key 25 Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Secure base Testing New in EM 12c End-to-end testing with real workloads

SQL unit testing for response time Identify and tune regressed SQL Integrated

Replay Load, performance testing for throughput Remediate application

superior comprehensive testing solution Seamless integration with Masking to

26 Secure base Testing New in EM 12c End-to-end testing with real workloads Create Test System Capture Workload SQL Performance Analyzer Replay Workload SQL unit testing for response time Identify and tune regressed SQL Integrated into SQL Tuning Advisor and SQL Plan Baseline Deploy Replay Clients base Replay Load, performance testing for throughput Remediate application concurrency problems Integrated with Oracle Application Testing Suite for superior comprehensive testing solution Seamless integration with Masking to preserve data privacy compliance 26 Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Masking Real Application Testing Workloads Real Application Testing Integration with Masking Copying production data to test systems puts sensitive information at risk Perform secure,

27 Masking Real Application Testing Workloads Real Application Testing Integration with Masking Copying production data to test systems puts sensitive information at risk Perform secure, production-scale testing Sensitive data found in workload capture files and STS are masked along with application data STS bind data (used with SPA) Workload Capture files (used with DB Replay) AWR sensitive bind data is purged Consistent masking across all data sources and workloads 27 Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Masking Internals of RAT workloads Real Application Testing routine called from data masking script to Initialize Sensitive columns are introspected from masking definition All identified sensitive

28 Masking Internals of RAT workloads Real Application Testing routine called from data masking script to Initialize Sensitive columns are introspected from masking definition All identified sensitive binds in STS are extracted and placed into masking staging tables All identified sensitive binds in base Replay capture files are extracted and placed into masking staging tables Build Mapping tables containing original sensitive and masked values using masking routines

29 Masking Definition Compatibility and Restrictions SQL Expression and Conditional format masking in masking definition not compatible base Replay capture files and SQL Tuning Sets may not contain entire data set Conditions may be based on values outside the workload Masking of literals in base Replay capture files and SQL Tuning Sets is not supported It is not a best practice to store sensitive data as literals.

30 Program Agenda Costs of production data in test environments Application data models and data discovery subsetting masking PeopleSoft data masking: a case study by Cornell U. 30 Copyright 2011, Oracle and/or its affiliates. All rights reserved.

31 Cornell Presentation ERP Masking Environment Overview Cornell OEM/ERP Environment Need for Masking at Cornell Key Project Tasks Challenges/Lessons Learned

32 Cornell Cornell OEM/ERP ERP Masking Environment Grid Control version Targets 181 bases (Mostly version , moving to 11.2) Peoplesoft Campus Solutions v8.9; including Contributor Relations, Student Administration, Admissions, Financial Aid, HR, Payroll and ELM. Marts Derek Messie - Cornell University Cornell University Oracle Open World October 6, 2011

33 Why Does Cornell Cornell ERP Masking Need Environment Masking? Production data often used in non-production environments Development Training QA External File Processing External file feeds to health, government or academic agencies. Breach Incident Identity theft protection services for all those potentially effected. Credit rating reports.

34 Cornell Masking ERP Masking Project Environment Scope The data masking project is focused on data in the PeopleSoft environment for phase 1. masking does NOT Impact production Non production databases impacted by project: All development environments All training environments All QA environments All Non production environments with Cornell data Derek Messie - Cornell University Cornell University Oracle Open World October 6, 2011

35 Cornell Masking ERP Masking Project Environment Scope Implement data masking for the following data elements in ALL non-production transactional and reporting systems. Social Security Number (National ID) Credit Card Number Driver s License Bank Account Number Patient Treatment Information Derek Messie - Cornell University Cornell University Oracle Open World October 6, 2011

36 Masking Cornell Requirements ERP Masking Environment / Constraints Element Social Security Number Found in PS Tables as National ID, SSN, etc. data columns. Masking Pack Algorithm : TableColumns Bank Account Number Masking Pack Algorithm : Substitute Masking Requirements / Constraints Non-blank and non-null values are masked with a unique random value is consistently masked for the person entity --- that is: the source value is masked to the same masked value in all locations across the PS database Notes: 1. There is a diminutive chance that a random value generated results in a value that is currently a valid SSN in the PS database; however, the masking process ensures that the masked value is not assigned to the person entity who owns the actual (original) value. 2. The random value generated abides by the PS edit rules where the following value(s) are not permitted: 000*, 666*, 700*-728*, 8*, 9*, 999*, , or The random value generated will not result in a value in the range of: through As Non-blank this block and of values non-null is reserved values so are that masked developers with a or random business value functional testers can set-up new person entities in the test db and use an SSN or National ID value in this range. is consistently masked --- that is: the source value is masked to the same masked value in all locations across the PS database Notes: 1. There is a diminutive chance that a random value generated results in a non-unique value. This could result in bank account numbers, which were separate and distinct in the original data, to now be masked to the same bank account number values (i.e. combined) in the masked data set. Derek Messie - Cornell University Cornell University Oracle Open World October 6, 2011

37 Masking Cornell ERP Algorithms Masking Environment Shuffle Fixed String Fixed Number Substring Null Values Random Strings Random Numbers Random Digits Random Dates Preserve Original Substitute Deterministic (hash function - i.e., values generated are consistent) Does NOT guarantee value uniqueness. Table Column Deterministic (table look-up) Can be used to generate unique values. User Defined Function Post-Processing Function

38 Cornell Basic ERP Masking Process Environment Flow Process - Overview Setup Execution pre-processing post-processing Establish Masking Rules Define Meta in OEM Masking Pack Oracle Masking Pack Masking Script (OEM) Schedule Job in OEM Masked base Gold Copy NONPROD NONPROD NONPROD NONPROD NONPROD NONPROD Derek Messie - Cornell University Cornell University Oracle Open World October 6, 2011

39 Challenges Cornell ERP Masking / Lessons Environment Learned Change for functional & technical groups External File Processing Embedded SSN s in text fields Validation Were values masked consistently? Are they unique? scripts to check row counts, distinct values. Derek Messie - Cornell University Cornell University Oracle Open World October 6, 2011

40 Cornell Acknowledgements ERP Masking Environment Thanks To Cornell University Tony Damiani Assistant Director (Business Intelligence) Gary Chaika DBA (Applications Middleware) Oracle Corporation Jagan Athreya - Director, base and Exadata Manageability Derek Messie - Cornell University Cornell University Oracle Open World October 6, 2011

Welcome to the Oracle Enterprise Manager Technical Webcast Series

Welcome to the Oracle Enterprise Manager Technical Webcast Series Please ensure that you are connected to the audio portion of this event which is available by dialing: Dial-in US/Canada : ( 888 ) 794-4691