Turbo charging SAS Data Integration using SAS In-Database technologies Paul Jones

Transcription

1 Turbo charging SAS Data Integration using SAS In-Database technologies Paul Jones

2 Agenda What is SAS In-Database Why do we do Who are we working with.. When? How.does it work

3 What is SAS In-Database? Integration SAS Applications are integrated to leverage standard database features. Examples Database Specific SQL SQL functions Stored Procedures In-Database The ability to embed and use SAS functions, framework, processes and applications inside the database. Examples SAS Format function SAS Scoring functions Predictive Modeling Functions

4 Why? Value Proposition Capability Value SAS In-Database Streamline Analytic Workflow Minimize data preparation Accelerate data discovery Decrease time to value Scalability and Performance Reduce data movement Leverage MPP systems for parallelization Data Consistency Reduce Data Redundancy Reduce Information Latency Fit for IT Enable Data Governance Increase Hardware Utilization Integrate with Resource Management Facilitate standardization on a single enterprise analytics platform

5 Who: SAS In-Database Partners Active IBM DB2 IBM Netezza Oracle Teradata / Aster Data EMC (GreenPlum)

6 When: SAS In-Database Available today Teradata SAS Scoring Accelerator for Teradata SAS Analytics Accelerator for Teradata (LA) Base SAS Procedures for Teradata SAS/Access Interface for Teradata format publishing capability Aster Data SAS Scoring Accelerator for Aster ncluster (LA) Base SAS Procedures for DB2 IBM Netezza SAS Scoring Accelerator for Netezza SAS/Access Interface for Netezza format publishing capability IBM DB2 SAS Scoring Accelerator for DB2 Oracle Base SAS Procedures for Oracle EMC Greenplum SAS Scoring Accelerator for Greenplum (LA)

7 When: SAS In-Database 9.3 Q Teradata SAS Scoring Accelerator for Teradata (4.1) SAS Analytics Accelerator for Teradata (GA) Aster Data SAS Scoring Accelerator for Aster ncluster SAS/Access Interface for Aster ncluster format publishing capability IBM DB2 SAS/Access Interface for IBM DB2 format publishing capability EMC Greenplum SAS Scoring Accelerator for Greenplum (GA) SAS/Access Interface for Greenplum format publishing capability

8 How - SAS Procedures and DBMS Tables (Review) SAS/Access Libname Engine: Any SAS Program may reference data residing in a database SAS/Access simulates SAS I/O for SAS Procedures A SQL select statement is generated for data retrieval, honoring column selections, sample options and where-clauses (incl function mapping etc.). Any further execution logic of the SAS procedures is not transformed to SQL and pushed down for common DBMS. Libname tera TERADATA server=tdpid user=uid password=pwd; proc XYZ data=tera.cdr_agg1; where substr(cust_id,1,2) = "04"; Run; TERADATA_2: Executed: on connection 1 SELECT "CUST_ID","START_CALL_DT","usage_type","call_dur_sum" FROM "cdr_agg1" WHERE (SUBSTR("CUST_ID", 1, 2) = '04' ) TERADATA: trget - rows to fetch: 720

9 HOW SAS In-Database Processing Overview Process Flow Diagram Procedure PROC FREQ PROC MEANS PROC RANK PROC REPORT PROC SORT PROC SUMMARY PROC TABULATE

10 Running SAS In-Database Procedures To run in-database procedures, these actions must be taken: 1. The SQLGENERATION system option or the SQLGENERATION LIBNAME option must be set to DBMS. By default, the SQLGENERATION system option is set to NONE and the in-database procedures are run using conventional SAS processing, not inside the database. Conventional SAS processing is also used when specific procedure statements and options do not support in -database processing. 2. The LIBNAME statement must point to a valid version of the DBMS: DB2 UDB9.5 Fixpack 7 running only on AIX or LINUX x64, Oracle 9i Netezza 5.0 or higher, Teradata server running version 12 or higher for Linux

11 How do I know it s working? The SASTRACE option reveals the SQL query passed to a DBMS through a SAS/ACCESS engine by an enabled procedure when in-database operation is occurring. OPTIONS sastrace=',,,d' sastraceloc=saslog nostsuffix; /* Invoke procedure here! */ OPTIONS sastrace=off; Look for: TERADATA_2: Executed: on connection 2 SELECT "Origin" FROM sas."cars" TERADATA: trget - rows to fetch: 428 Note: turning on SASTRACE tracing for an entire session can produce unwanted information that crowds the SAS log!

12 Why SAS Programmers love SAS! proc rank data=indb.order_item out=work.order descending ties=low; var quantity product_id; ranks QuantityRank ProductRank; run; WITH "subquery0" ( "COSTPRICE_PER_UNIT", "DISCOUNT", "ORDER_ID", "ORDER_ITEM_NUM", "PRODUCT_ID", "QUANTITY", "TOTAL_RETAIL_PRICE" ) AS ( SELECT "COSTPRICE_PER_UNIT", "DISCOUNT", "ORDER_ID", "ORDER_ITEM_NUM", "PRODUCT_ID", "QUANTITY", "TOTAL_RETAIL_PRICE" FROM "DB2_ORDER_ITEM" ) SELECT "table0"."order_id", "table0"."order_item_num", "table0"."product_id", "table0"."quantity", "table0"."total_retail_price","table0"."costprice_per_unit", "table0"."discount", "table2"."rankalias1" AS "QUANTITYRANK","table1"."rankalias0" AS "PRODUCTRANK" FROM "subquery0" AS "table0" LEFT JOIN ( SELECT DISTINCT "PRODUCT_ID", "tempcol0" AS "rankalias0" FROM ( SELECT "PRODUCT_ID", MIN( "tempcol1" ) OVER (PARTITION BY "PRODUCT_ID" ) AS "tempcol0" FROM ( SELECT "PRODUCT_ID", CAST( ROW_NUMBER() OVER (ORDER BY "PRODUCT_ID" DESC ) AS DOUBLE PRECISION ) AS "tempcol1" FROM "subquery0" WHERE ( ("PRODUCT_ID" IS NOT NULL ) ) ) AS "subquery2" ) AS "subquery1" ) AS "table1" ON ( ( "table0"."product_id" = "table1"."product_id" ) ) LEFT JOIN ( SELECT DISTIN CT "QUANTITY","tempcol2" AS "rankalias1" FROM ( SELECT "QUANTITY", MIN( "tempcol3" ) OVER ( PARTITION BY "QUANTITY" ) AS "tempcol2" FROM ( SELECT "QUANTITY", CAST( ROW_NUMBER() OVER ( ORDER BY "QUANTITY" DESC ) AS DOUBLE PRECISION ) AS "tempcol3" FROM "subquery0" WHERE ( ( "QUANTITY" IS NOT NULL ) ) ) AS "subquery4" ) AS "subquery3" ) AS "table2" ON ( ( "table0"."quantity" = "table2"."quantity" ) )

13 Using SAS Formats indb SAS PROC SQL support use of custom formats to enable custom aggregation rules onthe-fly in-db proc sql; SELECT distinct state_id FROM TDOrion.state where put( State_Code, $REGION12.)='WEST' ; quit; SQL_IP_TRACE: pushdown attempt # 1 SQL_IP_TRACE: passed down query: select distinct TXT_1."State_ID" from "sasorion"."state" TXT_1 where sas_put( State_Code, $REGION12. )= 'WEST' ;,

14 SAS Procedure SQL- and Format-Push Down SAS Procedures (inc SQL) support use of formats to enable custom aggregation rules on -the-fly. proc freq data=tdorion.order_fact ; format state_code $Region.; tables order_type*state_code; where employee_id=120444; run; Analyse frequency of order_types by region instead of by state. In SAS Procedure SQL push-down supports on-the-fly aggregation using SAS Formats. In this way, customers can still create and use formats on the fly AND have in -database processing. Business critical, high value formats (extreme 1:n relations) can be published to DB to provide an extra performance lift. If a format is not found in the database, raw value processing is substituted and formatting is deferred until the results ar e returned to SAS.

15 SAS Procedure SQL- and Format-Push Down Example SAS FREQ Procedure and Formats Options SQL_IP_TRACE=SOURCE; proc freq data=tdusr.cm_party; format addr_line_4_txt $REGION.; table addr_line_4_txt * gender_cd; where deceased_flg='n' and contactable_flg = 'Y' and (balance_chequing_acct_amt > or balance_savings_acct_amt > ); Run; PROC SQL; CREATE VIEW WORK._ZSQL0 AS SELECT COUNT(*) AS ZSQL1, CASE WHEN (COUNT(*) > COUNT(ADDR_LINE_4_TXT)) THEN ' ' ELSE MIN(ADDR_LINE_4_TXT) END AS ZSQL2, CASE WHEN (COUNT(*) > COUNT(GENDER_CD)) THEN ' ' ELSE MIN(GENDER_CD) END AS ZSQL3 FROM TDUSR.cm_party WHERE ((deceased_flg='n') and (contactable_flg='y') and (balance_chequing_acct_amt >15000) or (balance_savings_acct_amt>15000))) GROUP BY PUT(ADDR_LINE_4_TXT, $REGION.), GENDER_CD; QUIT; NOTE: SQL view WORK._ZSQL0 has been defined. SQL_IP_TRACE: passed down query: select COUNT(*) as "ZSQL1", case when COUNT(*) > COUNT(TXT_1."ADDR_LINE_4_TXT") then ' ' else MIN(TXT_1."ADDR_LINE_4_TXT") end as "ZSQL2", case when COUNT(*) > COUNT(TXT_1."GENDER_CD") then ' ' else MIN(TXT_1."GENDER_CD") end as "ZSQL3" from "sas_usr"."cm_party" TXT_1 where (TXT_1."DECEASED_FLG" = 'N') and (TXT_1."CONTACTABLE_FLG" = 'Y') and ((TXT_1."BALANCE_CHEQUING_ACCT_AMT" > 15000) or (TXT_1."BALANCE_SAVINGS_ACCT_AMT" > 15000)) group by cast(sas_put(txt_1."addr_line_4_txt", '$REGION12.0') as char(12)),txt_1."gender_cd" NOTE: SQL generation will be used to perform the initial summarization.

16 SAS to Database Function Mapping The SAS/ACCESS SQL generation engine maps specific functions used in where-clause in any Procedure/Data Step or used in Proc SQL programs into their database equivalent. The list of default functions mapped has been extended in SAS 9.2 M2 and furthermore the list is no longer static and can be customized To exclude functions from the mapping list or to include mapping of SAS functions to Database functions or UDFs Specific Libname options enable dynamic allocation of funtions: SQL_FUNCTIONS, SQL_FUNCTIONS_COPY

17 Enterprise Guide 4.2 & 4.3 SAS In-Database Awareness: Query Builder EG 4.2 uses implicit SQL pass through In the Query Builder Task Tools Validate Changes Now Allows user to check query prior to running to see if the SQL will be pushed down or NOT. EG 4.3 Allows knowledgeable users to use explicit SQL pass through In the Query Builder Task Options Options for this query Pass Through = generate PROC SQL code to force SQL to be pushed into DBMS (i.e. Connect to DBMS as XXX and Disconnect from XXX) Furthermore with this feature if there is an issue with using a function or something else that prevents in -database processing the query will fail immediately v.s. implicit which would attempt to pull the data out of the DBMS and execute the query in SAS

18 Data Integration Studio from Features UI Enhancements (Visual Indicators) SAS Table Indicator Teradata Table Indicator Teradata Push-down Indicator Validate push-down processing for DB s. Source/Target Table indicators Transformation Push-down indicators

19 Resources Good Introduction paper: SAS Presents In-Database Base Procedures in Working with a DBMS using SAS Enterprise Documentation: 9.2 SAS/ACCESS 9.2 for Relational Databases: Reference, Fourth Edition, Nov SAS 9.3 In-Database Products User s Guide

20