10/16/2011. Disclaimer. SAS-Teradata Integration & Value to the SAS User. What we ll cover. What is SAS In-Database?

Transcription

1 Disclaimer SAS-Teradata Integration & Value to the SAS User I represent SAS Institute Inc. and nothing I state should be interpreted as an endorsement on behalf of CMS. The opinions expressed are my own and do not reflect the view of the Centers for Medicare & Medicaid Services, the Department of Health and Human Services, or the United States government. Paul Grant Senior Technical Architect SAS Federal, LLC 2 What is SAS In-Database?» Pass-through The ability to embed and use SAS functions, framework, processes and applications inside the database. No large scale copying of data from DB to analytic system, increasing performance and reducing network traffic Highly scalable, exploiting the full parallel power of Teradata to scale SAS processing Can be leveraged in a wide range of analytic capabilities» Data loading and preparation» Exploratory Data analysis» Advanced analytics» Reporting SAS Analytics Specialized SAS Processing Large Result Set Data Warehouse SAS Analytics Traditional Processing Small Result Specialized SAS Processing Teradata Warehouse 3 4 What is SAS In-Database? Moves the data to SAS Moves SAS to the database Traditional Processing What is SAS In-Database? Database Integration SAS Analytics Specialized SAS Processing Large Result Set SAS Analytics Small Result SAS applications are integrated to leverage standard features of the database. SAS functions, framework, processes and applications are embed and used inside the database. Specialized SAS Processing Data Warehouse All data copied across the network Data summarized by SAS Teradata Warehouse Data is summarized by Teradata Summarization is done in parallel Examples Native Access Engines Database Specific pass-through Pushdown of key Base SAS Procedures Examples SAS Format function Analytic SAS Procedures SAS Scoring functions Predictive Modeling functions Only summarization sent to SAS 5 6 1

2 In-Database Procedures Why In-Database? Database Integration Proc FREQ View SAS Server Proc ACCESS Engine DB supported execute plan execute execute Proc PRINCOMP View SAS Server Proc ACCESS Engine DB supported execute plan execute execute Improve Performance and Scalability Leverage MPP architecture for parallelization and scalability Push data-intensive work to database Reduce data movement Increase utilization of RDMS infrastructure and resources Improve Data Governance Integrate with the R security and metadata Preserve SAS programmer knowledge and skills Run existing SAS code with little or no modification sas_sscp() Installed SAS functions 7 8» Pass-through SAS/ACCESS Engines Provide the ability to read and write data (including bulk loading) and execute database procedures Connect through database client API» Native vs ODBC Enables Pass-Through» Translates SAS into database specific» Maps many SAS functions to database functions Can surface data from both local and remote database servers Do not support DATA Step pass-through (Why??)» Except for the WHERE option/statement 9 10 SAS/ACCESS: Pass-Through SAS/ACCESS: Pass-Through Example Two types: Implicit Pass-Through» Passes SAS PROC to the database» Connects using a LIBNAME statement» Knowledge of database not required» Transparent to the user» Easiest Explicit Pass-Through» Passes database-specific to the database» Connects using a connection string in PROC» Knowledge of the database and its required» Provides greater flexibility and control over what is passed to the database. 11 options GENERATION= MSGLEVEL=I sastrace=',,,d' sastraceloc=saslog nostsuffix; /* Implicit pass-through */ Libname mytd teradata user= pw= ; Proc sql noprint ; select * from mytd.mydata ; quit ; /* Explicit pass-through */ Proc sql noprint ; connect to teradata as mytd (user= pw=); select * from connection to mytd (select * from mydata) ; disconnect from mytd ; quit ; NO coding changes! Code Teradata! 12 2

3 Node Node Node Node Amps Amps Amps Amps Node Node Node Node Amps Amps Amps Amps Node Node Node Node Node Node Node Node Amps Amps Amps Amps Amps Amps Amps Amps 10/16/2011 SAS Access: SAS-TD Common Functions ABS ACOS ARCOSH (ACOSH) ARSINH (ASINH) ASIN ATAN ATAN2 AVG COALESCE COS COSH COUNT DAY DTEXTDAY DTEXTMONTH DTEXTYEAR EXP HOUR INDEX (POSITION) LENGTH (CHARACTER_LENGTH) LOG LOG10 LOWCASE (LCASE) MAX MIN MINUTE MOD MONTH SECOND SIN SINH SQRT STD (STDDEV_SAMP) STRIP (TRIM) SUBSTR SUM TAN TANH TIMEPART TRIM UPCASE VAR (VAR_SAMP) YEAR 13» Pass-through 14 SAS In-Database: Pushdown Portions of selected Base SAS Procedures are translated to and pushed down to the DB for processing SAS In-Database: Pushdown Portions of selected Base SAS Procedures are translated to and pushed down to the DB for processing SUMMARY/MEANS FREQ RANK REPORT TABULATE SORT APPEND new for Teradata in 9.3 COPY new for Teradata in 9.3 SUMMARY/MEANS FREQ Exploratory Data Analysis RANK REPORT Reporting TABULATE SORT APPEND new for Teradata in 9.3 Data Preparation COPY new for Teradata in 9.3 Uses Implicit Pass-Through Transparent to the users 15 Uses Implicit Pass-Through Transparent to the users NO coding changes! 16 SAS In-Database: Pushdown Pushdown of Proc Freq options GENERATION= MSGLEVEL=I sastrace=',,,d' sastraceloc=saslog nostsuffix; Traditional Technique SAS Session Proc Freq; table state*credit; Pushdown Libname mytd teradata user= pw= ; Proc freq data= mytd.mydata tables state * credit ; Run ; Request all rows Select state, credit from credit data; Calculate frequency count SAS/Access to Teradata Select Teradata Select Select count(*), state, credit from... group by state, credit; Return only count NO coding changes! Rows Returned Traditional 9,000,000 Pushdown Time to Process 55 seconds 2 seconds Teradata and SAS Confidential

4 Example: Proc Rank code SAS Codewritten by the programmer: proc rank data=indb.myindb_order_item out=work.order descending ties=low; var quantity product_id; ranks QuantityRank ProductRank; run; Teradata generated by the enhanced proc and sent to the database: WITH "subquery0" ( "COSTPRICE_PER_UNIT", "DISCOUNT", "ORDER_ID", "ORDER_ITEM_NUM", "PRODUCT_ID", "QUANTITY", "TOTAL_RETAIL_PRICE" ) AS ( SELECT "COSTPRICE_PER_UNIT", "DISCOUNT", "ORDER_ID", "ORDER_ITEM_NUM", "PRODUCT_ID", "QUANTITY", "TOTAL_RETAIL_PRICE" FROM "DB2_ORDER_ITEM" ) SELECT "table0"."order_id", "table0"."order_item_num", "table0"."product_id", "table0"."quantity", "table0"."total_retail_price", "table0"."costprice_per_unit", "table0"."discount", "table2"."rankalias1" AS "QUANTITYRANK", "table1"."rankalias0" AS "PRODUCTRANK" FROM "subquery0" AS "table0" LEFT JOIN ( SELECT DISTINCT PRODUCT_ID", "tempcol0" AS "rankalias0" FROM ( SELECT "PRODUCT_ID", MIN( "tempcol1" ) OVER (PARTITION BY "PRODUCT_ID" ) AS "tempcol0" FROM ( SELECT "PRODUCT_ID", CAST( ROW_NUMBER() OVER (ORDER BY "PRODUCT_ID" DESC ) AS DOUBLE PRECISION ) AS "tempcol1" FROM "subquery0" WHERE ( ("PRODUCT_ID" IS NOT NULL ) ) ) AS "subquery2" ) AS "subquery1" ) AS "table1" ON ( ("table0"."product_id" = "table1"."product_id" ) ) LEFT JOIN ( SELECT DISTINCT "QUANTITY", "tempcol2" AS "rankalias1" FROM ( SELECT "QUANTITY", MIN( "tempcol3" ) OVER ( PARTITION BY "QUANTITY" ) AS "tempcol2" FROM ( SELECT "QUANTITY", CAST( ROW_NUMBER() OVER ( ORDER BY "QUANTITY" DESC ) AS DOUBLE PRECISION ) AS "tempcol3" FROM "subquery0" WHERE ( ( "QUANTITY" IS NOT NULL ) ) ) AS "subquery4" ) AS "subquery3" ) AS "table2" ON ( ( "table0"."quantity" = "table2"."quantity" ) ) PROC FREQ In-Database Computation PROC FREQ performs in-database computation by using implicit pass-through. The procedure generates queries that are based on the tables you request in the TABLES statement. The database executes these queries to construct initial summary tables, which are then transmitted to PROC FREQ. The procedure uses this summary information to perform the remaining analyses and tasks in the usual way (out of the database). So instead of transferring the entire data set over the network between the database and SAS software, the in-database method transfers only the summary tables. This can substantially reduce processing time when the dimensions of the summary tables (in terms of rows and columns) are much smaller than the dimensions of the entire database table (in terms of individual observations). Additionally, in-database summarization uses efficient parallel processing, which can also provide performance advantages. In-database computation is controlled by the GENERATION option, which you can specify in either a LIBNAME statement or an OPTIONS statement. By default, PROC FREQ uses indatabase computation when possible. There are no FREQ procedure options that control indatabase computation. PROC FREQ uses formatted values to group observations into the levels of frequency and cross-tabulation tables. If formats are available, then in-database summarization uses the formats. If formats are not available, then in-database summarization is based on the raw data values, and PROC FREQ performs the final, formatted classification (out of the database) » Pass-through Publishing SAS Formats into the Database SAS Server Proc SAS Format Library SAS Client sas_put() Proc FOO ACCESS Engine Format Definitions Key Points %indtd_publish_formats publishes Proc FORMAT style formats to the SAS Format Library implements intrinsic formats SAS_PUT() binds formats to processor Greater pass-through. The pipe narrows! SAS publishing agent Proc FORMAT Publishing Macro Format Catalog SAS/ACCESS Interface: Publishing Formats SAS/ACCESS Interface: Publishing Formats Publish SAS Formats in Your Teradata Server Jeff Bailey and Pat Bostic, SAS Institute Inc., Cary, NC

5 SAS Format Function - DB2» Pass-through Company Confidential - For Internal Use Only Company Confidential - For Internal Use Only SAS Analytics Accelerator for Teradata Current In-Database Capabilities Custom SAS functions installed on the database to calculate SSCP for use by: PROC PRINCOMP PROC FACTOR PROC VARCLUS PROC CANCORR PROC REG Custom generation for: PROC SCORE Periodic Data Aggregation (SAS/ETS) PROC TIMESERIES Base SAS Functions: PROC CORR Data Mining: DMDB (EM 6.2) DMREG (EM logistic and linear regression modeling) Stratified sampling (EM 6.2) 27 Base SAS & SAS Access Engine: Pass-Through» Implicit / Explicit Base SAS Procedures: SUMMARY/MEANS, FREQ, RANK, TABULATE, REPORT, SORT SAS Formats SAS Analytics Accelerator REG, VARCLUS, SCORE, CORR, CANCORR, FACTOR (Stat) TIMESERIES (ETS) DMINE, DMDB, DMREG (Data Mining) SAS Scoring Accelerator Allows deployment of Enterprise Miner models to the database for scoring 28 Company Confidential - For Internal Use Only SAS In-Database Direction SAS In-Database Direction Short-term goal To streamline and optimize the customers business process Data Preparation Data Exploration Analytics Long-term goal A database will be the next HOST in which SAS can be deployed. Allowing SAS to leverage the high performance compute architecture and database features seamlessly. Reporting As SAS continues on this journey, the next frontier is exploiting the massively parallel capabilities of the database. This will facilitate manipulating and loading large quantities of data while providing complex analytical algorithms that can be encapsulated as loadable extensions and run alongside the database. Taking advantage of the parallel capabilities of the database and moving the analytics to the data will unleash the power of the mathematics on massive amounts of data. It allows us to provide a high-end, enterprise class platform that combines in-memory analytics with a data platform that supports hardware failover and data replication, terabytes of storage, querying capabilities, ETL, etc. all of which are important to IT and to our customers Radhika Kulkarni, Vice President, Advanced Analytics 30 5

6 To Learn More» Pass-through On-line help within SAS or Enterprise Guide SAS Documentation Search 16,209 SAS papers from SAS Global Forum, SUGI, PharmaSUG, NESUG, SESUG, PhUSE, WUSS, MWSUG, PNWSUG and SCSU THANK YOU for being a valuable SAS customer Paul.Grant@SAS.com Copyright 2011 SAS Institute Inc. All rights reserved. 6