SAS 9 Boosts Performance with Parallel Capabilities of SAS/CONNECT

Transcription

1 SAS 9 Boosts Performance with Parallel Capabilities of SAS/CONNECT Cheryl Doninger R&D Director SAS Institute SAS Forum International Copenhagen 2004

2

3 SAS/CONNECT Multi-Process (MP) CONNECT multiple SAS sessions execute in parallel independent parallelism pipeline parallelism control and synchronization from parent or client SAS session OS controls scheduling of SAS processes to CPU s

4 Independent Parallelism Data Source A Proc Sort Data Source B Proc Sort 0 elapsed time

5 SAS/CONNECT Independent - Scale Up Extract Oracle Data PROC STEP Read and DATA STEP Summarize SAS Data Read and Summarize SAS Data Execute Simultaneously SMP Server

6 SAS/CONNECT Independent - Scale Out SAS Session 2 Parent SAS Session SAS Session n

7 Serial SAS Job libname annual path to annual sales lib ; proc sort data=annual.sales out=annual.ssales; by region; run; libname month path to monthly sales lib ; data month.march; do i = 1 to 52; /* create data set */ output; end; run; /* continue execution */

8 Modified to use SAS/CONNECT options sascmd=!sascmd autosignon=yes; rsubmit task1 wait=no; libname annual path to annual sales lib ; proc sort data=annual.sales out=annual.ssales; by region; run; endrsubmit; rsubmit task2 wait=no; libname month path to monthly sales lib ; data month.march; do i = 1 to 52; /* create data set */ output; end; run; endrsubmit;

9 Pipeline Parallelism Data Step Proc Sort Data Step Proc Sort 0 elapsed time

10 Piping Engine new feature of SAS/CONNECT 9 facilitates pipeline parallelism in both SMP environments and across remote machines overlaps dependent SAS procedure and/or data step execution pipe the output of first data step/proc as input to next data step proc reduces disk space requirements eliminates intermediate write to disk

11 SAS/CONNECT Piping Scale Up DATA STEP Read and Summarize PROC STEP DATA STEP Overlapped Execution SAS Data SMP Server

12 SAS/CONNECT Piping Scale Out SAS Session 2 Parent SAS Session SAS Session n

13 Sequential SAS Job libname outlib "c:\writetodisk"; libname inlib "c:\writetodisk"; /* redundant */ data outlib.intermediate; do i=1 to 5; put 'Writing row ' i; output; end; run; data outlib.final; set inlib.intermediate; do j=1 to 5;

14 Modified to Use Piping signon p1 sascmd= sas'; rsubmit p1 wait=no; libname outlib sasesock :pipe1"; data outlib.intermediate; do i=1 to 5; put 'Writing row ' i; output; end; run; endrsubmit; signon p2 sascmd= sas'; rsubmit p2 wait=no; libname inlib sasesock :pipe1"; libname outlib "d:\temp"; data outlib.final; set inlib.intermediate; do j=1 to 5; put 'Adding data ' j; n2 = j*2; output; end; run; endrsubmit; waitfor _all_ p1 p2;

15 When to Use SAS/CONNECT long running jobs independent data sources independent tasks tasks that can be overlapped utilize SMP hardware or processors on a network

16 Considerations for SAS/CONNECT I/O bottlenecks WORK library CPU bottlenecks

17 Money Laundering Disguising the existence, amount, source or ownership of funds and other assets in an attempt to avoid either detection of illegal activity or to hide evidence of illegal activity

18 The 3 Steps of ML Placement: placing, through deposits or other means, unlawful proceeds into the financial system. Layering: separating proceeds of criminal activity from their origin through the use of layers of complex financial transactions. Integration: using additional transactions to create the appearance of legality through the purchase of assets.

19 AML Functional Solution Overview Discovery

20 Component Architecture Data Management Alert Engine Investigation Data Store Watch Lists Profiles SAR New scenarios Administration Discovery Feedback st Qtr 2nd Qtr 3rd Qtr 4th Qtr

21 Data Management Topology

22 Extract and Transform AML ETL Process Load Dimension Staging Load Dimensions Flat File Dimension Data base Dimension Data Sets Fact Staging Load Facts Flat File Aggregates Fact Data Sets AML processing Data to be Loaded Load Bridges Fact Data base Flat File Bridge Data base Bridge Data Sets

23 ET Process with SAS/CONNECT Dimension Staging Account Account Flat File Dimension Party Household Party House hold Fact Staging Credit Debit Credit Debit Flat File Fact Inquiry Inquiry Aggregates Acct-Party Acct- Party Flat File Bridge Acct-Party Acct- Party Party-House Party- House

24 Load Process with SAS/CONNECT Load Dimensions Account Party Dimension Data base Credit Household AML processing Data to be Loaded Load Facts Debit Inquiry Acct-Party Fact Data base Load Bridges Acct-Party Bridge Party-House Data base

25 Benefits of Using SAS/CONNECT Cut daily load time in half Faster data availability for analysis Ability to recover from errors No nighttime for international organizations

26 Fennia Insurance - Business Problem Conflict with data updates and data backups "No flexibility with backups" Must get time of updates faster Regular job 1,5 h data-step+sort 93 mil. rows Windows2000Server, 2 CPU, 1Gb Memory

27 Start With a Simple Case

28 Comparison (times in seconds) No pipe Data step Real time 3:09 CPU time 2:21 Pipe (Real time 7:05) CPU time 3:04 Proc sort Real time 1:29:04 CPU time 8:43 Real time 25:09 CPU time 10:54 Total Real time 1:32:12 CPU time 11:04 Real time 25:09 CPU time 13:58

29 Results "Dead easy" Huge saving in real time (73 %): 1,5 h 25 min. CPU: consume 27 % more Not affecting memory usage Used same method in web-queries Used in more complicated processes

30 More Complex Case

31 SAS/CONNECT All About Performance Many SAS solutions support dimensional model Anti-Money Laundering (AML) Process Intelligence (PI) Customer Relationship Management (CRM) Supplier Relationship Management (SRM) others Any SAS application that: Takes a long time to run Processes a lot of data Has tasks that can be run in parallel or overlapped

32 For More Info Scalability and Performance Community AML website Jouni Javanainen Development Manager, Information Services, Fennia Insurance, Finland

33 Copyright 2003, 2004, SAS Institute Inc. All rights reserved. 33