Scalable Performance Data Server (SPDS) Steve Morton UK Technology Manager SAS Institute

Size: px

Start display at page:

Download "Scalable Performance Data Server (SPDS) Steve Morton UK Technology Manager SAS Institute"

Barbra Holland
5 years ago
Views:

1 calable Performance Data erver (PD) teve Morton UK Technology Manager A Institute

2 calable Performance Data erver A new high-performance data server for A applications un MP servers first Others will follow (AIX, NT...) A Clients on any platform with TCP/IP communications

3 Objectives of PD Exploit MP and multi-threading for fast I/O, parallel load + query Exploit bitmap-indexes as well as B-tree for fast where clauses High performance server for multiple concurrent query users Plug-compatible LIBNAME for transparent use in A programs

4 PD Components Multi-threading server with special features for parallel load + query LIBNAME engine for all A clients TCP/IP socket communication, RPCs for data transfer Name server to simplify location of PD servers ODBC server and driver for Windows clients

5 PD Multi-threading erver eparate server execution Fully threaded to exploit MP hardware Own file format Data + index components span multiple Unix file systems (>2Gb) Fast parallel load and index, parallel read and where, parallel sort Batch update Member level locking

6 PD LIBNAME Engine Executes on the client node Communicates with the PD components P P P Domain name server Data server LIBNAME proxy libname libref sasspds libspec server=node.port [more_options];

7 PD Domain Name erver Knows the PD location in a network Resolves logical libspecs into TCP/IP addresses and directory paths Defined in /etc/services file erves multiple PDs Hides physical details from endusers, aids administration

8 PD Client-erver P D e r v e r 1 P D e r v e r 2 A u s e r 3 A u s e r 1 P D D o m a i n N a m e e r v e r A u s e r 2

9 PD erver Registers with the PD name server at start-up time Performs login verification with the first LIBNAME statement issued by each client Forks LIBNAME proxy for each client PD administrator maintains user and LIBNAME configuration files

10 Details of the erver erver Client P D e r v e r T a s k L I B N A M E P r o x y 1 A U s e r 1 L I B N A M E P r o x y 2 A U s e r 2

11 PD LIBNAME Proxy Multi-threaded process, forked from PD on behalf of every A client Independent processing Creates, manipulates, and accesses all user-defined LIBNAME data objects Executes RPCs made from the LIBNAME engine Large (controllable) packet size for fast data transfer

12 PD ODBC erver ODBC client drivers link by TCP/IP to ODBC server ODBC server accepts passthru QL ODBC server communicates with PD server directly, performs joins etc. on server machine ODBC server returns result to client using large (controllable) packet size

13 ODBC client-server erver Client P D e r v e r T a s k D a t a e r v e r P r o x y O D B C e r v e r O D B C C l i e n t

14 PD Performance Data set P P P obs., 4KB each 10 indexes 85 variables PD on UN MP hardware A client on Windows95 PD more than 4 times faster than RL

15 PD Performance Action RL PD equential Read whole dataset 23:55 5:33 Read with WHERE sec sec. Read with WHERE and BY variable sec sec.

16 How is this acheived? egmented files, indexes Multi-threaded WHERE clause evaluation B-tree or bitmap indexes (cardinality) Parallel retrieval, sort Large packets for data transfer

17 PD comparison to A/HARE Large datasets mall-to-medium datasets Batch update Interactive update Intensive Query use, no update Limited Query use during update Member-level locking Record-level locking Multi-thread access ingle thread access

18 How does the user work? Write to PD LIBNAME as usual PROC COPY DATA steps. Read from PD LIBNAME as usual... DATA steps PROC steps PROC QL A/EI objects

19 How does the advanced user work? Index tuning PROC DATAET PROC QL ort tuning Heapsort vs. Quicksort Control segmentation Control buffers, packet-size Manage access security

20 PD Access ecurity UNIX file permissions P ACLs P P imple access control High security Flexible access granularity

21 In Conclusion. PD provides high-performance data store for A data tate-of-the-art indexing and parallel processing calable to very large storage Particularly suitable for Data Warehousing applications

22 Acknowledgment Thanks to the Advanced erver Department development team: Ami Ghosh - manager, libname engine, ODBC, name server, RPC Ed Blair - I/O component, libname proxy, RPC Phil Herold - WHERE, source management, utilities Doug edlak - B-tree index, ACLs Phil mith - bitmap index, sorting

Introducing the SAS ODBC Driver

1 CHAPTER 1 Introducing the SAS ODBC Driver Overview: The SAS ODBC Driver 1 What Is ODBC? 2 What Is the SAS ODBC Driver? 2 Types of Data Accessed with the SAS ODBC Driver 3 Understanding SAS 5 SAS Data