Every SAS Cloud has a Silver Lining. Letting your data reign in the cloud

Transcription

1 Every SAS Cloud has a Silver Lining Letting your data reign in the cloud

2 DSS SAS SYSTEM Current Single Virtual Server unit with 16 cores upgraded to 32 cores 256 Gb RAM 150 registered users Data collector from other government departments and external bodies Runs Models and builds reports from this data. Processing time Considerably long due to model running Goal Grow with increased data collection Integrate with new tools Deliver reports in shorter time frame Increased analytical capabilities Able to operate in the new DSS Infrastructure Hosting Zone (on premises cloud)

3 DATA IS JUST DATA ISN T IT? Structured data captured from operational systems Customer interaction directly with company Purchase detail Usage detail Addition of 3 rd party data to add value to operational data The beginning of CRM systems Beginning of Data Mining Big Data Social media Non customer interaction Unstructured data Anything that generates some form of data in large volumes

4 IN 1999, THE WORLD PRODUCED ABOUT 1.5 EXABYTES OF UNIQUE INFORMATION, OR ABOUT 250 MEGABYTES FOR EVERY MAN, WOMAN, AND CHILD ON EARTH.

5 IN 2008 THE WORLD PRODUCED 14.7 EXABYTES OF NEW INFORMATION, NEARLY TRIPLE THE VOLUME OF INFORMATION IN 2003.

6 WHAT WE DO WITH THE DATA 1990 Descriptive Analytics What happened and when did it happen? Standard reports Ad hoc reports Simple drill down queries onwards Predictive Analytics based on what happened, what is likely to happen? Statistical Analysis Forecasting Predictive Modelling Optimisation

7 ADVANCED ANALYTICS What happened? Why did it happen? When did it happen? What caused it to happen? Who made it happen? Will it happen again? What should we do to make happen /stop it from happening again? What is the outcome of this event?

8 WHAT DO WE WANT FROM THE DATA? What do we want to know? Why do we want to know? When do we want to know? Who will use the what we now know? How will this knowledge be used? What is the benefit of knowing this?

9 THE HARDWARE It s more than an abacus, coloured pencils, ruler, and graph paper

10 GETTING THE RIGHT BALANCE For today and tomorrow

11 Sunway TaihuLight (10.65 million cores). Probably Not 'It would take 7.2 billion people using electronic calculators 32 years, or two million desktop computers working together for one minute, to do the same calculation the computer can solve in just 60 seconds.'

12 How about this? Ummm, Yeah, No

13 THINGS TO CONSIDER You ll always get more data More analytics and reporting More, More, More Efficiency Fixed resources Elastic/Scalable resources Coding Scheduling Tools $$$$$$ ROI Lost Time

14 JUST HOW FAST ARE THE $$$$? Fixed resources Scalable/Elastic Resources One User Many Users

15 IN BUSINESS The difference between a great system and a bad system is Being No. 1 Or Being No 1

16 IN GOVERNMENT The difference between a good system and a bad system is Wise use of taxpayers money Or Being asked questions by the Auditor General

17 GRID SERVER VS VIYA Grid Server (on Premises or Cloud) Scalable (with limited CPU available) On or off premises Integrate other tools E.g. Hadoop etc. In house systems need to be maintained / upgraded In house-security by organisation Viya (Cloud) Elastic (Virtually unlimited CPU) Mainly off premises location Integrate other tools E.g. Hadoop etc. Minimal maintenance required (Host) Security by cloud host and organisation

18 Yes, I stole this from SAS

19 BEST OPTION SAS Grid Server configuration in an on premises cloud Resources in house Meets government security protocols (ASD) Internal systems and Infrastructure meet needs now and future Integration with tools currently used by DSS Hadoop Federation Server Easy to expand nodes due to virtual server environment Node configuration to suit business needs Advanced analytical capabilites

20 Data Tier Source and Target SAS Server Tier Client Tier RDBMS Grid Server (Node N+1) Yarn Node Manager Grid Server (Node N) SAS Foundation Yarn Node Manager Grid Server (Node 1) Grid Server (Controller) Yarn Resource Manager OOZIE Environment Administrators SAS Management Console Analytical data Warehouse/ marts SAS Connect SAS Grid Node SAS Foundation Yarn Node Manager SAS Connect SAS Foundation HIVE HUE SAS Environment Manager SAS Object SpawnerSAS Grid Node SAS Connect SAS Workspace Server SAS Object SpawnerSAS Grid Node SAS Stored Process SAS Server Workspace Server SAS Object Spawner HttpFS HDFS Name Node SAS Foundation SAS Connect Hadoop Administration ADFS Access Administration SAS Pooled Workspace SAS Server Stored Process SAS Server Workspace Server SAS Grid Server Hadoop SAS Data Step Batch SAS Pooled Server Workspace SAS Stored ServerProcess Server HDFS Data Node SAS Data Step Batch SAS Pooled ServerWorkspace Server HDFS Data Node SAS Data Step Batch Server SAS Object Spawner SAS Workspace Server SAS Stored Process Server End User SAS Enterprise Guide HDFS Data Node SAS Pooled Workspace Server ERP, applications & other Sources SAS Data Step Batch Server SAS Addin for Microsoft Office SAS Enterprise Miner Shared Storage Clustered Middle Tier Server Clustered Metadata Server SASGSUB HDFS Storage User Home Directories Shared Projects SAS Web Server SAS Metadata Server SAS Data SAS Grid Work Libraries SAS Depot SAS Web Application Servers SAS Studio Mid-Tier SAS SPDE SPDS Data SAS Config Files Hadoop config and client Jars SAS Web Infrastructure Platform Hadoop Client jars and Config Files OK I took this from SAS too, but modified it a little

21 THANK YOU Peter Stagg