Authors: Haidong Tang (Don) Xiao Ji (Samuel) Presenter: Haidong Tang (Don) June 2002

Transcription

1 Tracking your data warehouse using SAS Authors: Haidong Tang (Don) Xiao Ji (Samuel) Presenter: Haidong Tang (Don) June 2002

2 Agenda! Introduction of Shanghai Baosight Software Co., Ltd.! Why track your data warehouse! What to track! How to track using SAS! A tracking system using SAS! Conclusions

3 Introduction of Shanghai Baosight Software Co., Ltd.

4 Facts of the company Facts Date of establishment: Registered capital: Staff member: Character: Location: Business: April 2000 RMB 260 million 835 (39 PhDs, 136 masters) a listed company in Shanghai Zhangjiang Hi-tech Park, Shanghai Sales income over RMB320 million in 2001

5 Evolution of the company Shanghai Baosight Software Co., Ltd. (2001) Shanghai Baosight Information Technology Co. (2000) Shanghai Baosteel Software Co., Ltd. (1996) Shanghai Baoni Computer Integration Technology Co., Ltd. (1994) Shanghai Baosteel Computer System Engineering Co., Ltd.(1997) Automation Department, Shanghai Baoshan Iron & Steel Co., Ltd. (1979)

6 Relationship with Shanghai Baosteel Shanghai Baosteel Group Corporation Shanghai Baoshan iron & steel Co., Ltd. (Shanghai Baosteel) Shanghai Baosight Software Co., Ltd. (Shanghai Baosight) Other 30 companies

7 Qualified certificates Holdings of qualified certificates ISO 9001 Quality Assurance System Computer Integrator (the only company in Shanghai) High-tech enterprise Science and technology operation Information services Software import and export allowance Certificates applying CMM Level 3 (In progress) Assessing consultants from PIA (

8 Partnership, alliance & cooperation

9 Data Strategies Dept. Data Strategies Department business intelligence technology data warehouse consulting service data mining arithmetic Expertise data warehouse design and implementation data warehouse modeling technique

10 Data Strategies Dept. Successful cases in Shanghai Baosteel Technical quality data mart Production management data mart Production control information management system Contract placing and delivery cycle analysis system Bar & rod products quality statistics and analysis system Production ability and resources analytical system Enterprise comprehensive statistics system Quasi-on-line analysis system for balanced operation of weekly production plan Quality inspection statistics analysis system for hot and cold rolling IPC system for CAL unit of 1550 cold rolling mill

11 Why track your data warehouse

12 Data warehouse has been deployed, but how can I know whether ETL programs ran properly? how the disk space is expanding? the application servers running steadily? any error messages in Web server log? invalid user intruded? other concerned system-wide statistics available? can all information be organized together? can I get information by ? or can I get information easily from a Web?...?

13 Typical ways to solve the problems by scanning error messages in ETL log files carefully every day typing system command to check disk space, server status,... introducing Web logs analytical tools checking system information by system tools etc.

14 What we really need is A data warehouse tracking system to organize all information together, and integrate with and Web server, is an important part of the data warehouse system

15 Responsibilities of a data warehouse tracking system finding out whether ETL programs ran properly showing ETL statistics, such as speed, time consumed, etc telling status of application servers tracking DW disk volume expanding checking status of operational systems auditing user access and authentication providing integrated tracking reports by extracting others concerned statistics

16 What to track

17 Should at first decide the tracking points, such as: " data storage " ETL log files " server status " server logs " user access " many others

18 Deciding tracking points Data warehouse system Web Server Data Source (DB2,Oracle) ETL Operational Data Store Subject Data DW reports Mail Server IntrNet Server Share Server Telnet Ftp Clients Source system status ETL logs ETL statistics Data storage Subject logs Report logs Server status Server logs User access

19 What we really need Tracking reports Data Warehouse Tracking System Source System status ETL logs ETL statistics Data storage Report logs Server Status Server logs User access

20 How to track using SAS

21 Read and analyze the log files Execute system commands Present brief tracking report by Present tracking report by Web server Schedule the tracking program

22 Read and analyze the log files Execute system commands Present brief tracking report by Present tracking report by Web server Schedule the tracking program

23 Read and analyze log files A variety of log files in data warehouse system, ETL log files subject or reports log files server log files some others

24 Read and analyze log files Two types of log files " SAS log files such as SAS batch ETL programs, SAS batch reports programs " non-sas log files, such as Web log, other server logs

25 Read and analyze log files Example, read a SAS log file filename logs "/home2/a.log"; Pay more attention to data tmp(keep=error);! because of errors retain error 0;! was stopped infile logs lrecl=200 pad end=end; input s $ 200.; if index(s,'because of errors')>0 then error=error+1; if index(s,'was stopped')>0 then error=error+1; if end then output; run;

26 Read and analyze the log files Execute system commands Present brief tracking report by Present tracking report by Web server Schedule the tracking program

27 Execute system commands Approaches to execute system commands, %execute system x filename... pipe others

28 Execute system commands Ways to execute commands depend on OS purpose Example, to list a directory in Unix, x ls lart ; in Windows, options nosync noxwait; x dir>dir.txt ;

29 Execute system commands Get feedbacks from execution for further processing filename... pipe...

30 Execute system commands Example, checking disk volume filename msg pipe df k ; data diskvol; infile msg lrecl=200 pad end=end; input s $ 200.; if index(s,'/dev/sas_prod_lv')>0 or index(s,'/dev/home2lv') then do; filesystem=scan(s,1,''); volume=scan(s,2,''); free=scan(s,3,''); used=scan(s,4,''); output; end; keep filesystem volume free used; run;

31 Read and analyze the log files Execute system commands Present brief tracking report by Present tracking report by Web server Schedule the tracking program

32 Present brief tracking report by Advantages of delivering an tracking report flexible reliable easy to keep the reports personally can be deliver to pager or mobile if necessary

33 Present brief tracking report by The SAS System supported interfaces: " MAPI (such as Microsoft Exchange) " Vendor Independent Mail (VIM--such as Lotus, cc:mail) " SMTP

34 Present brief tracking report by Two approaches to send SMTP using SAS, " filename... socket complicated but direct and efficient " filename... easy but should configure system options first

35 Present brief tracking report by filename... socket example, send a text file filename outlog /home2/baodw/dwtracker.dat filename smtp socket ' :25'; filename log '/home2/baodw/ .log'; data _null_; infile smtp; input; file log; put 'INIT>' _infile_; file smtp; put 'HELO noic.bstl'; input; file log; put 'HELO>' _infile_; file smtp; put 'MAIL FROM: don<don@noic.bstl>'; input; file log; put 'FROM>' _infile_; file smtp; put 'RCPT TO: <don@baosteel.net>'; input; file log; put 'RCPT>' _infile_; file smtp; put 'DATA'; input; file log; put 'DATA>' _infile_; d='(' put(date(),date9.) ')'; file smtp; put 'SUBJECT: Data Warehouse Tracking Report ' d; put; do until(end); infile outlog end=end; input; put _infile_; end; put '.'; input; put 'QUIT'; input; run;

36 Present brief tracking report by filename... example, send a data set filename smtp 'don@baosteel.net' subject="tracking report"; data _null_; Should configure first file smtp;! HOST set dwtracker;! PORT put error; run;

37 Present brief tracking report by The result

38 Read and analyze the log files Execute system commands Present brief tracking report by Present tracking report by Web server Schedule the tracking program

39 Present tracking report by Web server Advantages of presenting tracking report by Web server easy to access easy to keep all reports centrally only one copy is necessary easy to introduce graph and many other Web features

40 Present tracking report by Web server Example, generate html report and backup log files %let yyyymmdd=%sysfunc(compress(%sysfunc(putn(%sysfunc(date()),yymmdd10.)),-)); %let pathbk=/home2/web; /* backup log files to Web directory */ %let cmd=cp -p job1.log &pathbk&yyyymmdd-job1.log%str(.txt); %put &cmd; %sysexec &cmd; /* generate html formatted tracking */ /* report to Web server */ filename_webout "&pathbk&yyyymmdd-dwtracker.htm"; data null_; file webout; datetime=symget('datetime'); put "<center>"; put '<font face="arial" size=5 color=red><strong>data WAREHOUSE TRACKING REPORT</strong></font>'; put '<table><tr><td><font face= "Arial" size=3 color=blue> <strong>time: ' datetime '</strong></font></td></tr></table>'; put "</center>"; run;

41 Present tracking report by Web server Example (cont.) ods listing close; filename _webout "&_pathbk&yyyymmdd-dwtracker.htm" mod; ods html file=_webout(no_top_matter no_bottom_matter); title '<center>server Status </center>'; proc print data=tmp_svr1 label; var svrname1 status; run; title '<center>log File</center>'; proc print data=tmp_log1 label; var logname1 flen dt error; run; title '<center>disk Volume </center>'; proc print data=diskvol label; var filesystem volume free used path; run; ods html close; ods listing;

42 Present tracking report by Web server The result

43 Read and analyze the log files Execute system commands Present brief tracking report by Present tracking report by Web server Schedule the tracking program

44 Schedule the tracking program Unix system at crontab Windows system windows job scheduler tools

45 Schedule the tracking program Example, Unix system crontab -l #dwtracker.sas * * * /sasv8/sas -autoexec /sasv8/auto.sas -config /sasv8/config.batch -sysin /home2/dwtracker/dwtracker.sas -dmsbatch -log /home2/dwtracker/dwtracker.log

46 A tracking system using SAS

47 Shanghai Baosteel enterprise data warehouse project starting from June 1999; depending on SAS total solutions; Web-based; on IBM S85 machine; >500G data now; using SAS/IntrNet, SAS/Share, Web servers...; fully benefited from the accomplished data marts

48 The tracking system for Baosteel DW tracks server status; ETL log files; large data sets; disk expanding; Web log files; others

49 Shanghai Baosteel DW tracking system provides integrated information; or Web access; feasibility for faraway DW system monitoring; easiness to add new tracking features; high efficiency high productivity high reliability

50 Conclusions

51 A tracking system is not just a tracking system, it is an important part of your data warehouse it is actually an information mart it provides functions of monitoring it is productive it is an efficient way to administrate the system

52 SAS provides strong & flexible functions SAS is a qualified tool to track your data warehouse SAS can be used to analyze what you track as well

53 Thanks

54 Contact information Haidong Tang (Don) Xiao Ji (Samuel)

55 Questions?