Qual I.T. Services Pty Ltd

Transcription

1

2 SAS/CONNECT Qual I.T. Services Pty Ltd

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19 proc qevent log= Security computer= zolmall out=work.loginfo; proc print data=work.loginfo; var eventid timegen account; where eventid in (528,538); run; Qual I.T. Services Pty Ltd

20

21

22

23 proc qwdisk root= u:\users out=work.usage; proc print data=work.usage; run; Qual I.T. Services Pty Ltd

24

25

26

27

28

29

30 Costing Warehouse Resources: A solution for Windows NT Server

31 Abstract This paper describes an automated data warehouse billing solution in a Microsoft Windows NT Server environment. Developed using BASE SAS and SAS/TOOLKIT software, this solution consists of scheduled batch programs which automatically retrieve, summarise, report and inform relevant departments of warehouse usage. Warehouse usage data is retrieved from the NT eventlog and file system. This data is summarised on a user and department basis. A standard billing report is generated detailing user identification, department, connection time, disk usage and monies charged. This report is then automatically ed to the relevant departments. In addition, this paper addresses the issues involved in assessing the client s needs and the development of an optimal solution. Introduction Accounting for computer resource usage has traditionally been the domain of the mainframe environment and more recently in the unix mid range environment. As the Windows NT Server gains wider corporate acceptance for the deployment of small to medium data warehouses, financial controllers will demand more equitable allocation of computer resource costs. Accounting for computer resource usage includes the allocation of costs for hardware, software, telecommunications & consumables. Where multiple clients exist, costs must be shared suitably, and also reported so that the clients can confirm the charged amounts. The allocation of costs based on usage relies on robust, accurate system monitors and data collectors. Fortunately performance monitoring and event logging has been built into Windows NT Server since its inception. This information is stored in operating system specific files that need to be extracted and imported into the SAS System before reports can be produced. This is a process that can be quite difficult in the Windows NT environment. This paper examines what information was considered suitable for the allocation of computer resource usage costs, how that information was retrieved into the SAS System and finally, transformed into billing reports for electronic dispatch to user departments. Requirements The basic requirements were to find a simple solution based on or around the SAS System to address; the equitable allocation of warehouse usage costs to departmental users, and automate the billing process in a Windows NT Server environment. Traditional approaches to resource billing are normally centered around Processor time Connect time Disk usage Processor Time The use of processor time as a measure of warehouse resource usage was discarded due to the apparent complexity and overheads involved in retrieving processor time per user per application in a Windows NT Server environment. Connect Time User connect time was considered a simple and reasonably equitable indication of warehouse resource usage. It is possible to determine connection duration by reviewing the login and logout times that are automatically recorded for a Windows NT Server. This can be viewed on any Windows NT machine using the Administrative Tool application called the Event Viewer. Provided the Login and Logout of users is being recorded by the system, it is easy to review the amount of time spent by any particular user, or group of users. Alternatively, it is possible to count the number of times that the user logged in, without taking into account the amount of time spent accessing the system. Problems with this approach include instances where users inadvertently remain connected to the warehouse or perhaps where a particular user consumed large quantities of warehouse machine resources to the extent that other users were forced to remain connected longer than would otherwise be required. Both these problems can be addressed through proactive administration settings of the Windows NT Server environment. Disk usage The amount of physical disk space used on the server was considered another appropriate chargeable resource.

32 For example, if a user / department consistently absorbs half of the available disk space, you will want to charge them accordingly in order to recoup the disk drive costs in an equitable manner. In most cases individual users will have their own home directories that can be used to monitor disk usage. However you must allow for the existence of subdirectories. Both connect time and disk usage can be read into the SAS System in order to produce resource usage statistics and appropriate billing reports. The second requirement was to automate the billing process, including the sending of the bill! The SAS System includes a link into most popular Windows mailing software that enables the reports to be generated and then automatically ed to the appropriate person. Technical Details Event logging The NT Eventlog is a service that starts whenever you boot Windows NT Server. The NT event log is based on files with the extension evt in the %systemroot%\system32\config directory. By default, three logs exist called the security log, the application log and the system log. Logon and logoff events are recorded in the security log. Logon and logoff event logging must be enabled through the use of the Windows NT Server User Manager. This administration program also allows the configuration of other audit policies. Event log settings such as file size and wrapping must be specified using the log settings functionality provided in the Windows NT Server Event Viewer. Unfortunately you cannot access event log files directly because other applications, including Windows NT Server might want to log or read events. Instead you must access the event log through the event log Application Programmer Interface (API) which forms part of the Win32 API. Disk usage information File information is recorded differently depending on the file system installed. Windows NT provides two such systems: File Allocation Table (FAT) and New Technology File System (NTFS). The FAT file system is reliant on the home directory as the source of the file owner (user). An accurate calculation of the disk space held by each user can be achieved by scanning recursively from each home directory. The newer NTFS file system provides built in security including the ability to assign permissions at the individual file level. The home directory method can be applied when the NTFS file system is used. However, a better option is to utilise the extra file attributes (such as file owner) which are recorded under the NTFS file system. In this way accurate disk usage information is available for entire disk volumes. The retrieval of file size information may be accomplished in several different ways. An advantage of using file functions in the Win32 API is that extra NTFS file attributes (such as file owner) can also be easily retrieved. The Windows NT API The Windows NT operating system includes a set of APIs. These APIs are well documented and can be used to access the functionality and resource set of the Windows NT operating system. The event logging and file API functions are provided within the Win32 API. These API functions can be called through many software languages, such as C, Fortran and Pascal. The SAS System also has the ability to call these API functions, however other languages such as C are better suited to the task. A successful combination of C and the SAS System can be achieved with SAS/TOOLKIT software. By using SAS/TOOLKIT software, a modular program can be written in C and then called from the SAS System as you would with any SAS System function or procedure. Problems with accessing the Windows NT API from the SAS System As mentioned, the SAS System has the ability to call Windows NT API functions. Release 6.11 of the SAS System provided the functionality to access routines which reside in external Dynamic Link Libraries (DLL). A text file describes the DLL routine that the SAS System is going to access. The module family of SAS System call routines allows DLL routines to be accessed from the DATA step, the IML procedure and SCL code. This is useful, however there are some limitations.

33 Problems include: The SAS System has insufficient buffer sizes to handle the returned data from some APIs. For example, in the ReadEventLog API for Windows NT, the returned data may be any size, where the SAS System can only accept a 200 character buffer. These overflows may corrupt memory. The SAS System does not currently support UNICODE character sets. Some of the newer Windows NT API functions only have Unicode versions (2 bytes per character), such as NetUserGetInfo. The overhead for the SAS System may become significant with recursive calls to APIs. For example, the recursive disk space reading routine is slow even when written entirely in C. This, combined with the additional overhead of calling the APIs through the SAS System would significantly degrade the execution time. C and accessing the API C (or C++) is the language of choice for many application developers when accessing internal Windows NT system information. This is because Windows NT itself is written largely in C. Therefore, a C program can easily call Windows NT API functions and more importantly handle data returned from API function calls with no compatibility issues. The problem with C is that it is not a flexible reporting tool. SAS/TOOLKIT software SAS/TOOLKIT software can overcome the problems with accessing the Windows NT API from the SAS System. It enables C routines to be created and integrated with the SAS System as BASE SAS code ie. functions, procedures and formats. Furthermore, these routines can be used to harness the reporting powers of the SAS System. Final Solution QUILT NT Pack V1.0 was used as the final solution. It consists of several data collection procedures and functions written using C and SAS/TOOLKIT software. This Windows NT Server data collection software enables the retrieval of connect time and disk usage system information into SAS data sets. An example procedure is named QEVENT, which reads data from the Windows NT event log directly into a SAS data set: proc qevent log= Security computer='qualserv out=work.loginfo(keep=eventid timegen account where=(eventid in (528,538))); run; BASE SAS code can be applied to the resulting SAS data set to produce quick and effortless reports for billing purposes. Another procedure QWDISK measures disk usage. proc qwdisk; root= u:\users out=work.usage; run; Final report distribution was accomplished by . Additional functions determine user addresses from the Windows NT Server user account database. Reports generated were automatically ed to the appropriate user / department. The SAS System supports two types of electronic mail interfaces: Mail Application Programmer Interface (MAPI) and Vendor Independent Mail (VIM). These protocols provide a route to most popular mailing software. In addition, the completed BASE SAS program may be scheduled to run in batch using the native Windows NT AT scheduling software or a 3 rd party program scheduling package. The Results Following is a sample report easily produced from the implemented system: Conn Disk Time Space Dept Person HH:MM MB Dev Gyzen, R 18: Kempe, R 7: Matthews, M 42: Rastogi, A 0: ==== ===== ===== Dev 67: The Future The QEVENT procedure will continue to access information if events are stored in the existing Windows NT Event Log. Future versions of QUILT NT Pack could include other measurement units such as processor time and memory usage. Future incarnations to the Windows NT operating system may require an update to

34 QUILT NT Pack. For example, changing the event log API to a Component Object Model (COM) object would require a revision of the QEVENT procedure. SAS/TOOLKIT software should provide the flexibility to address such changes. Conclusion User connect time and disk usage provide a simple and equitable indication of warehouse resource usage in a Windows NT Server environment. SAS/TOOLKIT software can be used by a C developer to provide custom SAS procedures and functions which retrieve the appropriate Windows NT Server warehouse resource information. Once the system resource data is extracted into SAS data set form, the SAS System is an ideal tool for reporting on the warehouse usage. Usage and Referenc, Version 6, First Edition, Cary, NC: SAS Institute Inc., Richard Simon, Windows NT Win32 API SuperBible: Waite Group, Microsoft Corporation, MSDN (Microsoft Developer Network). Author Information Michael Matthews Qual I.T. Services Pty Ltd GPO Box 7021 Sydney NSW 2001 michael_matthews@qual-it.com.au Acknowledgements Tim Churches of the NSW Health Department and Rudy Gyzen of Qual I.T. Services contributed to the preparation of this paper. Trademarks SAS and SAS/TOOLKIT software are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Microsoft, Windows, Windows NT and MSDN are registered trademarks of Microsoft Corporation. References Nell Cote, F. Burris Jackes, Hudson Williams, Baseline Accounting and Usage-based Chargeback on Unix Systems, New York, NY: Hudson Williams, 1998 Christy L. Garrison, A Bundled, End-to-End Solution for Performance Analysis, Cary, NC: SAS Institute Inc. Gary Mehler, Taking Advantage of the SAS System on Windows NT, Cary, NC: SAS Institute Inc., Microsoft Corporation, Microsoft Windows NT The Foundation, Redmond, WA: Microsoft Corporation, 1997 SAS Institute Inc., SAS Companion for the Microsoft Windows Environment, Second Edition, Cary, NC: SAS Institute Inc., pp. SAS Institute Inc., SAS/TOOLKIT Software: