Informix Excel Reports Technical Guide

Informix Excel Reports Technical Guide Index Revision History... 1 Introduction... 1 Overview... 2 Scheduled Jobs... 3 HTML Pages... 3 JavaScript... 3 Auto-Refreshing Workbooks... 4 Parameterised Web Queries... 4 General Instructions... 6 Hyperlink Base... 7 CGI-SQL... 8 CGI-CSV... 9 Revision History Version Date Name Description 1.0 19/09/2007 Doug Lawry First generic version Introduction This document gives a technical overview of custom Web Query driver scripts that enable deployment of reports in Excel containing data dynamically fetched from an Informix database server, both in terms of how it works and how it can be used in practice. A thorough knowledge of Informix and UNIX is assumed. Informix Excel Reports Technical Guide OLTP Holdings Page 1 of 9

Overview UNIX scheduler cron jobs Web browser Internet Explorer Web query Excel CSV report program Shell script / 4GL Web server Apache HTML page FrontPage Public FTP server /home/ftp/pub Latest CSV picker cgi-csv.cgi Generic SQL runner cgi-sql.cgi Template and data XLS, CSV Auto-refresh sheet XLS Informix database dbname@server Routine reports particularly those with long run times can be run overnight every week or month by the UNIX scheduler. These Shell scripts, which either have embedded SQL statements or call 4GL programs in more complex cases, create zipped CSV files on a public FTP server. Users would typically access these files and Excel templates into which they can be loaded via Web links, but they can equally be accessed directly from the FTP server. Reports that require parameters from the user can be accessed via Web pages that use the CGI generic SQL SELECT statement runner CGI-SQL to return dynamic results displayed with the Internet Explorer Excel plug-in. Alternatively, Excel applications can be designed to refresh their contents automatically, either directly from the database via CGI-SQL, or using the CGI-CSV script to return the contents of the latest CSV file in a specified folder from the public FTP server. Web Queries embedded in Excel applications that allow parameters to be entered by the user are created using Import Data with IQY query definition files. Informix Excel Reports Technical Guide OLTP Holdings Page 2 of 9

Scheduled Jobs A typical Shell script run overnight by cron to create a routine report to be presented by CGI-CSV does the following: Sets common environment variables Goes to a public FTP sub-folder Runs SQL statements to unload data Retains any SQL error messages Inserts comma separated column headings Converts unload file to CSV using unl2csv (see unl_utils at www.iiug.org) Renames the completed file using the current date A typical file name produced might be: STATS-2007.09.19-.csv.gz Alternatively, some cron jobs might populate data warehouse tables for subsequent access by HTML pages or dynamic auto-refreshing Excel workbooks. HTML Pages Web pages can be written with Microsoft FrontPage - and placed in the Web server s default document directory - using HTML forms to open a parameterised CGI-SQL URL when the form is submitted. It is recommended that views are created to simplify the contents of the hidden SQL field. Any question marks in the SQL will be replaced with form fields Param1, Param2, etc. Input validation can be specified using the FrontPage GUI and is implemented via generated JavaScript. Results are returned by CGI-SQL as tab-delimited text but presented using the Excel browser plug-in (displayed as a worksheet within an Internet Explorer window) simply by setting the correct Excel MIME type in the header. Links to a formatted Excel template workbook might be provided on the initial page, with instructions on how to paste in the new data. JavaScript The following custom JavaScript function can be useful in these HTML pages: // Open URL on the current host's FTP server in new window function ftp(path) { window.open("ftp://" + window.location.host + "/" + path) } This prevents any need for an FTP server URL to be hard-coded in links from the Web page, as it can be derived from the host name of the current HTTP session. Informix Excel Reports Technical Guide OLTP Holdings Page 3 of 9

Auto-Refreshing Workbooks Ideally, Excel report templates should automatically refresh their content when they are opened. This is achieved via the Excel Web Query feature (Data / Import External Data / New Web Query) using either CGI-SQL or CGI-CSV (see next sections). Excel Database Query (ODBC or OLEDB) is comparatively undesirable, as this would require drivers to be installed on users PCs and allow uncontrolled access. Note that some Excel templates cannot be refreshed automatically if this would exceed the practical limit on the volume of data returned by a Web Query, which can be less the maximum number of rows if there are a large number of columns. The procedure to add a Web query with no parameters is as follows: 1. Open a new worksheet. 2. Choose Data / Import External Data / New Web Query from the menu. 3. Type in a URL (see CGI-SQL and CGI-CSV) and click the Go button. 4. Click Options and untick Treat consecutive delimiters as one. 5. Tip: Sometimes you need to tick Disable date recognition. 6. Click Import and then Properties. 7. Untick Adjust column width if desired. 8. Tip: tick Refresh data on file open for dynamic workbooks. 9. Click OK twice. Subsequent changes can be made using the External Data toolbar (enable with View / Toolbars). See also General Instructions. Parameterised Web Queries Some Excel workbooks will need to contain embedded CGI-SQL queries specified with the Data / Import External Data / Import Data menu option, with selection criteria taken from enterable cells within the workbook. CGI-SQL only accepts single SELECT statements, but procedures can be used to return a data set using syntax such as: SELECT * FROM TABLE (FUNCTION some_procedure(?,?)) AS x1 (return1, return2) Whatever form of SELECT statement is used, question marks should be used in the select statement to indicate where variable values should be substituted using CGI parameters Param1, Param2, etc. Before adding a parameterised Web Query to a worksheet, you must first create an Excel Web Query File (suffix IQY). This is a plain text file, so you can use Notepad, Wordpad, Gvim, etc. Some examples follow: WEB 1 http://server-name/cgi-bin/cgi-sql.cgi Format=Excel&SQL=["SQL"]&Param1=["Start"]&Param2=["End"] Informix Excel Reports Technical Guide OLTP Holdings Page 4 of 9

WEB 1 http://server-name/cgi-bin/cgi-sql.cgi Heading=None&SQL=["SQL"]&Param1=["Start"]&Param2=["End"] As well as the SELECT statement itself ( SQL ), for each question mark place marker it contains (as for PREPARE in 4GL and Java), CGI-SQL needs a parameter ( Param1, Param2, etc.), giving names to be used in Excel ( Prefix, Start Date, etc.). Note that the second IQY file has Heading=None instead of Format=Excel to suppress column headings and tabulation from appearing in Excel. Once you have created the IQY file, you can then complete the remaining steps in Excel. The following is a step-by step guide to an example approach, with the SQL statement hidden on the worksheet together with the IQY file contents for future reference: 1. Open a new empty worksheet in Excel. 2. Enter SQL: in cell A1 and the SELECT statement in cell B1. 3. If you paste in the contents containing line feeds, make sure you have the cursor in the function bar (not the cell), and make it readable afterwards by highlighting cells B1:J1 (perhaps) and using Format Cells / Alignment / Vertical Top + Wrap text + Merge Cells, increasing the row height as necessary and changing the font to Courier New. 4. Enter IQY: in cell A2 and the IQY file contents in cell B2 for future reference (optional), repeating step 3 with cells B2:J2 as necessary. 5. Enter a parameter label in cell A3 and a value in cell B3. 6. Define input constraints and provide floating help using Data / Validation from the menu. 7. Repeat 5 and 6 as necessary. 8. Leave a blank row and go to column A on the next row and choose Data / Import External Data / Import Data from the menu. 9. Locate the IQY file and click the Open button. 10. Click Properties and change the settings of Refresh data on file open (usually but not in this example), Adjust column width and Preserver cell formatting (usually necessary) as desired. Click OK. 11. Click Parameters. For each one, click Get the value from the following cell, click in the input field and click on the cell containing the parameter value, ticking Refresh automatically when cell value changes as desired (used in this example although this is abnormal). Click OK when all parameters have been mapped to cells. 12. Click OK to complete the Import Data process. 13. Select rows 1 and 2 and choose Format / Row / Hide from the menu. Note that Excel malfunctions if a cell containing a date is mapped in directly as a Web Query parameter. To get around this, a hidden cell containing a text version of the date value should be set up using the formula illustrated below, where C1 (which you will need to substitute) is the location of the cell where the user will enter a date: =TEXT(C1, dd/mm/yyyy ) This hidden cell should then be mapped to the Web Query in the Parameters dialogue. See also General Instructions in the next section. Informix Excel Reports Technical Guide OLTP Holdings Page 5 of 9

General Instructions The following formatting might typically be applied to Excel reports: Screen Layout (except pivot tables and charts) 1. Highlight the column headings. 2. Press the Bold toolbar button. 3. Choose Data / Filter / AutoFilter from the menu. 4. Use toolbar buttons to set the Fill Color to Gray-25% and set borders around these cells. 5. While still on the column headings, highlight from there to end of file with Shift-Ctrl-End. 6. Press the Left Justify toolbar button. 7. Choose Format / Column / AutoFit Selection from the menu. 8. Go to column A of the first data row and choose Window / Freeze Panes from the menu. Print Layout 1. Choose File / Page Setup from the menu. 2. On the Page tab, set Scaling to Fit to 1 page(s) wide by 9999 tall. 3. Set Orientation to Landscape if necessary. 4. On the Header/Footer tab, press the set Custom Header button. 5. Enter text such as &[Tab] in Center section as desired. 6. On the Header/Footer tab, press the set Custom Footer button. 7. Enter &[File] / Page &[Page] of &[Pages] / &[Date] &[Time] in Left / Center / Right section. 8. Variables markers &[name] can be inserted in 4 and 6 using provided buttons. 9. On the Sheet tab, set Rows to repeat at top (typically $1:$1 ). Pivot tables are created from data worksheets as described below (note that this cannot be performed on Web Queries directly as you can with ODBC): 1. Select the base data worksheet and place the cursor in the Web query table (if any). 2. Choose Data / PivotTable and PivotChart Report from the menu. 3. Select PivotChart report (with PivotTable report) if desired. 4. Describe the columns or area to be included (done automatically for Web Queries). 5. Click Next and then Layout. 6. Drag columns objects to the required locations. 7. Double-click columns to change their properties. 8. Tip: Advanced / AutoSort can be useful. 9. Click OK when the Layout is finished. 10. Click Options and change settings as desired. 11. Tip: tick Refresh on open for dynamic workbooks. 12. Click OK and then Finish. Subsequent changes can be made using the Pivot Table toolbar (enable with View / Toolbars). To replace the contents of a work sheet without loss of format: 1. Open the file containing new data to be imported. 2. Right click at the top left to highlight all cells. 3. Select "Copy" from the drop-down menu. 4. Open the existing formatted work sheet. 5. Right click at the top left to highlight all cells. 6. Select "Paste Special" from the drop-down menu. 7. Choose "Values and number formats" and click "OK". Drop-down lists to pre-select a value for a field that does not appear in the body of a pivot table (known as "page fields") only provide for selection of one value or all. However, multiple selection can be achieved as follows: 1. Click the right mouse button on the drop-down list. Informix Excel Reports Technical Guide OLTP Holdings Page 6 of 9

2. Choose "Field Settings" (Excel 2000+) or "PivotTable Field" (Excel 95). 3. Click the left mouse button on all values you don't want in "Hide Items" and click "OK". 4. Click the right mouse button on the drop-down list. 5. Choose "Table Options" (Excel 2000+ only). 6. Make sure "Subtotal hidden page items" is not checked and click "OK". Internet Explorer can sometimes fail to detect that a file on an FTP server has been updated, and instead returns the version previously viewed from a local copy. The following steps should be taken whenever you suspect this is happening: 1. Select Tools / Internet Options from the Internet Explorer menu. 2. Stay on the "General" tab, which is displayed by default. 3. Click "Delete Files" within the "Temporary Internet files" box. 4. Leave "Delete all offline content" unchecked and click "OK". Hyperlink Base Excel workbooks containing Web Queries should be developed and tested on another system before being deployed on the live system. This means that Web Query URLs routinely need to be changed throughout a workbook to point to the appropriate server when the workbook is redeployed on another machine, which can be time-consuming. A way was therefore developed to do this. Installation 1. Obtain the file WebQueryServer.xla from the author. 2. Copy it into %APPDATA%\Microsoft\AddIns. 3. Start Excel and choose Tools / Add-Ins from the menu. 4. Tick Web Query Server and click OK. 5. Right click in the toolbar area are choose Customize. 6. On the Commands tab, choose Macros under Categories. 7. Drag Custom Button to the toolbar of your choice. 8. Right click on the new button and change Name to Web Query Server. 9. Select Change Button Image and choose your preferred icon. 10. Right click on the new button again select Assign Macro. 11. Enter the macro name WebQueryServer.Main. 12. Click OK and Close. Usage 1. Open the Excel workbook containing your Web Queries. 2. Choose File / Properties from the menu. 3. Select the Summary tab. 4. Set Hyperlink Base to the desired value. 5. Tip: you should also set Title, Author and Company. 6. Click OK and then the Web Query Server toolbar button. The add-in goes through your workbook changing all Web Query URLs to be relative to the provided hyperlink base. Note that Excel does not honour relative URLs in Web Queries and will not otherwise take account of your hyperlink base entry. Informix Excel Reports Technical Guide OLTP Holdings Page 7 of 9

CGI-SQL This is a general purpose infrastructure tool that dynamically returns the results of a single Informix SQL SELECT statement to a Web page or Excel Web Query, avoiding the need for any ODBC or OLEDB drivers on PCs and controlling the size and run-times of queries attempted. It can be called with a URL such as: http://server-name/cgi-bin/cgi-sql.cgi?sql=select+*+from+table-name+order+by+1 This invokes a Shell script located in the Apache CGI directory cgi-bin which will need modification for each installation: cgi-sql.cgi After setting environment variables, the Shell script runs ESQL-C program cgi-sql.exe located in the same CGI directory in place of itself. ESQL/C was the only language available in which the Informix SQL DESCRIBE statement can be used, and this is crucial for deriving the names of columns returned by the specified SELECT statement. However, the SELECT statement is actually run in a sub-process using "dbaccess", as wrapper statements OUTPUT and UNLOAD are not available in ESQL/C. The following CGI parameters are evaluated: Requiremen t Either And Name Value Comments SQL Heading Database MaxRows Password Param1-99 Format Single SELECT statement Free text Database name Maximum rows to select Program access password Query variables 1 to 99 "Excel" "Text" Required Default NULL Default $CGISQLDATABASE Default 65535 Fail!= $CGISQLPASSWORD Default NULL Default, tab-delimited data Formatted HTML These parameters are extracted using C functions contained in a general purpose CGI library by Andrew Girow: www.geocities.com/siliconvalley/bay/1927/yacgi.html Either GET or POST protocols can be used. However, POST is generally used wherever possible to hide CGI parameters and because GET has a limit on the total size. The ESQL/C program does the following: CGI parameters are extracted and validated or defaulted where possible. Any errors result in a message being returned via standard output as plain text HTML In particular, if parameter Password was specified, it must match environment variable CGISQLPASSWORD. Variable place markers (? ) in the SELECT statement and Header parameter (if supplied) are successively replaced with the values of CGI parameters Param1, Param2, etc. The SQL statement is checked to ensure that it is a single SELECT statement, and the number of rows returned (defaulting to the maximum that can be loaded into an Excel worksheet) is limited by insertion of a FIRST clause. If Format=Text was specified, the select statement will be prefixed with OUTPUT TO filename, otherwise a tab-delimited file suitable for loading into Excel will be created instead using UNLOAD TO filename. If environment variable EXPLAIN is set to ON, the Informix statement SET EXPLAIN ON will be applied beforehand. Informix Excel Reports Technical Guide OLTP Holdings Page 8 of 9

After writing the resulting SQL to a file, dbaccess executes it as a sub-process with the database name taken from environment variable CGISQLDATABASE. However, a timer in the ESQL/C program will interrupt and kill the process if it takes longer than the number of seconds held in environment variable CGISQLTIMEOUT or a default of 5 minutes if not set. A plain text or Excel MIME type header is returned, followed by any heading text to be displayed as specified in the Header CGI parameter. For Excel format, the ESQL/C program opens the database, uses DESCRIBE to determine the column names returned by the SELECT statement without actually running it, replaces underscores with spaces, upshifts the first letter of the resulting separate words, and displays them with a tab separator between columns. For Excel format, the report data is then displayed as is with the final tab on each line suppressed. Text format produced by OUTPUT, however, is beautified with a separate Awk script: cgi-sql.awk A general purpose CGI-SQL front-end Web page for IT users is also available: cgi-sql.html This is particularly useful for generating a complete URL to use in a new Excel Web Query. It should be run using Internet Explorer, as other browsers do not handle some of its Microsoft FrontPage validation features. CGI-CSV This is a Shell script that returns the contents of a CSV report (already produced by a scheduled job) to a Web page or Excel Web Query. It is located in the Apache CGI directory cgi-bin (some settings may vary depending on the platform): cgi-csv.cgi It is called from Excel Web Queries with a URL such as http://server-name/cgi-bin/cgi-csv.cgi?name=report-name and does the following: An Excel MIME-type is returned in the header if the URL contains Format=Excel. Otherwise, data is returned as a text/plain preformatted HTML page. The value of the CGI Name parameter is extracted from the URL and downshifted. This is assumed to be the name of a sub-directory of the public FTP area. The latest compressed CSV file in this directory is identified. Its contents are returned converted to tab-delimited text. This last stage uses gunzip, sed and csv2unl (see unl_utils at www.iiug.org). Informix Excel Reports Technical Guide OLTP Holdings Page 9 of 9