MSQuery-DM ad hoc reporting from SQL Server 9/25/2013

Size: px

Start display at page:

Download "MSQuery-DM ad hoc reporting from SQL Server 9/25/2013"

Amberlynn Hamilton
6 years ago
Views:

1 The ad hoc report feature in the Digital Measures web application has limited options for selecting the rows and columns to be reported. An alternative is to use MSQuery, or some other desktop tool such as SAS, against a copy of DM data updated weekly on SQL Server. For more complex reports, a custom DM report should be requested. The database is DigitalMeasures on server ardbprodone.ad.uky.edu. Contact Diane Gagel if you have questions about your query. The query script can be sent via and pasted into the MSQuery tool. MSQuery selections should be saved for reuse. MSQuery is a graphical tool to be used with relational databases for building Standard Query Language (SQL) select statements. Select statements identify the table or view (in our case DM screen), the columns (fields on screen) and the where clause (the criteria for selection). Using Excel MS Office 2007 (MS Office 2010 should be similar): Open MS Excel Data -> From Other Sources -> From Microsoft Query After choosing Microsoft Query, you will be prompted to Choose Data Source, there are 3 tab options: 1) Databases: Select the database for creating a new query 2) Queries: Select from existing queries to run, change or copy 3) OLAP Cubes None currently available for DM data 1

Option 1 - To create a new query, under the Databases tab, choose the data source called DMCopySQLServer. {The data source must be created first see DSN creation instruction handout.

Depending on the authentication specified in the DSN setup, you will use either the LinkBlueID (recommended set up for DM), or a SQL Server ID.

2 Option 1 - To create a new query, under the Databases tab, choose the data source called DMCopySQLServer. {The data source must be created first see DSN creation instruction handout.} Recommended: uncheck Use the Query Wizard to create/edit queries Click OK. You may prompted to login to the DigitalMeasures SQL Server database. Depending on the authentication specified in the DSN setup, you will use either the LinkBlueID (recommended set up for DM), or a SQL Server ID. When logging on, never choose the option to save UserID and Password Option 2: to use an existing query, under the Queries tab, scroll to find the query saved previously, select the query and click Open. TIP: if you want to create a query based on the copy of an existing query, open the existing query and then Save As with a different name before making the changes. 2

3 MS Query graphical interface for specifying selection criteria and viewing results: 3

These views were specifically created for college ad hoc reporting. You will not have access to data in other tables or views.

4 Creating a new query from scratch, with the MSQuery wizard disabled. 1) Select the table (view) on which the query will be based. After selecting the database, you should be prompted with the Add Tables dialog, to select the tables or views on which the query will be based. Use only the entries that begin with CollView_. These views were specifically created for college ad hoc reporting. You will not have access to data in other tables or views. Scroll to the desired table, and double-click the name, or select and click the Add button: This creates a table field-selection box in the tables area of the work window: Click the Close button to exit the Add Tables dialog window. The Add Tables dialog can be initiated from the menu, Table -> Add Tables... Important Tips: Tip 1a. If more than one table is selected, join criteria must be specified or else the result set could be very large and will likely not be the desired result set. Define the query criteria one table at a time. If more tables are needed, add them one at a time. This will simplify creating joins and verifying the results. 4

menu. This will prevent the query from being executed with every new addition to the selection criteria or

NOT RECOMMENDED: Automatic Query checked RECOMMENDED: Automatic Query unchecked To delete a table from

5 Tip 1b. Before choosing the first column in the first selected table, uncheck the Automatic Query option in the Records menu. This will prevent the query from being executed with every new addition to the selection criteria or result set. NOT RECOMMENDED: Automatic Query checked RECOMMENDED: Automatic Query unchecked To delete a table from selection: If the wrong table is selected it can be removed once the Add Tables dialog is closed, by selecting the table in the Table area. Select columns to include in the results area, by double-clicking the column name in the table. If the asterisk is double-clicked, all columns are copied in, which is not always desirable. 5

2) Specify the query criteria to limit the result set. In most cases you will want to specify selection criteria, otherwise all the rows in a table will be returned.

The values button attempts to find all possible values for a field and can require a significant amount of time to return the results.

6 2) Specify the query criteria to limit the result set. In most cases you will want to specify selection criteria, otherwise all the rows in a table will be returned.. From Criteria menu select Add Criteria. Using Add Criteria dialog window, select the field, operator and value. It is NOT RECOMMENDED to use the Values button with the DM views. The values button attempts to find all possible values for a field and can require a significant amount of time to return the results. Use the DM data dictionary to find the possible values (or the DM entry screen could also provide valid values). TIP: For best performance for college level queries, always include the AuthCollCode as a criterion and specify the appropriate college. Click Add button to add the criteria to the criteria area: 6

The screen shot below shows the criteria area appearance when the and operator is used:

7 Specify another criterion: Click Add button: Continue to add criteria with the Add Dialog, note that MSQuery defaults the logical connection operator as And. The screen shot below shows the criteria area appearance when the and operator is used: AuthCollCode = NU AND Status = Published AND DTY_PUB = The results include publications where all are true. 7

The screen shot below shows the criteria area appearance when the or operator is used: Status = Published or DTY_PUB = 2009.

) Use the Records menu Query Now function to run the query and populate the results area.

8 The screen shot below shows the criteria area appearance when the or operator is used: Status = Published or DTY_PUB = The results would include publications where either is true (resulting in all published items and no restriction by year.) Use the Records menu Query Now function to run the query and populate the results area. Select the Last Record icon circled below in green, to scroll to the last record, and show the record count. It is good practice to check the record count on the query results to be sure you re in the ballpark expected count, and to double-check results after changes have been made to the criteria. 8

Saving the Query After the query specifications have been created or changed to yield the desired results, Save the query using the File -> Save option (for new query or to update an existing query).

9 Saving the Query After the query specifications have been created or changed to yield the desired results, Save the query using the File -> Save option (for new query or to update an existing query). Use the Save As option to save an existing query to a new copy. When naming queries, use a name that describes the query, for example DM IntellContr Abstracts for UserID for Year MSQuery files use the extension.dqy. Accept the windows default location for storing the query which will be some place under the Desktop -> user directory -> AppData -> Microsoft -> Queries. It is not readily apparent how to permanently set the location in MSQuery. When you get a new desktop, remember to retrieve copies of MSQuery files. 9

10 Saving the results to MS Excel Use File -> Return Data to Microsoft Office Excel to save the data to an excel spreadsheet. 10

11 DM View contents Refer to the DM Data Dictionary on the APFA Digital Measures page to identify the column names associated with the fields on the DM screens, and for the valid values for the selection criteria. All views have the following fields in common: Field Name AuthCollCode ConID DMUserID DMUserName DispNameLastFirst FNAME PFNAME MNAME LNAME NAME_PREFIX NAME_SUFFIX CREDABBR Description Authorization college code the college to which the faculty member is or was once appointed. This field controls access to the data for ad hoc reporting. Contribution ID is a unique identifying number for the DM entry. (The field is called ID in the DM data dictionary.) Internal DM ID number of the faculty member to which the information belongs. (The field is called User_ID in the DM data dictionary.) Faculty member s LinkBlue ID Faculty member s name Last, First and Middle Initial First name from Personal Contact Information screen Preferred first name from Personal Contact Information screen Middle name from Personal Contact Information screen Last name from Personal Contact Information screen Name prefix from Personal Contact Information screen Name suffix from Personal Contact Information screen Credential abbreviation from Personal Contact Information screen Additional common fields for tables with Shared Data (Intellectual Contributions, OSPA grants, etc.) USER_REFERENCE_CREATOR AUTHOR_LIST or contributor list Yes indicates that the item was initially entered by this faculty member, and all other faculty associated with the item share the single DM entry. DM entry protocol indicates that primary contributor should enter the item. To obtain unique items, specify in criteria: USER_REFERENCE_CREATOR = Yes System calculated list of contributors associated with an entry 11

12 View Name CollView_AwardHonor CollView_Consulting CollView_Education CollView_EndAppt CollView_FacDev CollView_IntellCont_AllContr CollView_IntellProp_AllContr CollView_InterestExpert CollView_MedCont CollView_Member CollView_PCI CollView_PerformExhibit_AllContr CollView_Presentations_AllContr CollView_Presentations_AllContr_ UniqueTitles CollView_Res_NonOSPA CollView_Res_OSPA CollView_ServColl CollView_ServDept CollView_ServProf CollView_ServPubl CollView_ServUniv CollView_YearlyData DM Screen Name Awards and honors Consulting Education Endowed professorship and chair appointments Faculty development activities Intellectual Contributors multiple entries for all contributors Intellectual Property multiple entries for all contributors Interests and Expertise Media contributions Professional Memberships Personal and Contact Information Artistic and Professional Performances and Exhibits Presentations multiple entries for all contributors Presentations unique titles by faculty user multiple entries for all contributors. This is a special view which eliminates duplicate presentations with the same title, and provides the most recent start date. Research (Non-OSPA) Research (OSPA) Service to College Service to Department Service to Profession Service to Public Service to University Yearly Data (academic appointment status for calendar year Jan-Jun or Jul-Dec Scheduled Teaching, DOE data, and other views are not currently provided, since data from these screens is reportable (and in more detailed form) in the institutional data warehouse. 12

13 DM screen field names vs DM internal field names: The field names visible on the DM screens are labels, and are not the true field names in the DM database tables. Refer to the DM Data Dictionary available from the APFA Digital Measures page. As illustrated in the screen shot below, the data dictionary provides the field name (Code), whether the field is coded (Display = Drop Down, and the valid codes (Options). The exact spelling of the option values must be used in the selection criteria: Example: On Intellectual Contributions screen, the Current Status field is called STATUS in the data dictionary, and the valid values are In Preparation; Not Yet Submitted Working Paper Submitted Revising to Resubmit Not Accepted Accepted Published To select in preparation items, the selection criteria would include: STATUS = In Preparation; Not Yet Submitted 13

DM Date fields: Some activity items occur on a specific date, others may take place over a period of time, and therefore DM entry requires (respectively) a single date or date range.

14 DM Date fields: Some activity items occur on a specific date, others may take place over a period of time, and therefore DM entry requires (respectively) a single date or date range. In addition, DM allows for partial date entry of either: the year only; the month and year; or month, date, and year. In the DM Data Dictionary, each component of a date field on the screen is stored in a separate month, day or year field. Single Date Example: Intellectual Contributions, Date Published In the DM Data Dictionary, the following fields are present under Date Published: DTM_PUB Month Published stored as full name of month or quarter January (1st Quarter/Winter), February, March, etc. DTD_PUB Day Published stored as 01, 02, 03, DTY_PUB Year Published stored as 2001 To support selection of data by partial dates, each date field on the screen has an associated start-date and end-date pair in the data table, to store the effective period of the partial date. These start and end date pairs are not listed in the DM Data Dictionary, but are present in the data tables in SQL Server. PUB_START PUB_END DM calculated publication start date, stored as yyyy-mm-dd DM calculated publication end date, stored as yyyy-mm-dd Single Date Example: (continued) Date entered on DM Screen Internal Calculated (yyyy-mm-dd) DTM_PUB DTD_PUB DTY_PUB PUB_START PUB_END January (1st Quarter/Winter) January (1st Quarter/Winter)

Date Range Example: Consulting, Start and End Dates In the DM Data Dictionary, the following fields are present under Start Date: DTM_START Month Started stored as full name of month (no quarters)

15 Date Range Example: Consulting, Start and End Dates In the DM Data Dictionary, the following fields are present under Start Date: DTM_START Month Started stored as full name of month (no quarters) January, February, March, etc. DTD_START Day Started stored as 01, 02, 03, DTY_START Year Started stored as 2001 In the DM Data Dictionary, the following fields are present under End Date: DTM_END Month Ended stored as full name of month (no quarters) January, February, March, etc. DTD_END Day Ended stored as 01, 02, 03, DTY_END Year Ended stored as 2001 In the DM tables the following calculated dates are present: START_START DM calculated item start date start, stored as yyyy-mm-dd START_END DM calculated item start date end, stored as yyyy-mm-dd END_START DM calculated item end date start, stored as yyyy-mm-dd END_END DM calculated item end date end, stored as yyyy-mm-dd Date entered on DM Screen Internal Calculated (yyyy-mm-dd) DTM_START DTD_START DTY_START START_START START_END January January Date entered on DM Screen Internal Calculated (yyyy-mm-dd) DTM_END DTD_END DTY_END END_START END_END June June

When specifying selection criteria for dates, consider the usage directions indicated in the DM Data Entry Guidelines document, per example below.

16 When specifying selection criteria for dates, consider the usage directions indicated in the DM Data Entry Guidelines document, per example below. Date selection criteria examples: Screens with single dates: a) Select all items published in calendar year 2011: When selecting by calendar year or years, it is simplest to use the DTY_ field..and DTY_PUB = 2011 Multiple years:.and DTY_PUB in ( 2010, 2011 ) However, the PUB_END calculated date field could also be used:.and PUB_END between and Multiple years:.and PUB_END between and

17 b) All items published in fiscal year Fiscal year selection requires consideration of the month, therefore need to use the internal calculated date fields PUB_START and PUB_END Reminder how PUB_START and PUB_END are populated: If no month is provided, the PUB_START and PUB_END span an entire calendar year. Date entered on DM Screen Internal Calculated (yyyy-mm-dd) DTM_PUB DTD_PUB DTY_PUB PUB_START PUB_END January (1st Quarter/Winter) January (1st Quarter/Winter) Which dates are used impacts what results will be returned: Using PUB_START and PUB_END will result in double-counting publications with no publication month information:.and ((PUB_START GE ) AND (PUB_END LE= )) Using the selection above, for fiscal year 2011 (7/1/2010 thru 6/30/2011), the publication would be selected based on the PUB_START date of 1/1/2011. Using the selection above, for fiscal year 2012 (7/1/2011 thru 6/30/2012), the publication would be selected again based on the PUB_END date of 12/31/2011. Using PUB_END date only, will result in single-counting publications with no publication month information:.and ((PUB_END between and )) Using the selection above, the publication would be counted only in fiscal year

18 Screens with date ranges: Selections are more complex to accommodate items which may occur on a single day, or may be active for an extended period, or open-ended. Again consider the data entry instructions for the screen. These examples use the Consulting table, which has the following date entry instructions: *Start Date (blank if one day) *End Date (specify if one day) a) Select consulting entries which occurred in calendar year 2011: Use DTY_END field to find items which occurred on a single date in the period or a continuous activity which ended in the year Use DTY_START and DTY_END to find items which are still active, but started on or before the selection year The OR operator is required because either expression will satisfy the criteria. Using the AND operator would select only items ending in AND ( (DTY_END = 2011 ) OR ((DTY_START between 1900 and 2011 ) AND DTY_END = )) The calculated full date fields could also be used:.and (((END_START >= ) AND (END_END <= ))OR ((START_START between and ) AND END_END = )) 18

19 b) Select consulting entries which occurred in fiscal year 2011: Fiscal year selection requires consideration of the month therefore need to use the internal calculated date fields START_START and END_END. Use END_END date to find items which occurred on a single date in the fiscal year or which were continuous activities which ended in the fiscal year Use START_START and END_END to find items which are still active (have no End Date, but started on or before the selection fiscal year The OR operator is required because either expression will satisfy the criteria. Using the AND operator would select only items ending on AND ( (END_END between and ) OR (( START_START between and ) AND END_END = ) ) 19

20 Selection Criteria with OR operator In SQL syntax, the AND operator takes precedence over the OR operator. Parenthesis are used to control how the OR d fields are evaluated to return the appropriate results. Example: retrieve all journal articles for the faculty with linkblueid of DLDAVE0 published or submitted in calendar year The following are equivalent SQL statements: Form A) select DispNameLastFirst, TITLE, CONTYPE, STATUS from Coll_View_IntellContributions where DMUserName = DLDAVE0 and CONTYPE like Journal Article% and (DTY_PUB = 2011 OR DTY_SUB = 2011 ) Form B) select DispNameLastFirst, TITLE, CONTYPE, STATUS from Coll_View_IntellContributions where (DMUserName = DLDAVE0 and CONTYPE like Journal Article% and DTY_PUB = 2011 ) OR (DMUserName = DLDAVE0 and CONTYPE like Journal Article% and DTY_SUB = 2011 ) When the OR operator is involved, MSQuery builds SQL script using Form B and graphically represents the criteria grouped in parenthesis with two lines with the or indicator to the left. It is important to remember to include all criteria on each line, if it should be evaluated together to produce desired results. For example, the DM selection criteria graphically represented below, without the fully populated second line, does not produce the same results as above. Instead, it is equivalent to the SQL statement select DispNameLastFirst, TITLE, CONTYPE, STATUS from Coll_View_IntellContributions where (DMUserName = DLDAVE0 and CONTYPE like Journal Article% and DTY_PUB = 2011 ) OR (DTY_SUB = 2011 ) which would produce a list of journal articles published in 2011 for DLDAVE0, appended to a list of all intellectual contributions for all faculty submitted in

SQL Script View To see the SQL select statement produced by your specifications, go to the View menu A SQL Dialog box will open with the SQL script.

For significant changes, or for new scripts, it is recommended that the script be typed in notepad or some other text editor and then copied into the SQL dialog box.

21 SQL Script View To see the SQL select statement produced by your specifications, go to the View menu A SQL Dialog box will open with the SQL script. Individuals proficient in building SQL select statements may wish to edit the script directly. It is cumbersome to type script in the dialog box, which cannot be resized. For significant changes, or for new scripts, it is recommended that the script be typed in notepad or some other text editor and then copied into the SQL dialog box. The DM technical support person can send DM script via for copying into the dialog box. Important notes: 1) When scripts are pasted into the dialog box, the unchecked state of the Automatic Query option is ignored, and MSQuery attempts to retrieve the information immediately upon selecting OK on the SQL Dialog window. If the where clause does not limit the rows sufficiently, this could cause MSQuery to give the Not Responding message. When this occurs, the task should be ended in the Task Manager. 2) SQL scripts with aggregation items in the select clause cannot be represented graphically. This means that the table and selection criteria areas cannot be represented by MSQuery, but the query is still good. The results should be returned and can be saved to Excel. 21

22 SQL Expression examples To select a set of values use In() STATUS in ( Accepted, Submitted, Published ) To select a range of values use between START_START between and To select with wildcard, use like and the % CONTYPE like Journal Article% - starts with Journal Article CONTYPE like %Journal Article - ends with Journal Article CONTYPE like %Journal Article% - Journal Article anywhere in text To exclude a set of values, use NOT ( ) NOT (STATUS = Accepted ) To select unique published new books, excluding book chapters: Given these CONTYPE values for books: Book Book Review Book, Chapter Book, Chapter in Non-Scholarly Book-New Book, Chapter in Non-Scholarly Book-Revised Book, Chapter in Scholarly Book-New Book, Chapter in Scholarly Book-Revised Book, Chapter in Textbook-New Book, Chapter in Textbook-Revised Book, Non-Scholarly-New Book, Non-Scholarly-Revised Book, Scholarly-New Book, Scholarly-Revised Book, Textbook-New Book, Textbook-Revised SELECT field names.. FROM CollView_IntellCont_AllContr where AuthCollCode = 'NU' and STATUS = 'Published' and CONTYPE like Book% and not(contype like Book, Chapter% ) and not(contype like %Revised% ) and not(contype like Book, Review% ) and USER_REFERENCE_CREATOR = Yes 22

Joins Should generally not be necessary, unless information from the two screens below is required in conjunction with productivity (activity) information: CollView_PCI Personal Contact Info 1 Row

23 Joins Should generally not be necessary, unless information from the two screens below is required in conjunction with productivity (activity) information: CollView_PCI Personal Contact Info 1 Row per faculty member CollView-YearlyData Academic Appointment Multiple rows per faculty member Joins will be covered at the next session. Avoid for now or ask for scripts from the DM Technical Support person. Joins will be based on the key field DMUserName which ties the faculty member s activity data with the YearlyData (academic appointment) Example, list all intellectual contributions published for faculty in academic year 2012, along with title series, rank, and department. 23

Notes about Yearly Data: A faculty member will have two Yearly Data records in a calendar year to capture academic status information for first half of year (01 Jan-Jun) and the second half of year

24 Notes about Yearly Data: A faculty member will have two Yearly Data records in a calendar year to capture academic status information for first half of year (01 Jan-Jun) and the second half of year (02 Jul-Dec). Since DM does not support using exact effective dates on the Yearly Data record, this design allows for capturing the status at various two points in the calendar year to support fiscal year reporting. Yearly data rows for one faculty member, who started employment 8/16/2011: Calendar Year (TYY_TERM) Time Period (TYT_TERM) Internal Date TERM_START Internal Date TERM_END JulDec JanJun JulDec JanJun JulDec The academic appointment status effective on the TERM_END date is stored with each row. Results sets joining with Yearly Data, therefore may have duplicate activity rows depending on the selection on Yearly Data. Simplest options to deal with duplicates: a) select the Yearly Data for the last period in the selection or b) add distinct option and remove TERM_START and TERM_END from select list, this will result in one row per activity, unless the title series, rank, or some other academic status has changed for the individual. 24

REFERENCE GUIDE FOR MANUAL DATA INPUT v1.1

REFERENCE GUIDE FOR MANUAL DATA INPUT v. TABLE OF CONTENTS Introduction User Roles Logging in to VIVO Site Administration Menu Navigating VIVO Data Input Overview Profile Fields Mapping a CV to a VIVO