Introduction To The Data Browser

Transcription

1 Introduction To The Data Browser Software v5 - January 2017 Strictly for distribution solely within EDI, surveybe and their clients under the conditions agreed. Surveybe reserves the copyright on all software manuals and training materials. ECONOMIC DEVELOPMENT INITIATIVES (EDI) LTD Prospect House, T: + 44 (0) Crendon Street, E: info@surveybe.com High Wycombe, W: Buckinghamshire W3W: agrees.means.stay HP13 6LA UK 0 P a g e

2 Table of Contents 1. Introduction How is Data Stored in Surveybe? Data Browser The Data Browser Window The SQL Editor and the Result Panel Writing SQL Statements in the Data Browser Testing SQL Statements Selecting Columns from a Table Using WHERE Clause Selecting Columns From a Table Using the AND/OR Operators Selecting Plot with Largest Area Checking the Distance Between 2 GPS Coordinates How to Query and View Data Held in Interview Files Checking for Empty Answers or Null Values Calculating Average Weight and Height Checking Status of Interview Files Joining Roster Tables Calculating Duration of Interviews Examining Reference Data Testing SQL Performance Appendix A: The Command Menu and Database Structure Browser Answer Schema Information_Schema Materialized Schema Appendix B: The current. keyword and Other Specialist SQL Single Variable Dynamic Text Dynamic Text Select Statements Dynamic Question Row source Boolean Expressions P a g e

3 1. Introduction The Data Browser is a tool which can be accessed from within the Designer. You can use it to develop and test SQL statements for your questionnaire directly against reference or interview data. Similarly, you can use it to retrieve reference or imported answer data before exporting your interview files to Stata, SPSS or CSV format. Although the Data Browser may initially look a bit intimidating, with a little learning it s straight forward to use and can bring significant benefits to your project. This guide will help you get the best from the Data Browser. It supplements the main Surveybe Designer User Manual and covers the following topics: How data is stored within Surveybe Introduction to the Data Browser Window How to query the Surveybe materialized database which holds questionnaire tables using the Data Browser. This includes the questionnaire main table, roster tables, comments table and reference tables. How you can improve the performance of your questionnaire within the Implementer Please note that the Data Browser does not allow you to edit or delete any data from the database it is simply for viewing purposes and therefore, facilitates data processing, checking and questionnaire design accuracy. Although the Data Browser doesn t currently have an export function for the results set you generate, you can copy and paste the results into compatible applications like Google Sheets or Microsoft Word or Excel. We d strongly encourage all users to read this and try the Data Browser as its instant feedback will: 1) Help inexperience SQL users explore SQL syntax and visualise the Data Structures 2) Help more experience users develop sophisticated functionality for their questionnaires 2. How is Data Stored in Surveybe? Before we get into the details of how the Data Browser works, it is important to understand how data is stored within Surveybe. Every questionnaire has at least one table, the Questionnaire Table. Every question that is not on a roster is part of the Questionnaire Table. A Questionnaire Table needs to have at least one question. When you create a questionnaire with rosters you will need to define tables for each of the rosters, usually a table for each roster. When data is collected and the interview files are imported into the database (Designer > Import interview files) each of these tables create a data file. To help illustrate this, let s take the following example. Let s say we have a simple household survey where we want to list the name of all household members and record their age. To record the name and age of each household member, we have added a roster. As we need to have at least one 2 P a g e

4 question which sits directly on the Questionnaire Table, we have added a Timestamp Answer-Type question to record the time the interview started. The structure of the questionnaire looks like this in the Designer: New Questionnaire > Questionnaire identifiers -- Household ID : HHID Questionnaire > Screen -- Interview start Time: Start_Time > Roster: Mem_ID --- Household Member Name: MemName --- Household Member Age: Age When the interview files are exported from the Designer, the answers linked to the Questionnaire Table will look something like this. Table Name: Household Column: Questionnaire Identifier Column: Question Variable 1 HHID TimeStampQ 1 01/01/ :00: /01/ The answers linked to the roster table will look like this: Table Name: Household_Member Column: Questionnaire Identifier Column: Table ID Column: Question Variable 1 Column: Question Variable 2 HHID Mem_ID MemName Age 1 1 Steven Wise Louise Wise Thomas Wise Matt Stone Sarah Stone 25 In the data file generated for Questionnaire Table, HHID is the Questionnaire Identifier (used to identify interview file) and TimeStampQ is the questionnaire variable of the Time stamp question. In the data file generated for the Roster Table, HHID is the Questionnaire Identifier, Mem_ID is the Table ID, MemName and Age are the question variables in the table Household_Member. 3 P a g e

5 In order to retrieve any record from a table in the database, you will need to make use of SQL statements in the questionnaire and within the Data Browser. 3. Data Browser There are two ways in which you can access the Data Browser. When a Questionnaire is selected in the Designer, clicking the File menu followed by Data Browser, or by using the keyboard shortcut Ctrl+Alt+D will open a new tab that is connected to the materialized database for the selected Questionnaire. Each questionnaire held in the Surveybe Designer creates a separate materialized database. If you are working on multiple questionnaires, you can open a separate Data Browser for each questionnaire. The Data Browser is useful for four main purposes: The materialized database can be used to develop and test any SQL statement that is included within your Questionnaire. This includes dynamic answer queries, table prepopulation statements, enablement rules, dynamic text statements and validation rules. You will need to ensure that your interview file contains some dummy data so that the result can be visualised within the Data Browser. Note: You may need to modify some SQL statements. 4 P a g e You can use the Data Browser to directly access reference tables that have already been uploaded into your questionnaire. You can use SQL statement to query and view all or a subset of data from your reference tables.

6 Data contained within interview files that have already been collected can be viewed or queried. If used appropriately during the piloting stage the Data Browser could help eliminate data problems that may normally only be seen later during data processing. Similarly, after field work, it can be used to view or query data held in interview files before they are exported to Stata, SPSS or CSV format. You can also use SQL statements to perform more sophisticated queries using the collected data. By using the JOIN clause in your SQL statement, you can retrieve data from two or more tables and create a temporary table that can be viewed within the Data Browser. We have shown an example of how to use INNER JOIN (JOIN) in section 7.4. To close a Questionnaire Data Browser click on found at the top of the left hand corner. Alternatively, click on the X button located in the top corner of the window. 4. The Data Browser Window The Data Browser is composed of 4 main areas; Command Menu, SQL Editor, Result Panel and Data Structure Browser. You will mainly need the Command Menu and the Result panel to run and view your queries. The Data Structure Browser and Command Menu are explained in Appendix A. Please note that the Data Browser will timeout after 30 minutes of inactivity. If you try to run commands after the timeout, the browser will ask you to login. To use the browser again, close the current tab and re-open it from the File menu. 5 P a g e

7 4.1 The SQL Editor and the Result Panel The SQL Editor area is where SQL Statements are entered. It can hold and run multiple commands. If you choose to run multiple commands at once you will need to separate them with the standard SQL semi-colon ; character. Select Run to execute all the commands at once. Alternatively, if you have multiple commands within the Editor, you can run each of them individually by selecting the statement then clicking Run Selected. Please ensure that you set the materialized schema first. If you switch the Auto Complete to Full, as you type the command suggestions will appear. The Result Panel shows the result of the executed SQL command(s). The time taken to execute an SQL command will also appear below the result; see screenshot below. This will give you an indication of how efficient your statement is. If you have multiple commands and you don t use the semi-colon to separate them, the result panel will display a syntax error. Whether you want to test an SQL command or view data held in your interview files, it is important to note the following key points. These have also been explained in more details in the sections below. 1) The database contains three schemas; ANSWER, INFORMATION and MATERIALIZED. The three schemas are explained in detail in Appendix A. The Materialized schema is used to hold all tables 6 P a g e

8 that make up the data structure defined in the questionnaire and therefore, all table names need to be fully qualified when used in the Data Browser. This means that each time a SQL statement refers to a table we need to tell the browser which schema contains the table. To do this, we use Select ColumnName FROM Materialized.Table rather than simply using Select ColumnName From Table. Note: To work efficiently and avoid having to type in Materialized. before every table, you may issue and run the command Set Schema Materialized to force the browser to assume that all commands are running within the MATERIALIZED schema. Using Select ColumnName From Table will then work. Use a semi-column after each SQL command if you want to display the results within the same Results Panel. The Materialized Schema has been explained in Appendix A. Once you ve run the command set schema materialized, you may delete it from the SQL Editor panel if you want. Any other commands that you write hereafter will still execute if written correctly. If you close and re-open the Data Browser you will need to set the materialized schema again. Note: If you have multiple browsers opened for different questionnaires you will need to set the schema in each browser. 2) You ll find that some SQL used in questionnaires (using the.current syntax) won t work directly in the Data Browser. You will need to adapt this SQL to make it run by using full SELECT statements. For example, instead of using current.columnname, use Select ColumnName from Materialized.Table or use Select ColumnName from Table if you ve already set the materialized schema. 3) When SQL runs in the Implementer it runs against the answer data for a single questionnaire. In the Data Browser it runs against all the interviews you ve imported for that questionnaire. If you want to want to check if an SQL command is returning the correct result, you can create one interview file containing some dummy data in the Implementer and import it into the Designer. You can then write and test the SQL statement against that interview file. Only have one interview imported if you are doing a test, otherwise you will need to add a filter condition in the SQL. 7 P a g e

9 Set Schema Materialized; Use full SELECT statements; If testing multiple commands separate them with semi-column; Results of SQL commands Time taken to execute SQL command In the Result Panel, when the SQL command is executed, the variables being referred to in the SQL will each generate a separate column, for example, Questionnaire Identifier, Table ID, Question Variable. You can sort the result set by clicking on a column heading. You can select then copy & paste all or part a results table into other applications like Google Sheets or Microsoft Excel or Word. 8 P a g e

10 5. Writing SQL Statements in the Data Browser SQL statement used within the surveybe questionnaire is often a simplified version of the SQL statement that is actually executed on the database. Within a questionnaire, Surveybe allows the non-standard keyword current. which is taken to mean the variable in the current context. For example, when dealing with a household roster, using current.memname will be taken to mean the value of column memname for a specific row, rather than the more standard SQL meaning of the value of the column memname for all rows. Hence it is applying a WHERE clause to the column. This means that anytime a statement such as current.varname is used it is transformed into SELECT varname FROM tablecontainingvarname WHERE ID=? However, when using the Data Browser, you will need to write full SQL SELECT statements. The relational database used by Surveybe is the H2 database. The full description of H2 s SQL syntax can be found on the H2 website at While the syntax of the SQL statements can be tested without data as described above, testing with data is recommended, as the rows returned by the SQL may sometimes give unexpected results. This sometimes happens when answers give a value of NULL, or multiple rows are returned unexpectedly, or data types are mismatched. Data can be generated pre-pilot by running some dummy interviews in the Implementer and then importing the interviews back into the Designer. Testing again during piloting may be useful, as data generated during piloting is likely to be more realistic than dummy data. The most basic SELECT statement has only 2 parts: (1) what columns you want to return and (2) what table(s) those columns come from. For example, Select Mem_ID as ID, MemName as Value From Household_Member HHID Mem_ID MemName Age 1 1 Steven Wise Louise Wise Thomas Wise Matt Stone Sarah Stone 25 Based on the table above, we can note that that this SQL statement will generate multiple records as there is more than one member is listed in the table. If you want to filter records and extract only those records that fulfil a specific condition you need to use the WHERE clause. Again, if there is more than one member who is over 18 years old listed in the interview files, the SQL statement will generate multiple records. For example, Select Mem_ID as ID, HHMemName as value From Household_Member where Age > 18 If you want to retrieve a single value (for example, to display in a Label) you will need to use the Top 1 clause in order to return only one record. For example, Select TOP 1 HHMemName from Household_Member where Age > 18 9 P a g e

11 If you have an empty database, i.e, there are no interview files imported into your Designer, when you run an SQL statement in the Data Browser, no rows will be returned. 6. Testing SQL Statements You can develop and test SQL statements for your questionnaire directly against reference or interview data. SQL Statements can be used in: Labels and Dynamic Text where you can only retrieve a single answer. Dynamic Answers where you can retrieve one or more answer options. Tables which can be pre-populated with data from a Reference Table. You can Pre-populate one or more Question/Label. Enablement and Skip conditions You can enable questions by adding Enablements and disable questions by adding skips based on one or more condition. Validations where you set logic patterns. When you test your SQL command in the Data Browser, remember to: SET SCHEMA MATERIALIZED so you don t have to type MATERIALIZED. before all the table names Only have one interview imported otherwise you ll need to filter out the answers from the other interviews. If you want to view all columns and rows of data contained within a particular table, you can use used Select * from Table. This allows you to view all records contained within the interview for that given table so that you can cross check the result of the SQL syntax that you want to test. 10 P a g e

12 Note: The column ID and Parent_ID displayed in the result panel is not part of your interview data it s a system generated value that identifies the row in the database table that reflects the current context. When writing SQL statements within your questionnaire you should not use these columns or values. Below are a few examples of how the Data Browser can be used to check if the SQL statement is returning the correct result. 6.1 Selecting Columns from a Table Using WHERE Clause In this example, we have shown an example of a simple SQL statement which can be run against the Household Members roster in a household questionnaire. Please remember to import an interview file containing some dummy data in the roster so that you can visualise the result. The following statement has been used on the Questionnaire Table to populate a Dynamic Answer- Type question with a list of household members who are over 18. The records of the household members are located in the Household_Member roster table. The SQL syntax used is: Select Mem_ID as ID, MemName as Value from Household_Member where Age > 18. Before running the test SQL statement, you can use Select * from Household_Members to view all the records founds in the Household_Members roster table and this can be used as a mean of comparison to check if SQL syntax that is being tested is returning the correct result. 11 P a g e

13 As you can note from the result set that has been generated, there are only 2 household members who are over 18. If needed, you can edit and run an SQL command again. 6.2 Selecting Columns From a Table Using the AND/OR Operators The AND & OR operators can be used to filter records based on more than one condition. In this example, we want to test the result of a Dynamic Answer-Type question by selecting all household members who are male and over 18. To do this, we have imported one interview file into the Designer containing dummy data. To double check that the result being displayed is correct, we have selected the household ID, name and age of all household members listed in the interview file, we have used: Select ColumName1, ColumnName2, ColumnName3 From Table. 12 P a g e

14 In this example, HHID is the Questionnaire Identifier and Mem_ID is the Identifier of the Household_Members Table. To select the members who are male and over 18 we have used: Select Mem_ID as ID, MemName as value from Household_Members where Sex = 1 AND Age > 18. The result is displayed in the screenshot below. 13 P a g e

15 To select household members who are female or are over 18, we have used the following SQL: Select Mem_ID as ID, MemName as value from Household_Members where Sex = 1 OR Age > 18. As you can note, the OR operator displays a record if either the first condition OR the second condition is true. In this case, all household members meet at least one condition. 6.3 Selecting Plot with Largest Area In this example, we want to list all the plots owned by a household and check if the SQL command used to calculate and display the largest plot of land works as expected. Before we retrieve the name of the plot with the largest area, we need to ensure that the area has been recorded in the same unit. If not, you can use a computed answer to convert the areas into the same unit (in this case, hectare). Computed answers have been explained in detail in section 4.6 of the main Designer user manual. The Computation SQL used in the computed Numeric Answer-Type question to convert all area units into hectares is: 14 P a g e

16 SELECT CASE WHEN current.area_unit = 1 THEN current.area WHEN current.area_unit = 2 THEN current.area * WHEN current.area_unit = 3 THEN current.area / ELSE 0 END Variable of question used to record area unit (Q3 in example Area_unit above). The area can be recorded in: 1 Hectares 2 - Acres 3 - Square Metres Variable of question used to record plot area (Q2 in example above) Area We have used current. in the above computation SQL as all the question being referred to within the SQL statement are located within the same table (Plots) and the SQL is being used within the Questionnaire. The following SQL command should return the name of the plot which has the largest area. Select Top 1 PlotName from Plots order by AreaSM Desc Top 1 PlotName Plots Order by AreaSM Desc Limit the number of rows returned to one Variable used to record name of plot (Text Q1 in screenshot above) Name of roster where all questions are located Sort AreaSM in descending order. AreaSM is the variable of computed question used to convert all area units into hectares. To check the result in the Data Browser, - In the Implementer, fill the Plots roster with some dummy data and import the interview file into the Designer. In this example, the interview file contains data as shown in the screenshot above. - Type in Set Schema Materialized in the Data Browser (if you haven t already done so) - To select and view all the columns from the Plots roster within the interview file use Select * from Plots order by Plot_ID The result will be sorted by Plot_ID 15 P a g e

17 - To select the plot (question variable: Plotname) with the largest area (question variable: AreaSM) from the Plots Roster use: Select Top 1 PlotName from Plots order by AreaSM Desc As you can note the result shows that Plot D has the largest area, that is, 20 hectares. If the roster contains more than one plot with the same area, the SQL command will return the first row. 6.4 Checking the Distance Between 2 GPS Coordinates In order to calculate the distance between two GPS coordinates you will need to use the following SQL statement. Select Distancebetween (Getlatitude(select GPS1 from Table), Getlongitude(select GPS1 from Table), Getlatitude(select GPS2 from Table), Getlongitude(select GPS2 from Table), 'K') Note that the functions used in this statement are user defined functions in the Surveybe database and not standard SQL. You can calculate the distance between two coordinates in Kilometres, Statute Miles, Nautical Miles or Meters. In the above general SQL syntax, K denotes Kilometres. If you don t add any specifier within the SQL statement, the result will be returned in Metres. The specifiers that can be used to calculate the distance are as follows: 16 P a g e

18 Unit Kilometres Statute Miles Nautical Miles Specifier K M N GPS coordinates recorded using Android Implementer include latitude, longitude and altitude. The DistanceBetween function given above takes the latitude and longitude into account in order to calculate the difference between two coordinates. The altitude is ignored. The function works by first extracting the latitude and longitude from both GPS coordinates before calculating the difference. In the following example, we have used the Data Browser to calculate the distance between 2 GPS coordinates in Metres, Kilometres, Statute Miles and Nautical Miles. To do this, - Import one interview file which contains both GPS coordinates (GPS 1 & GPS 2) into the Designer. - Open the Data Browser for the questionnaire. Set the materialized schema by typing set schema materialized; into the Data Browser. - When writing the SQL statement, specify the unit in which you want to calculate the distance. Use a semi-colon in between the statements if you want to display the results within the same Result panel. Click on the Run button to execute the commands. Meters: Select Distancebetween (Getlatitude(select GPS1 from Table), Getlongitude(select GPS1 from Table), GetLatitude(select GPS2 from Table), Getlongitude(select GPS2 from Table)) Kilometers: Select Distancebetween (Getlatitude(select GPS1 from Table), Getlongitude(select GPS1 from Table), GetLatitude(select GPS2 from Table), Getlongitude(select GPS2 from Table), 'K') Statute Miles: Select Distancebetween (Getlatitude(select GPS1 from Table), Getlongitude(select GPS1 from Table), GetLatitude(select GPS2 from Table), Getlongitude(select GPS2 from Table), 'M') Nautical Miles: Select Distancebetween (Getlatitude(select GPS1 from Table), Getlongitude(select GPS1 from Table), GetLatitude(select GPS2 from Table), Getlongitude(select GPS2 from Table), 'N') 17 P a g e

19 The results of all the SQL statements will be displayed in the same Result panel. You may need to scroll down in order to view the result that does not fit within your screen. In the following screenshot, we have executed the SQL command individually. Distance between GPS coordinate 1 and 2 in Metres 18 P a g e

20 Distance between GPS coordinate 1 and 2 in Kilometres Distance between GPS coordinate 1 and 2 in Statute Miles 19 P a g e

21 Distance between GPS coordinate 1 and 2 in Nautical Miles Similarly, to cross check the result, you could use literals of the coordinates instead of selecting the variable names from the tables. The SQL command shown in the following screenshot should give you the same result (in Nautical Miles) as above as they are the same coordinates recorded in the interview file that was imported into the Designer. 20 P a g e

22 7. How to Query and View Data Held in Interview Files The materialized schema contains not only reference tables but also Questionnaire Table, comments table and any table linked to your rosters (if your questionnaire contains any). This means that if you have imported any interview files into the Designer Import tab, you can view data contained in the tables in your interview files before exporting them to Stata, SPSS or CSV format. You can use SQL commands to query the data within your interview files. Again, in order to avoid typing in materialized. in front of all table names, you can type SET SCHEMA MATERIALIZED in the Data Browser. 7.1 Checking for Empty Answers or Null Values After data collection, you can use the Data Browser to check if there your interview files contain any un-answered questions. Using Select * From Table after setting the materialized schema or using Select * From Materialized.Table will display all data held in the Table for all interview files contained in the Materialized schema. You will need to do the same for all the tables in your questionnaire. The following example shows how to select all data contained in the Household_Members table within the two interview files that have been imported into the Designer. You can use the horizontal bar found at the bottom Result panel to view more columns. This allows you to view the all data in the Roster Table at a glance. As you can see the question with variable Marital_Status contains the value null which means that the question wasn t answered by the interviewers. 21 P a g e

23 This gives you the opportunity to follow up with the interviewers/field members before exporting the interview files to Stata, SPSS or CSV format Calculating Average Weight and Height In this example, we have shown you how you can use the Data Browser to calculate the average weight and height of children who are under five from data collected in the Anthropometry section of a household survey. To keep it simple, we have only imported 3 interview files into the Designer; altogether the files contain the records of three children who are younger than 5 years old. Before calculating the average height and weight you can select and check the data recorded in all 3 interview files. In the example below, we have selected the variables memname, age, height and weight from the roster table household_members for all the children who are under five. The General syntax to calculate the average (mean) value is Select AVG(columnName)From Table Therefore, using the following SQL command will give you the average weight and height of the three children in the Database. 22 P a g e

24 The following syntax shows how you can round the average value of height to 2 decimal places: Select ROUND (AVG (weight), 2) from household_members; 7.3 Checking Status of Interview Files If the result of each interview file is recorded at the end of the survey, you can use the Data Browser to check the status of all interview files at a glance. For example, you can find out how many interviews were completed, how many were partly completed and how many were not completed. In the following, the result of each interview file has been recorded via a Single Response question (Radio Button) within the Implementer. To check the status of each file after data collection, import all you interview files in the Designer. We have imported 50 interview files in this example. 1) Type in SET SCHEMA MATERIALIZED and run the following SQL commands to query the Questionnaire Table and find the interview results. Don t forget to separate the commands with semi-colon ; if you want to view the results at the same time in the results panel. Note: The Questionnaire table is called Household and the Drop-Down list question recording the result of the interview is called intvresult. There may be interview files where the result was not recorded; we have therefore also checked for Null answers. 23 P a g e

25 Total count of interviews imported into Designer 31 interviews were completed 2 households need to be revisited 1 interview file was partially completed 14 interview files where the respondent(s) refused to be interviewed 1 household no longer exists 49 interview files where the status was recorded. Checking the number of interview files where the result is NOT NULL will also give the same result. 1 interview file where interview result was not recorded 24 P a g e

26 We have checked for the total number of interviews by counting the total number of rows in the Questionnaire Table Household as each interview file imported creates a new row. We have counted the result of each interview by referring to the response value used to record the answer in the Implementer We have used NULL to check if the database contains any interview file where the result was not recorded. You can check for the interview file where the interview result was not recorded. To do this, select the questionnaire identifier(s) and identify the interview file where the interview result is null (empty) as shown in the screenshot below. 7.4 Joining Roster Tables You can use the Data Browser to create a temporary table by combining data from two or more tables. Inner join (also known as JOIN) can be specified in either the FROM or WHERE clause. The combination is done by using the JOIN clause and is based on a common field between the two tables. The general SQL syntax that s used to join tables is SELECT columnname(s) FROM Table1 JOIN Table2 ON Table1.columnName=Table2.columnName; The following example shows how you can join the Plots and Crops tables and view the name of all the crops recorded against each plot by combining the two tables. 25 P a g e

27 Plots PlotID (Table ID) PlotName Crops CropID (Table ID) CropName PlotID (common ID) In order to create the temporary table in the Data Browser, you will need to: Import your interview files so that the data that has been recorded in both tables can be viewed. In this example, we have imported 2 interview files into the Designer. Set schema Materialized; in the Data Browser. The Data Browser in the screenshot below shows the information held in the Plots and Crops rosters within both interview files. Household_ID is the identifier of the questionnaire. In Household_ID 1, two plots and two crops have been recorded. In Household_ID 2 one plot and one crop has been recorded. We will join the two tables based on the common field PlotID. The following variables were selected to be displayed in the result below: questionnaire identifier (Household_ID), the table identifiers (PlotID and CropID), name of the plots (PlotName) and name of the crops (CropName). 26 P a g e

28 Instead of selecting all the columns, you can specify the column you want. If any of the columns used in the SELECT statement is found in both tables (such as Household_ID) you will need to specify the name of the table when specifying the column as shown in the example below. 27 P a g e

29 7.5 Calculating Duration of Interviews If you add a Time Stamp question at the start and at the end of your questionnaire, you can easily calculate the total duration of an interview after data collection. You can view the result in the Data Browser check if interviews are being conducted within the expected time duration. Don t forgot to import your interview files into the Designer first and run set the schema by typing in Set schema Materialized into the Data browser for the selected questionnaire. Please note that Time Stamp answers are stored as strings (texts) in the Surveybe database. This means that in order to calculate the difference in time, you will need to parse the string as timestamps in the SQL command. The PARSEDATETIME function parses a string and returns a timestamp. The format of the parsed timestamp need to be as follows: dd/mm/yyyy HH:mm:ss dd Day in month MM Month in year Yyyy year HH Hour in day (0-23) mm Minute in hour ss Second in minute The general SQL command used convert the string into a timestamp is: SELECT PARSEDATETIME ((TIMESTAMP), 'dd/mm/yyyy HH:mm:ss') The SELECT statement in your SQL command should refer to the column(s) you want to display when it s executed in the Data Browser. In the following example, the SQL command is retrieving the columns from the table called Household. We also want to retrieve the value of a unique identifier so that we know which interview we are talking about. Select HHID, Start_time, PARSEDATETIME ((Start_Time), 'dd/mm/yyyy HH:mm:ss') from household HHID Start_Time End_Time PARSEDATETIME ((Start_Time), 'dd/mm/yyyy HH:mm:ss') PARSEDATETIME ((End_Time), 'dd/mm/yyyy HH:mm:ss') Household Questionnaire Identifier used to identify interview Question variable of Timestamp question at the beginning of the questionnaire Question variable of Timestamp question at the end of the questionnaire Parsing the question Variable Start_Time 'dd/mm/yyyy HH:mm:ss' as a timestamp data type Parsing the question Variable End_Time 'dd/mm/yyyy HH:mm:ss' as a timestamp data type Questionnaire Table where question variable HHID, Start_Time & End_Time are located Using the above SQL command in the Data Browser shows how the timestamp answer is saved in Surveybe and how it is saved when it s parsed as a timestamp data type (yyyy-mm-dd hh:mm:ss). 28 P a g e

30 Parse string as timestamp 1 Parse string as timestamp 2 Therefore, to calculate the difference in hours and in minutes we have used the DATEDIFF function as shown in the screenshot below. Difference between 2 timestamps in minutes Difference between 2 timestamps in Hours 29 P a g e

31 Select clusterid, hhid, Start_time, End_Time, (select datediff (minute, (PARSEDATETIME ((Start_Time), 'dd/mm/yyyy HH:mm:ss')), (PARSEDATETIME ((End_Time), 'dd/mm/yyyy HH:mm:ss')))),(select datediff (HOUR, (PARSEDATETIME ((Start_Time), 'dd/mm/yyyy HH:mm:ss')), (PARSEDATETIME ((End_Time), 'dd/mm/yyyy HH:mm:ss')))) from household; Clusterid, hhid Start_time, End_Time (select datediff (minute, (PARSEDATETIME ((Start_Time), 'dd/mm/yyyy HH:mm:ss')), (PARSEDATETIME ((End_Time), 'dd/mm/yyyy HH:mm:ss')))) (select datediff (HOUR, (PARSEDATETIME ((Start_Time), 'dd/mm/yyyy HH:mm:ss')), (PARSEDATETIME ((End_Time), 'dd/mm/yyyy HH:mm:ss')))) Household Questionnaire identifiers of questionnaire Time Stamps questions in questionnaire Returns difference between the two timestamps in minutes (0-59) Returns difference between the two timestamps in hours (0-23) Questionnaire Table where questionnaire identifiers and time stamp question are located In the above example, we can note the enumerators have taken less than hour to complete the following interviews while the others have taken at least 2 hours. Cluster ID 1 and HHID 1 Cluster ID 6 and HHID 1 This information allows you to take further actions depending on the context of your project. 8. Examining Reference Data If you are building the questionnaire yourself, you would have a copy of your reference table in your chosen PC location. However, if you do not have access to the reference tables, for instance if you obtained the questionnaire file from a colleague, when you import the file into the Designer, they will be saved in the Surveybe_questionnaire_data folder. This folder can be found here C:\Users\WindowsUserName\surveybe_questionnaire_data where WindowsUserName is the username under which you are running the Surveybe Designer. Alternatively, you can use the Data browser to view all or a subset of information held in a reference table (Note: the Import tab in the Data tab only shows a preview of 5 rows). To view all information held in your reference table use: Select * From Materialized.ReferenceTable or 30 P a g e

32 Set Schema Materialized; and use Select * From ReferenceTable This will display all the rows from your reference table in the Result Panel. (Use the horizontal or vertical scroll bar to see all the rows and columns if required). Similarly, you can also use the Data Browser to query your reference table(s). For instance, in the example below, the SQL command has been used to view all the districts within Region 2, Arusha in the Reference Table TZWards. 31 P a g e

33 9. Testing SQL Performance The Data Browser gives the user an opportunity to test how long it takes to run an SQL statement. For performance testing, it s useful to have realistic data in the database. Also, please note that the PC where your Designer is held is likely to have a better technical specification that field units running the Implementer. Therefore, the time taken to run the SQLs on your PC might be considerably more on a slower tablet or netbook. The screenshot below shows how long it took to run the SQL statement. If you feel that the SQL is taking too long to run, you can either improve the SQL or re-design part of your questionnaire. Please find below some general advice on how to optimise the questionnaire design: Questionnaire Design Surveybe is designed to handle large and complex questionnaires. It is however true that the more complex a questionnaire the more resource it will need to run and on lower powered PCs this can have an impact on performance. The following should help: Multiple level of rosters can multiply the effect of small inefficiencies in the questionnaire design. Try to use rosters where the data requires it rather than simply to create grid structures in your questionnaire. A set of 12 questions on the Questionnaire Table will run faster than 4 rows of 3 questions in a roster. This may have no measurable impact on the Questionnaire Table of a small questionnaire, but if you are the asking questions of each pupil in each class in a school in a hierarchy of rosters the impact will be multiplied. 32 P a g e

34 Use sub-screens in rosters or use single view roster if there are too many columns (variables) on a screen, particularly in very large rosters, so the implementer can draw them on screen more quickly. Note: rows can only be added on standard view rosters. Try to use screen enablements instead of skips if you have large sections of your questionnaire that need to be enabled or disabled. Screen enablements on 10 separate top level screens will run faster than a single skip that covers several hundred questions on these screens. Try to keep any reference tables as small as possible, especially if they are referenced frequently (for example by enablements in low level rosters). Try to avoid using large numbers of single tick boxes that are subject to enablements as a lot of processing arises when they are enabled and disabled, use radio buttons instead. If the questionnaire is complex, you will particularly note an improvement in the time the (radio button) question takes to become enabled. 33 P a g e

35 A roster hierarchy can create many disabled questions. For example if you want to ask questions about the head of a household it s more efficient to do this on the Questionnaire Table than to have these questions disabled on the household members roster for each household member except the head. 34 P a g e

36 Don t refer to fields you don t need to in SQL as the SQL will fire each time the fields it refers to are changed. Implementer Loading time The loading time of the Implementer is affected by: The size, complexity and efficiency of the questionnaire currently loaded Whether or not there s an interview currently loaded the process will be potentially much slower if the Implementer was closed without returning to the dashboard as in addition to the normal start-up processing the current needs to be opened. Return to the dashboard before closing the Implementer. 35 P a g e

37 Appendix A: The Command Menu and Database Structure Browser The Command Menu You can use the Command Menu to control the Data Browser and execute the following functions: The Data Structure Browser The structure browser is used to view and navigate the materialized database structure. The database contains three schemas; ANSWER, INFORMATION_SCHEMA and MATERIALIZED Answer Schema The ANSWER schema is used by the Designer to process the data in imported interview files and to track which interviews have been imported. The Data Browser will not allow you to run queries against this table. Information_Schema INFORMATION_SCHEMA is an ANSI standard set of read-only views which provide information about all of the tables, views, columns, and procedures in a database. The Data Browser allows users to 36 P a g e

38 view the tables in the ANSWER schema, but not select data from them. The user may select from the tables in INFORMATION_SCHEMA, but will only see the rows relating to the read-only user role. Materialized Schema The MATERIALIZED schema is used to hold ALL tables that make up the data structure as defined by the Questionnaire. The Data Browser provides read only access to all of the tables in the MATERIALIZED schema. Expanding the MATERIALIZED schema will list the tables for a Questionnaire and expanding each table will show the columns for the table. There are 2 fixed tables in the MATERIALIZED schema; BLOBS and COMMENTS. The BLOBS (Binary Large Object) table is used to hold the binary data from file-based answers such as image questions. The COMMENTS table is used to hold comments made on questions, and also to store the alternative values entered for Other (Specify) answer options. Materialized Schema: Questionnaire Table Roster Table(s) Reference Tables Fixed Tables: Blobs & Comments Fixed Tables The Table FOODUNIT has been expanded to show all the columns it contains. When used in the SQL Editor, all table names normally need to be fully qualified. That is, use SELECT * FROM MATERIALIZED.Household rather than simply SELECT * FROM Household To avoid having to type in MATERIALIZED for every table you may issue the command SET SCHEMA MATERIALIZED; to force the browser to assume that all commands are running within the MATERIALIZED schema. Using SELECT * FROM TABLEA will then work. Alternative you can click and select the Table from the Materialized schema when writing the statement in the SQL Editor. 37 P a g e

39 When using the editor to issue multiple commands separate the commands with the standard SQL ; (semi-colon) character. 38 P a g e

40 Appendix B: The current. keyword and Other Specialist SQL Anytime a statement such as current.varname is used it is transformed by the Implementer into SELECT varname FROM tablecontainingvarname WHERE ID=? This syntax won t work outside the Implementer, so, when testing SQL containing the current. prefix in the Data browser a valid WHERE clause must be used to identify the relevant row. There are several options for this: The variable ID can be set to some fixed value you choose, based on what s actually in your tables You can specify the table identifier (or identifiers if you are looking at the questionnaire base table and the questionnaire has more than one identifier) of the chosen row You can use a SQL select statement to calculate a suitable ID Single Variable Dynamic Text In dynamic text SQL, the single-variable pattern [varname] is internally converted to [current.varname] and then transformed as described above. Dynamic Text Select Statements Any current.varname clauses are transformed as above and then the statement is executed. For example: SELECT current.name is current.age years old will be transformed into SELECT (SELECT name FROM MEMBERS WHERE ID=?) is (SELECT age FROM MEMBERS WHERE ID=?) years old Dynamic Question Row source Dynamic question row source is a standard SQL select statement that must return a column named ID that is of type BIGINT and a column named VALUE that is of type VARCHAR. Any current.varname clauses are transformed as described above and then the statement is executed. Boolean Expressions There are many places where the Questionnaire SQL takes the form of an expression that evaluates to true or false. These statements are transformed using a 2-step process. First, any current.varname clauses are transformed as described above. The transformed statement is then used to generate a CASE WHEN statement that returns 1 for true or 0 for false. SELECT CASE WHEN transformed Statement THEN 1 ELSE 0 END AS RESULT; So for example, the following validation statement current.age <> current.r1_age + 1 Is first transformed to 39 P a g e

41 SELECT age FROM MEMBERS WHERE ID=? <> SELECT R1_AGE FROM MEMBERS WHERE ID=? + 1 Note that the second clause is invalid when used as a stand-alone statement because of the + 1, although it is valid as a clause in the CASE WHEN. To run it stand alone it s necessary to prefix it with SELECT And is then wrapped in a CASE WHEN to be executed SELECT CASE WHEN SELECT age FROM MEMBERS WHERE ID=? <> SELECT R1_AGE FROM MEMBERS WHERE ID=? + 1 THEN 1 ELSE 0 END AS RESULT; When testing Booleans it s often helpful to break the statement down into the two sides of the Boolean and test those individually to see what results they give, before combining the two into the final statement. The screenshot below shows the result of testing the statement above, substituting the parameters with the fixed value 1. Notice the prefix of a SELECT when running the second clause as a stand-alone statement as per the comments above. 40 P a g e