Managing & Accessing Data Effectively with Databases

Transcription

1 Managing & Accessing Data Effectively with Databases RCS 2017 Brown Bag Series Bob Freeman, PhD Director, Research TechOps HBS 12 April,

2 Overview RCS Services & whoami Benefits and drawbacks of databases (DBs) Use cases Examples Using DBs via external programs 2

3 RCS Services Statistical Support Technology Training Support Research Infrastructure Support Methodological Consulting Use of IT for Research Consulting Research Grid Access, Training, Management, and Support Consultation on Revise and Resubmit Requests Statistical Programming Data Collection, Cleaning, Parsing, Transfer, and Matching HBS Compute Grid, Storage, & FASRC Odyssey Training Statistics Package Training Programming Package Training Database System Training License Negotiation, Installation, Training, and Help-Desk Support for HBS Research Software Management of HBS Researchers on Odyssey Supercomputing Cluster and Amazon Web Services Database Programming and Administration Advocacy, Researching, and Testing of New Methods and Approaches Visualization Training Data Security and Usage Agreement Support 3

4 Benefits & Drawbacks of DBs 4

5 Types of DBs? Three major types: relational, hierarchical, & network Most common: relational DB management systems 5

6 Relational DBs Data is stored in tables Columns (also known as fields) describe the data Rows (also known as records) contain the data 6

7 Relational DBs 7

8 Relational DBs 8

9 Why Use a DB? DBs were designed for storing and retrieving data Include powerful analysis tools Can handle large & complex datasets Have evolved to handle disparate data types: free-form (unstructured) text, images, Keeps act of storing data separate from analysis Can enforce quality control (constraints) during data entry (e.g. ensure dates are proper & realistic) Offload work from front-end program (e.g. R or Stata) to the DB Ask the DB to bring in only the data that you need. I.e. No need to load 1000-column wide data file 9

10 Relational DBs 10

11 Why Not Use a DB? Small data, simple solution (more than you need) Learning curve for SQL, but can be rather small* Data is not appropriate type or shape "It's a black box!" or "My data is trapped!" *Depends on how you wish to engineer things! 11

12 Use Cases 12

13 Relational DBs 13

14 Research Inbox Metrics Data from FY16 and Q1-Q2 FY17 14

15 Using DBs for a Data-driven Workflow Goal: Contrast group sharing promotional materials against those working solo to see if there is a difference in sales Data: 20 franchises with approximately 4 6 sales reps/franchise Approach: Create a mobile-enabled, web-based sharing system, tracking movements through uploading, viewing, and sharing. In 2 4 weeks! Tech: FileMaker Pro, FileMaker Pro Server, and FileMaker WebDirect (no web programming!) 15

16 DB Use Case #2 16

17 Examples 17

18 DB Use Examples FileMaker Pro SQLite MariaDB 18

19 Using DBs via External Programs 19

20 Simple GUI Add-ons Navicat, MySQL Workbench, etc Give you GUI controls in addition to SQL commands Can be very powerful 20

21 Access the DB via Direct Connections Direct connections is one method: Go through company's DB driver (connector, library) You work/program in your environment of choice, as long as it can connect to the driver/library Each DB requires its own driver 21

22 Access the DB via ODBC Open Database Connectivity is the other method! Modular, generic approach Uses a broker (manager) to hide DBspecific commands Your program can use generic SQL Each DB requires its own driver between ODBC Manager and the DB Great Windows support; Easier said than done on Mac 22

23 Thoughts Implicit assumption: work is done outside of the DB Demo is for local machine (localhost), but there's no reason why you cannot specify: host=researchgrid.hbs.edu ^^ coming very soon to a compute grid near you 23

24 PseudoCode Example 01: load necessary libraries for database 02: create and open the database connection 03: initialize the pointer (cursor) into the database system 04: execute the SQL query 05: fetch the query results 06: iterate over the data and print the results 07: close the database connection 24

25 ODBC Use Examples Attempting the Impossible!! FileMaker Pro SQLite MariaDB 25

26 Research Computing Services Please talk to your peers, and We wish you success in your research!