Generic Model Organism Database. Lavanya Rishishwar

Size: px

Start display at page:

Download "Generic Model Organism Database. Lavanya Rishishwar"

Raymond Banks
5 years ago
Views:

1 Generic Model Organism Database Lavanya Rishishwar

2 Outline Purpose Genome database Basics of webserver & database GMOD 4/7/2016 Generic Model Organism Database 2

3 Presentation Assumption What do we understand: Sequencing and computational genomics process The output from different groups What we do not understand: The end goal Database (DBMS) and web service technologies 4/7/2016 Generic Model Organism Database 3

4 Outline Purpose Genome database Basics of webserver & database GMOD 4/7/2016 Generic Model Organism Database 4

5 Purpose By the time, functional annotation finishes their part, you will have a lot of data to deal with For a biologist, this is more information than they will ever care for It doesn t mean what you did was useless. The data now requires presentation. This is where you have to start like a real bioinformatician balancing the biology and computer science 4/7/2016 Generic Model Organism Database 5

6 Purpose What is needed: An intuitive platform that can help a biologist pose questions, inspect the data and make inferences Doesn t mean it has to be fancy But it does mean that it needs to have all the data in a neatly organized fashion that doesn t overwhelm the end user This is not 2007 where you do everything from scratch The field has matured a lot, explore before developing Do not reinvent the wheel 4/7/2016 Generic Model Organism Database 6

7 Purpose You want to create something that is useful Not something that is purely ornamental and no one will ever use it Whatever you create, should have a purpose What is the purpose here? What is your client (CDC/Dr. Xin Wang) looking for? How can you give the client exactly what they want and in a way that they will understand? 4/7/2016 Generic Model Organism Database 7

8 Outline Purpose Genome database Basics of webserver & database GMOD 4/7/2016 Generic Model Organism Database 8

9 Database Organized collection of data Can be flat files to more sophisticated systems What was wrong with flat files? Why did we require advanced systems? 4/7/2016 Generic Model Organism Database 9

10 Genome Database A database specializing in genomic information What is different from other databases: What is stored? Who uses it? Who do you query it? The natural relationship between the dataset Linkouts to existing databases Information presentation manner less text, more visual Q: Okay, so how do I make a database? 4/7/2016 Generic Model Organism Database 10

11 What do you require? Data (and knowledge of what that data is) A Database Management System (DBMS) A database developer/administrator A frontend designer 4/7/2016 Generic Model Organism Database 11

12 What do you require? Data (and knowledge of what that data is) Output from different groups A Database Management System (DBMS) MySQL hosted at compgenomics server A database developer/administrator Developer one of you; Administrator Troy Hilley A frontend designer One of you 4/7/2016 Generic Model Organism Database 12

13 Outline Purpose Genome database Basics of webserver & database GMOD 4/7/2016 Generic Model Organism Database 13

14 Database Management System (DBMS) Q: What is a DBMS? A: Collection of tools to help in creating, storing, modifying and extracting information from a database Q: Why not something like Excel? A: Issues with scalability, consistency, redundancy, simultaneous access and within data connectivity 4/7/2016 Generic Model Organism Database 14

15 Database Management System (DBMS) Scalability Spreadsheets/flat files store data in a single file. As data grows, basic operations becomes unviable. Consistency The data needs to be in the same format for pattern searching. Difficult to achieve in spreadsheets, not possible in flat files. 4/7/2016 Generic Model Organism Database 15

16 Database Management System (DBMS) Redundancy Redundant information uselessly increases data size and may interfere with pattern searching. Simultaneous access Limited simultaneous access in spreadsheets/flat files. 4/7/2016 Generic Model Organism Database 16

17 Database Management System (DBMS) Within data connectivity Difficult to maintain in spreadsheets and flat files. E.g. hypothetical database for outbreaks. Three types of data: Strain information Hospital information CDC personnel information If any data requires update, every single one needs to be updated! 4/7/2016 Generic Model Organism Database 17

18 Database Management System (DBMS) DBMS was specifically designed to resolve these issues Q: What are my options for a DBMS? A: Many! Relational DBMS: MySQL, Oracle, Microsoft Access, Postgre SQL Non-relational: MongoDB, CouchDB, Google Spanner 4/7/2016 Generic Model Organism Database 18

Relational DBMS Figure from http://www.ibm.

19 Relational DBMS Figure from 4/7/2016 Generic Model Organism Database 19

20 Relational DBMS Are defined in Structured Query Language (SQL) Looks like this: CREATE TABLE STATION (ID INTEGER PRIMARY KEY, CITY CHAR(20), STATE CHAR(2), LAT_N REAL, LONG_W REAL); INSERT INTO STATION VALUES (13, 'Phoenix', 'AZ', 33, 112); Why do I point this out? Ideally, you will be implementing dynamic browsers that fires these queries 4/7/2016 Generic Model Organism Database 20

21 Database Basics DBMS (Backend) Graphical User Interface (GUI) 4/7/2016 Generic Model Organism Database 21

22 Webserver Basics Webserver any computer connected to the internet that provides some sort of service These services are provided through specific protocols. E.g. HTTP, FTP HTTP : Hypertext transfer protocol, FTP : File transfer protocol A special software on the server facilitates this communication (answers the request) HTTPD (HTTP Daemon) i.e., the web server Most widely used web server Apache This will be setup for you already. You will most probably require specific changes to the setup once you are ready with the browser 4/7/2016 Generic Model Organism Database 22

23 Security Two levels: Security at access to the server Responsible: Sys-admin and frontend designer/developers Security at access to the database Responsible: Database administrators How much should you care? Do not create a security hole in the server. This will not only open your browser to the attacks but will also open the whole Georgia Tech system to the attacks. What if you (un)intentionally create a security hole Troy has some fool-proofing in place. If you bypass it somehow, Georgia Tech s IT staff monitors the network constantly and they typically detect the attack within an hour and block it. And then they will find you. And they will make sure you never do such an act ever again. 4/7/2016 Generic Model Organism Database 23

24 Frontend Your normal webpage HTML Hypertext markup language Styling CSS (Cascading style sheets) More library functions Javascript/Jquery Fancy templates - bootstrap 4/7/2016 Generic Model Organism Database 24

25 Server-side scripting Special type of programming language Executes on and by the webserver. Can t run locally. E.g. PHP (originally: Personal homepage. Now: Hypertext preprocessor. Don t ask how.), JSP, Perl via CGI, R, Python I recommend PHP for you easy to pick, constructs very similar to Perl/Java/C, GMOD uses it, this is a class project 4/7/2016 Generic Model Organism Database 25

26 Outline Purpose Genome database Basics of webserver & database GMOD 4/7/2016 Generic Model Organism Database 26

27 Who is this crazy looking guy? William James (Jim) Kent. Know that name. Respect this guy. He is one of greatest, perhaps the greatest, bioinformatics programmers ever. He was deeply involved in the assembly of the public human genome project. Along with his PI and a cluster of 50 PCs, this guy raced with Celera (JCV; was using a most powerful civilian supercomputer) and finished the draft human genome 3 days before them. Bam! If you were in the fall class, you compiled the James Kent Source tree. Almost all his. He speaks nothing but the truth. 4/7/2016 Generic Model Organism Database 27

He knows what a genome browser should be Genome browsers facilitate genomic analysis by presenting alignment, experimental and annotation data in

28 He knows what a genome browser should be Genome browsers facilitate genomic analysis by presenting alignment, experimental and annotation data in the context of genomic DNA sequences. Melissa S Cline & James W Kent, 2009 Genome browsers aggregate data 4/7/2016 Generic Model Organism Database 28

29 The UCSC Genome Browser Clicking on any of these takes you to a page full of details CDKN2A 4/7/2016 Generic Model Organism Database 29

30 Tracks don t have to be genes They are many kinds of genomic information Homologous sequences Conservation Protein domains Signaling peptide CRISPR elements Regulatory elements RNAs Transcription factor binding sites Conservation of genomics sequences Extremely important in modern times are tracks displaying *-seq data 4/7/2016 Generic Model Organism Database 30

31 What s good about the UCSC GB? Arguably the most advanced genome browser, it is much more than a tool for looking at genomes It integrates a huge amount of data for each gene it displays. The UCSC also has a graphical front end for downloading from its huge backend database 4/7/2016 Generic Model Organism Database 31

32 This UCSC browser does so much more It hosts the ENCODE project, one of the largest, probably the largest, assemblies of functional genomic data. It let s you jump between orthologous regions in different genome: CDKN2A It s a massive, massive database backend of over 6500 tables. 4/7/2016 Generic Model Organism Database 32

33 So why aren t there dozens of UCSC Implementations? It s really, really, really hard to install. It s impossible to understand unless you ve tried to do it. The UCSC genome browser works so well for the genomes that it has because it is so very, very specialized for those genomes. Each track in the UCSC browser has been lovingly crafted. 4/7/2016 Generic Model Organism Database 33

34 Browser choices There are a number of choices out there for a genome browser There are really just 2 big ones: UCSC GMOD & GBrowse/JBrowse We already discussed why you don t use the UCSC browser for projects 4/7/2016 Generic Model Organism Database 34

35 GMOD Model Organism Database (MOD) groups realized this issue and developed the Generic Model Organism Database (GMOD) Generic => Yes, valid for any type of organism Model Organism => That is a lie Database => Core purpose 4/7/2016 Generic Model Organism Database 35

36 GMOD 4/7/2016 Generic Model Organism Database 36

37 GMOD GMOD is an open source software Has a number of tools that fall into three main components: Data Management Annotation Visualization Next set of slides discusses the different utilities in it 4/7/2016 Generic Model Organism Database 37

38 GMOD Data Management 4/7/2016 Generic Model Organism Database 38

39 GMOD Data Management 4/7/2016 Generic Model Organism Database 39

40 GMOD Data Management 4/7/2016 Generic Model Organism Database 40

41 GMOD Data Management 4/7/2016 Generic Model Organism Database 41

42 GMOD Data Management 4/7/2016 Generic Model Organism Database 42

43 GMOD Annotation 4/7/2016 Generic Model Organism Database 43

44 GMOD Annotation 4/7/2016 Generic Model Organism Database 44

45 GMOD Annotation 4/7/2016 Generic Model Organism Database 45

46 GMOD Annotation 4/7/2016 Generic Model Organism Database 46

47 GMOD Visualization 4/7/2016 Generic Model Organism Database 47

48 GMOD Visualization 4/7/2016 Generic Model Organism Database 48

49 GMOD Visualization 4/7/2016 Generic Model Organism Database 49

50 GMOD Visualization 4/7/2016 Generic Model Organism Database 50

51 Genome Browser 4/7/2016 Generic Model Organism Database 51

52 You can use colors for information This shows genes that we thought were horizontally transferred Darker genes had more programs that indicated them being horizontally transferred 4/7/2016 Generic Model Organism Database 52

53 You can also have specialized tracks We had a track of virulence factors in the first year Clicking on any of them took you to details for the gene, a link to VFDB, etc. 4/7/2016 Generic Model Organism Database 53

54 This goes beyond colors You can alter how tracks are show in other ways Add and remove tracks, change the link that appears over a feature in the genome. 4/7/2016 Generic Model Organism Database 54

55 You can do even more customization You can make a transcriptome browser. It doesn t have to be a genome Not super exciting from this view. Just the predicted coding region of an assembled contig (mrna) 4/7/2016 Generic Model Organism Database 55

56 All of this is in the conf 4/7/2016 Generic Model Organism Database 56

57 Configuration file (.conf)

58 Making a new Track ### TRACK CONFIGURATION ### [ExampleFeatures] feature = remark glyph = generic stranded = 1 bgcolor = orange height = 10 key = Example Features

59 Genome Browser The images are not created beforehand they are generated on the runtime The server does the processing and sends out images to the user 4/7/2016 Generic Model Organism Database 59

60 Genome Browser 1. The user types in the URL: gbrowse2012.biology.gatech.edu 4/7/2016 Generic Model Organism Database 60

61 Genome Browser 2. Browser interprets and sends the request to HTTP Server 4/7/2016 Generic Model Organism Database 61

62 Genome Browser 3. Web Server receives the request and serves the client, i.e., starts Gbrowse 4/7/2016 Generic Model Organism Database 62

63 Genome Browser 4. In case of success, relevant hypertexts and multimedia is generated by accessing the database 4/7/2016 Generic Model Organism Database 63

64 Genome Browser 4/7/2016 Generic Model Organism Database 64

65 Genome Browser 5. The output traverses the same path back 4/7/2016 Generic Model Organism Database 65

66 Genome Browser 6. The whole process repeats again when the user interacts with the browser 4/7/2016 Generic Model Organism Database 66

67 GBrowse & JBrowse The previously discussed process is GBrowse s implementation JBrowse is another implementation of the same browser but in Javascript Pros: Reduced server load, smoother UI, better images Cons: Still lacks some of the features of GBrowse 4/7/2016 Generic Model Organism Database 67

68 Populating a Genome Database Most of the genome browsers (including GBrowse/JBrowse) expect input files in a specific format known as GFF GFF => General Feature Format Text based, table like format Capable of storing any type of information 4/7/2016 Generic Model Organism Database 68

General Feature Format (GFF) Format description: GFF General Feature Format is a tab separated format used for describing different omics features The

69 General Feature Format (GFF) Format description: GFF General Feature Format is a tab separated format used for describing different omics features The format has 9 mandatory columns as shown below: Seqname Source Feature Start End Score Strand Frame Attribute 4/7/2016 Generic Model Organism Database 69

70 Conclusion of GBrowse/JBrowse Will it ever be the greatest genome browser? No. That will always be the UCSC browser Will it remain the easiest to install for some time? Probably JBrowse seems more promising though Will you get the best return on time spent Yep 4/7/2016 Generic Model Organism Database 70

71 Conclusion of the talk A database is an efficient and organized method of storing data It allows faster mining patterns from the data Two component system: backend storage and frontend interface Most widely used option: GMOD Web based genome browser options: GBrowse/JBrowse, UCSC 4/7/2016 Generic Model Organism Database 71

72 Questions? 4/7/2016 Generic Model Organism Database 72

Genome Browser. Background and Strategy

Genome Browser. Background and Strategy Genome Browser Background and Strategy Contents What is a genome browser? Purpose of a genome browser Examples Structure Extra Features Contents What is a genome browser? Purpose of a genome browser Examples