Bioinforma)cs Resources - Genbank -

Transcription

1 Bioinforma)cs Resources - Genbank - Lecture & Exercises Prof. B. Rost, Dr. L. Richter, J. Reeb Ins)tut für Informa)k I12

2 Preliminary Schedule April 13 th Intro, General Overview (1. sh.) April 20 th Sequence Databases (2. sh.) April 27 th Sequence Databases (3. sh.) May 4 th Structure Databases (4. sh.) May 11 th Lecture cancelled May 18 th SQL (5. sh.) May 25 th SQL, NoSql (6. sh) June 1 th Lecture cancelled June 8 th NoSql 2 (7.sh.) June 15 th MongoDB, JavaScript (8.sh.) June 22 nd Node.js Applications (9.sh.) June 29 th PredictProtein Jul 6 th Wrap Up, Q&A Jul 20 th Exam * These exercises can earn you a bonus

3 Na)onal Center for Biotechnology Informa)on, NCBI /07_19_2013/images/milestonesPic6.jpg first ideas in the middle of the 80s division of the Na)onal Library of Medicine (NLM) inside the Na)onal Ins)tutes of Health (NIH) poli)cal mission founded in 1988 David Lipman

4 NCBI s poli)cal mission as defined by the bill: 1. design, develop, implement, and manage automated systems for the collec)on, storage, retrieval, analysis, and dissemina)on of knowledge concerning human molecular biology, biochemistry, and gene)cs; 2. perform research into advanced methods of computer-based informa)on processing capable of represen)ng and analyzing the vast number of biologically important molecules and compounds; 3. enable persons engaged in biotechnology research and medical care to use systems developed under paragraph (1) and methods described in paragraph (2) ; and 4. coordinate, as much as is prac)cable, efforts to gather biotechnology informa)on on an interna)onal basis.

5 Selected NCBI Accomplishments Blast GenBank at NCBI Human Genome PubMed Central 1994 NCBI website 2003 Entrez Gene / DTDs Genomes OMIM PubMed NIH Public Access Genome Reference Consor)um 1000 Genomes Project

6 NCBI Resources NCBI currently hosts a vast bunch of resources hap:// grouped according to various criteria - meta data, project-centric - method oriented - topic oriented sorted in the sec)ons: databases, downloads, submissions, tools, howtos

7 Genbank s Origin features/innovations/images/light/thumbnails/ 21.jpg Walter Goad, Los Alamos Na)onal Laboratory Los Alamos Sequence Database 1979 Crea)on and release of GenBank in 1982 End of 1982: 2000 sequences Move to NCBI in 1992

8 Minutes from 20 th anniversary of GenBank in Among them is a memo on Los Alamos Na)onal Laboratory sta)onery dated May 9, 1980, that reads: Monday, May 12 at 10:30 Steve Simon invites you for cake and coffee to celebrate 100,000 bases now in the DNA sequence library. taken from haps:// genbank-turns-20

9 Growth of GenBank and WGS -doubling approx. every 18 months, diagram for release 225, Apr current version: release 225: 260,189,141,631 bases in Genbank, 2,784,740,996,536 bases in WGS -taken from hap://

10 Growth of GenBank and WGS -current release 225: 208,452,303 sequences in Genbank, 621,379,029 sequences in WGS -taken from hap:// release 225, Apr. 2018

11 References for GenBank one current cita)on source: GenBank. Nucleic Acids Res Jan; 42(Database issue):d32-7. doi: /nar/ gkt1030. Epub 2013 Nov 11. PMID: the most recent: Genbank. Nucleic Acids Res Jan 4; 46(D1): D41 D47. Published online 2017 Nov 13 th. doi: /nar/gkx1094 PMCID: PMC

12 References for GenBank more general for NCBI services: Database resources of the Na)onal Center for Biotechnology Informa)on. Nucleic Acids Res Jan 4; 44(Database issue): D7 D19. Published online 2015 Nov 28. doi: /nar/ gkv1290 part of the Interna)onal Nucleo)de Sequence Database Collabora)on (INSDC) together with EMBL Nucleo)de Sequence Database (EMBL- Bank), part of the European Nucleo)de Archive (ENA) and the DNA Data Bank of Japan (DDBJ)

13 Most Growing Divisions Division Description Release 197 (8/2013) Annual Increase (%) WGS* Whole-genome shotgun data 2,035,032,639,807 from Release 219 TSA* Transcriptome shotgun data 149,038,907,599 from Release 219 WGS* Whole-genome shotgun data TSA* Transcriptome shotgun data PHG Phages VRL Viruses BCT Bacteria ENV Environmental samples INV Invertebrates PAT Patented sequences PLN Plants GSS Genome survey sequences VRT Other vertebrates MAM Other mammals * not... distributed... with the release; there specific project... server sections... TOTAL All GenBank sequences

14 Top Organisms (Rel. 207) Organism Entries Non-WGS base pair Homo sapiens Mus musculus Rattus norvegicus Bos taurus Zea mays Sus scrofa Danio rerio Triticum aestivum Oryza sativa Japonica Group Arabidopsis thaliana

15 Top Organisms (Rel. 219) Organism Entries Non-WGS base pair Homo sapiens 24,231,652 18,893,466,733 Mus musculus 9,883,173 10,229,286,664 Rattus norvegicus 2,197,781 6,528,984,315 Bos taurus 2,229,235 5,429,379,063 Zea mays 4,197,803 5,227,077,026 Sus scrofa 3,298,802 5,071,347,463 Hordeum vulgare ssp. vulgare 1,346,798 3,235,834,212 Danio rerio 1,729,033 3,190,913,255 Ovis canadanensis canadanensis 72 2,590,574,434 Triticum aestivum 1,812,814 1,942,831, Oryza sativa Japonica Group 1,378,262 1,642,328, Escherichia coli 118,884 1,571,576,

16 Distribu)on of Sequence Files(Rel. 207) Division Number of Files BCT 178 CON 317 ENV 81 EST 478 HTG 142 INV 126 PAT 219 PLN 107 TSA 175 VRL 34 Release 207 consists of 2333 text files in total. Release 225 consists of 3120 text files in total.

17 Distribu)on of Sequence Files(Rel. 2019) Division Number of Files BCT 350 CON 359 ENV 97 EST 483 HTG INV 153 PAT 290 PHG 4 PLN 145 PRI 56 SYN 10 TSA 230 VRL 48 Release 219 consists of 2225 text files in total.

18 Database Files (Rel. 225) GenBank comes in a set of compressed text files available via FTP see kp://kp.ncbi.nih.gov/genbank/gbrel.txt 3120 ASCII files (listed in division plus addi)onal list files) in the range of MB uncompressed ~885 GB each file consists of two por)ons

19 Database Files Part 1: highly conserved database file headers GBBCT1.SEQ Genetic Sequence Data Bank April NCBI-GenBank Flat File Release Bacterial Sequences (Part 1) loci, bases, from reported sequences Part 1: sequence entries for that division described in the header

20 ! ! GBSMP.SEQ Genetic Sequence Data Bank! December !! GenBank Flat File Release 74.0!! Structural RNA Sequences!! 2 loci, 236 bases, from 2 reported sequences!! LOCUS AAURRA 118 bp ss-rrna RNA 16-JUN-1986! DEFINITION A.auricula-judae (mushroom) 5S ribosomal RNA.! ACCESSION K03160! VERSION K ! KEYWORDS 5S ribosomal RNA; ribosomal RNA.! SOURCE A.auricula-judae (mushroom) ribosomal RNA.! ORGANISM Auricularia auricula-judae! Eukaryota; Fungi; Eumycota; Basidiomycotina; Phragmobasidiomycetes;! Heterobasidiomycetidae; Auriculariales; Auriculariaceae.! REFERENCE 1 (bases 1 to 118)! AUTHORS Huysmans,E., Dams,E., Vandenberghe,A. and De Wachter,R.! TITLE The nucleotide sequences of the 5S rrnas of four mushrooms and! their use in studying the phylogenetic position of basidiomycetes! among the eukaryotes! JOURNAL Nucleic Acids Res. 11, (1983)! FEATURES Location/Qualifiers! rrna ! /note="5s ribosomal RNA"! BASE COUNT 27 a 34 c 34 g 23 t! ORIGIN 5' end of mature rrna.! 1 atccacggcc ataggactct gaaagcactg catcccgtcc gatctgcaaa gttaaccaga! ACCESSION M34766! 61 gtaccgccca gttagtacca cggtggggga ccacgcggga atcctgggtg ctgtggtt! VERSION M ! //! KEYWORDS 5S ribosomal RNA.!! SOURCE Acetobacter sp. (strain MB 58) rrna.! LOCUS ABCRRAA 118 bp ss-rrna RNA 15-SEP-1990! DEFINITION Acetobacter sp. (strain MB 58) 5S ribosomal RNA, complete sequence.! ORGANISM Acetobacter sp.! Prokaryotae; Gracilicutes; Scotobacteria; Aerobic rods and cocci;! Azotobacteraceae.! REFERENCE 1 (bases 1 to 118)! AUTHORS Bulygina,E.S., Galchenko,V.F., Govorukhina,N.I., Netrusov,A.I.,! Nikitin,D.I., Trotsenko,Y.A. and Chumakov,K.M.! TITLE Taxonomic studies of methylotrophic bacteria by 5S ribosomal RNA! sequencing! JOURNAL J. Gen. Microbiol. 136, (1990)! FEATURES Location/Qualifiers! rrna ! /note="5s ribosomal RNA"! BASE COUNT 27 a 40 c 32 g 17 t 2 others! ORIGIN! 1 gatctggtgg ccatggcggg agcaaatcag ccgatcccat cccgaactcg gccgtcaaat! 61 gccccagcgc ccatgatact ctgcctcaag gcacggaaaa gtcggtcgcc gccagayy! //! ! !

21 The GenBank Flat File Format a sequence entry consists of many records (lines) each record consists of two parts Part 1: columns 1-10 / Entry Field Name Part 2: remaining line with the content

22 Part 1/1 a keyword, beginning in column 1 of the record (e.g., REFERENCE is a keyword) a subkeyword beginning in column 3, with columns 1 and 2 blank (e.g., AUTHORS is a subkeyword of REFERENCE) or a subkeyword beginning in column 4, with columns 1, 2, and 3 blank (e.g., PUBMED is a subkeyword of REFERENCE)

23 Part 1/2 blank characters, indica)ng that this record is a con)nua)on of the informa)on under the keyword or subkeyword above it a code, beginning in column 6, indica)ng the nature of an entry (feature key) in the FEATURES table

24 Part 1/3 a number, ending in column 9 of the record: - This number occurs in the por)on of the entry describing the actual nucleo)de sequence and designates the numbering of sequence posi)ons two slashes (//) in posi)ons 1 and 2, marking the end of an entry

25 Part 2 The second part of each sequence entry record contains the informa)on appropriate to its keyword in posi)ons 13 to 80 for keywords in posi)ons 11 to 80 for the sequence

26 Entry Field Types (incomplete) Locus: A short mnemonic name for the entry, chosen to suggest the sequence's defini)on; mandatory keyword/exactly one record. Defini4on: A concise descrip)on of the sequence; mandatory keyword/one or more records Accession: - the primary accession number is a unique, unchanging iden4fier assigned to each GenBank sequence record. - to be used for cita)ons from GenBank - mandatory keyword/one or more records.

27 Entry Field Types (incomplete) Version: - compound iden)fier consis)ng of the primary accession number and a numeric version number associated with the current version of the sequence data in the record - op)onally followed by an integer iden)fier (a "GI") assigned to the sequence by NCBI - mandatory keyword/exactly one record

28 Entry Field Types (incomplete) DBLINK: provides cross-references to resources that support the existence a sequence record; op4onal keyword/one or more records Keywords: short phrases describing gene products and other informa)on about an entry; mandatory keyword in all annotated entries/one or more records

29 Entry Field Types (incomplete) Source: Common name of the organism or the name most frequently used in the literature; mandatory keyword in all annotated entries/one or more records/includes one subkeyword Organism: Formal scien)fic name of the organism (first line) and taxonomic classifica)on levels (second and subsequent lines); mandatory subkeyword in all annotated entries/two or more records

30 Entry Field Types (incomplete) Reference: - Cita)ons for all ar)cles containing data reported in this entry - includes seven subkeywords and may repeat - mandatory keyword/one or more records Journal: lists the journal name, volume, year, and page numbers of the cita)on; mandatory subkeyword/one or more records op)onal subkeywords: Authors, Consor)um, Title, Medline, Pubmed, Remark

31 Entry Field Types (incomplete) Features: table containing informa)on on por)ons of the sequence that code for proteins and RNA molecules; sites of biological significance; op4onal keyword/one or more records Origin: - specifica)on of how the first base of the reported sequence is opera)onally located within the genome - mandatory keyword/exactly one record - followed by sequence data (mul)ple records) //: entry termina)on symbol; mandatory at the end of an entry/exactly one record

32 Columns Detailed Locus Format Contents 'LOCUS' spaces Locus name space Length of sequence, right-justified space bp space spaces, ss- (single-stranded), ds- (double-stranded), or ms- (mixed-stranded) NA, DNA, RNA, trna (transfer RNA), rrna (ribosomal RNA), mrna (messenger RNA), urna (small nuclear RNA), left justified space 'linear' followed by two spaces, or 'circular' space The division code space Date, in the form dd-mmm-yyyy (e.g., 15-MAR-1991)

33 six or eight characters six character format: Accession Format - single uppercase leaer - 5 digits eigth character format: - two uppercase leaers - 6 digits primary accession number always the first one

34 Features (Incomplete) authorita)ve source: hap:// feature table contains informa)on about: - gene and gene products - regions of biological significance - can enumerate differences between various reports - provides cross-references to other data collec)ons - allows hierarchical rela)on between the features

35 Layout first line of the feature table is a header includes the keyword FEATURES and the column header Loca)on/Qualifiers each feature consists of: - descriptor line containing a feature key and a loca)on - a con)nua)on line for the loca)on may follow - feature qualifiers may follow the descriptor line - key: column 6-20, loca)on starts in column 22 - qualifiers on subsequent lines at column 22 star)ng with a /

36 A Few Frequent Features CDS: sequence coding for amino acids in protein (includes stop codon) exon: region that codes for part of spliced mrna gene: region that defines a func)onal gene, possibly including upstream (promotor, enhancer, etc) and downstream control elements, and for which a name has been assigned mrna: messenger RNA... > 60 features currently

37 Loca)on and Qualifiers Loca)on: - a loca)on can be: a single base, a span of bases, a site between two bases, a join of sequences,... - examples: 23, , 23^24, join (23..56, ) Qualifiers: - format: from column 22 /qualifier_name[=value] - types: free text, enumera)on or controlled vocabulary, cita)ons, sequences, feature labels

38 Database Cross References /db_xref hap:// db_xref/ Qualifier: /db_xref="database:idendfier Defini4on: database cross-reference: pointer to related informa)on in another database Scope: all feature keys Example: /db_xref="swiss-prot:p12345 currently > 120 databases available

39 Anatomy of a Genbank Flat File...

40 Anatomy of a Genbank Flat File Locus line...

41 Anatomy of a Genbank Flat File Accession Number, Version and GI number...

42 Anatomy of a Genbank Flat File Feature table with annotations...

43 Useful Resources from NCBI Materials: Electronic bookshelf hap:// factsheets/ kp://kp.ncbi.nih.gov/pub/factsheets/ Factsheet_Books.pdf NCBI manuals text books

44 Useful Resources from NCBI Processes, e.g. Prokaryo)c Genome Annota)on Pipeline designed for bacterial and archaeal genomes mul)-level process including protein-coding gene predic)on and func)onal genome unit like rrnas, trnas, small RNAs, pseudogenes control regions, repeats, inser)on elements a.s.f. combina)on of ab-inido predic)on and homology based methods

45 Useful Resources from NCBI reference databases: RefSeq hap:// comprehensive, integrated, non-redundant, wellannotated set of sequences, including genomic DNA, transcripts, and proteins stable reference for genome annota)on, esp. subset of RefSeqGene reference sequences reference coordinates accessible via BLAST, Entrez and FTP

46 RefSeq created by: - Eukaryo)c Genome Annota)on Pipeline - Prokaryo)c Genome Annota)on Pipeline - Manual cura)on - Submission to INSDC members reflect current knowledge of sequences data and biology format consistency Accession number contains an _

47 RefSeq Growth

48 Databases Accessible via Entrez

49 Computa)on: Blast at NCBI

50

51

52

53

54 Searching the NCBI / Entrez provide an integrated search interface to the different NCBI databases: Entrez Programming U)li)es (E-u)li)es) Base-URL: hap://eu)ls.ncbi.nlm.nih.gov/entrez/eu)ls/ > 40 databases stable interface of nine server-side programs hap://

55 Entrez Guidelines if you use the eu)ls against the guidelines you might be banned! >100 requests: weekends or outside US peak )mes (9pm-5am, EST) not more than 3 request per second provide and tool name: &tool=<...>& =<...>! registra)on with and tool name with NCBI may relax these restric)ons supported by BioPython

56 Construc)ng URLs parameter: &lowercasename excep)on: &WebEnv no required order null values and inappropriate parameter are generally ignored no spaces, use + instead use URL encodings for special character like: %22 for or %23 for # or %40

57 Einfo Esearch EPost ESummary EFetch ELink EGQuery ESpell ECitMatch E-u)li)es

58 External Interfaces to Entrez / API there are a number of APIs to access the various services from NCBI, described at: hap:// base URL: hap://eu)ls.ncbi.nlm.nih.gov/entrez/eu)ls/ basic searching: - esearch.fcgi?db=<database>&term=<query> - Input: Entrez database (&db); any Entrez text query (&term) - Output: List of UIDs matching the Entrez query

59 ESearch text search eu)ls.ncbi.nlm.nih.gov/entrez/eu)ls/esearch.fcgi responds to a text query with the list of matching UIDs in a given database (for later use in ESummary, EFetch or ELink), along with the term transla)ons of the query

60 ESummary document summary downloads eu)ls.ncbi.nlm.nih.gov/entrez/eu)ls/ esummary.fcgi responds to a list of UIDs from a given database with the corresponding document summaries

61 EGQuery global query eu)ls.ncbi.nlm.nih.gov/entrez/eu)ls/ egquery.fcgi responds to a text query with the number of records matching the query in each Entrez database

62 EInfo database sta)s)cs eu)ls.ncbi.nlm.nih.gov/entrez/eu)ls/einfo.fcgi provides the number of records indexed in each field of a given database, the date of the last update of the database, and the available links from the database to other Entrez databases without &db: lists all available databases

63 EFetch data record downloads eu)ls.ncbi.nlm.nih.gov/entrez/eu)ls/efetch.fcgi responds to a list of UIDs in a given database with the corresponding data records in a specified format

64 ELink Entrez links eu)ls.ncbi.nlm.nih.gov/entrez/eu)ls/elink.fcgi responds to a list of UIDs in a given database with either a list of related UIDs (and relevancy scores) in the same database or a list of linked UIDs in another Entrez database

65 ELink checks for the existence of a specified link from a list of one or more UIDs creates a hyperlink to the primary LinkOut provider for a specific UID and database, or lists LinkOut URLs and aaributes for mul)ple UIDs

66 EPost UID uploads eu)ls.ncbi.nlm.nih.gov/entrez/eu)ls/epost.fcgi accepts a list of UIDs from a given database, stores the set on the History Server, and responds with a query key and web environment for the uploaded dataset

67 ESpell spelling sugges)ons eu)ls.ncbi.nlm.nih.gov/entrez/eu)ls/espell.fcgi retrieves spelling sugges)ons for a text query in a given database

68 ECitMatch batch cita)on searching in PubMed eu)ls.ncbi.nlm.nih.gov/entrez/eu)ls/ ecitmatch.cgi retrieves PubMed IDs (PMIDs) corresponding to a set of input cita)on strings

69 Iden)ficators records are iden)fied by an integer ID called UID UID are database specific like GI numbers, PMIDS, MMDB-IDs UID are as well input and output especially useful in combina)on with the History server a full descrip)on of parameters and syntax can be found at: hap://

70 Selected UIDs Entrez Database UID common name E-utility Database Name Books Book ID books Conserved Domains PSSM-ID cdd dbvar dbvar ID dbvar EST GI number nucest Gene Gene ID gene Genome Genome ID genome MeSH MeSH ID mesh NCBI Web Site Web Site ID ncbisearch Nucleotide GI number nuccore PubMed PMID pubmed

71 Entrez Core Engine EGQuery, ESearch, and ESummary two tasks: - assemble a list of UIDs that match a text query (ESearch) - retrieve a brief summary record called a Document Summary (DocSum) for each UID ESummary) EGQuey: global version of ESearch esearch.fcgi?db=database&term=query esummary.fcgi?db=database&id=uid1,uid2,uid3,...! expanded into more complicated Entrez queries

72 Entrez Databases (EInfo, EFetch, and ELink) EInfo: - provides detailed informa)on about each database - including lists of the indexing fields in the database - available links to other Entrez databases

73 Entrez Databases (EInfo, EFetch, and ELink) added value to the raw data: - supports a variety of display formats: EFetch UID lists in XML and plain text (&retmode) for all databases, other formats (&rettype) are database specific - hap:// table/chapter4.t._valid_values_of retmode_and/? report=objectonly - efetch.fcgi?db=database&id=uid1,uid2,uid3 &rettype=report_type&retmode=data_mode!

74 Entrez Databases (EInfo, EFetch, and ELink) added value to the raw data: - links to records in other Entrez databases manifested as list of associated UIDs - UIDs must be valid in source database (&dbfrom) - elink.fcgi? dbfrom=protein&db=gene&id= ,

75 Entrez History Server simple: in the GUI accessible via the respec)ve tabs you can store temporarily sets of UIDs as input for later queries through other tools each list of UIDs is specified by: - &query_key (integer label) - &WebEnv (cookie string)

76 EPost: Crea)on of a stored UID list - EPost can be used upload a UID list - returns &query_key and &WebEnv! ESearch: - stores the results if given &usehistory=y! ELink: - stores the results if given &cmd=neighbor_history!

77 Usage of stored UID lists Use of stored lists: esummary.fcgi?db=database&webenv=webenv &query_key=key! one web environment can hold mul)ple result lists lists in the same web environment can be combined with AND, OR, NOT by default every call creates a new environment -> give &WebEnv in subsequent calls to store the lists in the same web environment

78 Sketching Pipelines get DocSummaries or entries for keywords or IDs: - ESearch -> ESummary/EFetch - EPost -> ESummary/EFetch filter/limit a record set: - EPost/ELink -> ESearch more advanced queries: - ESearch -> ELink -> ESummary/EFetch - EPost -> ELink -> ESearch -> EFetch

79 storing results: - esearch.fcgi? db=<database>&term=<query>&usehistory=y - input: any Entrez text query (&term); Entrez database (&db); &usehistory=y - output: web environment (&WebEnv) and query key (&query_key) parameters specifying the loca)on on the Entrez history server of the list of UIDs matching the Entrez query - example: hap://eu)ls.ncbi.nlm.nih.gov/entrez/eu)ls/ esearch.fcgi?db=pubmed&term=science%5bjournal %5d+AND+breast+cancer+AND+2008%5bpdat %5d&usehistory=y

80 Associa)ng Search Results with Exis)ng Search Results: - esearch.fcgi? db=<database>&term=<query1>&usehistory=y - esearch.fcgi? db=<database>&term=<query2>&usehistory=y&web Env=$web1 - Input: Any Entrez text query (&term); Entrez database (&db); &usehistory=y; Exis)ng web environment (&WebEnv) from a prior E-u)lity call - Output: Web environment (&WebEnv) and query key (&query_key) parameters specifying the loca)on on the Entrez history server of the list of UIDs matching the Entrez query

81 E-u)lity Webinar haps:// v=icfvvexp30o