1. HPC & I/O 2. BioPerl

Transcription

1 1. HPC & I/O 2. BioPerl

2

3 A simplified picture of the system User machines Login server(s) jhpce01.jhsph.edu jhpce02.jhsph.edu 72 nodes ~3000 cores compute farm direct attached storage Research network Gbps LAN Gbs Mbs data transfer server transfer01.jhsph.edu.!..!. Ethernet switches (10Gpbs 40Gps) DCS01 DCS02 DCS03 DCL01 /users < NFS exported file systems -----> Lustre file system

4 Review of technical notions n Central Processing Unit (CPU) n The part of the computer that executes instructions (programs) n Random Access Memory (RAM) n Very fast volatile memory that is used like a scratchpad by the cpu n Mass Storage (Disk) n Where data & apps are kept more or less permanently. Very slow compared to RAM n Network (ethernet, internet) n Computers and devices communicate over networks. n These days it s mostly ethernet.

5 Review of sizes n Storage and memory sizes n 1 Byte = 8 bits = 1 character n 1 megabyte (GB) = 10 6 bytes n 1 gigabyte (GB) = 1000 MB = 10 9 bytes n 1 terabyte (TB) = 1000 GB = bytes n 1 petabyte (PB) = 1000 TB = bytes n Typical sizes n USB stick GB n laptop disk drive GB n Enterprise Storage Appliance 100B n Scale-out cluster storage > 1PB

6 key technical notions (Network) n Bandwidth n Latency n How much data per second can you pump through a pipe. n measured in Gigabits per second (Gbps). n How long does it take for that first piece of data to get through? n measure in nano, micro or (gasp!) milliseconds n A practical demonstration n

7 Time scales for data transfer Latency Comparison Numbers L1 cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns 14x L1 cache Mutex lock/unlock 25 ns Main memory reference 100 ns 20x L2 cache, 200x L1 cache Compress 1K bytes with Zippy 3,000 ns Send 1K bytes over 1 Gbps network 10,000 ns 0.01 ms Read 4K randomly from SSD* 150,000 ns 0.15 ms Read 1 MB sequentially from memory 250,000 ns 0.25 ms Round trip within same datacenter 500,000 ns 0.5 ms Read 1 MB sequentially from SSD* 1,000,000 ns 1 ms 4X memory Disk seek 10,000,000 ns 10 ms 20x datacenter roundtrip Read 1 MB sequentially from disk 20,000,000 ns 20 ms 80x memory, 20X SSD Send packet CA->Netherlands->CA 150,000,000 ns 150 ms

8 What if we multiply all the time scales by 1 billion to humanize them? Main memory reference 1.6 min Brushing your teeth Send 2KB over 1 Gbps network 5.5 hr From lunch to end of work day Read 1 MB sequentially from memory 2.9 days A long weekend Round trip within same datacenter 5.8 days A medium vacation Reading 1MB from SSD SSD random read 1.7 days A normal weekend SSD read 1 MB sequentially 11.6 days Waiting for almost 2 weeks for a delivery Reading 1MB from Disk Seek 16.5 weeks A semester in university Read 1 MB sequentially from disk 7.8 months Almost producing a new human being Total time: 1 year Internet packet Round trip from California to Netherlands 4.8 years Average time it takes to complete a bachelor's degree

9 The main lesson n Do as much computing as you can in RAM n Avoid disk i/o as much as possible n If you must go to the disk suck in entire files at a time rather than fetching one record at a time.

10 BioPerl on the Cluster

11 BioPerl n Bioperl provides object-oriented software modules for many of the typical tasks of bioinformatics programming. n Manipulating individual sequences n Accessing genomic data directly from databases n Transforming formats of database/ file records n Searching for ``similar'' sequences n Creating and manipulating sequence alignments n Searching for genes and other structures in DNA n Developing machine readable sequence annotations

12

13 Bioperl Module Groups

14 BioPerl IO modules SeqIO SearchIO AlignIO TreeIO MapIO Matrix::IO Assembly::IO Ontology::IO More FASTA, GenBank, EMBL, etc. BLAST, FASTA, HMMER ClustalW, Phylip, MSF, etc. Newick, Nexus, NHX MapMaker Scoring, Phylip Ace, Phrap InterPro, GO, SO

15 Bio::SeqIO n The principal class for input/output n methods n new -- opens a new seqstream for I/O n next_seq -- gets the next entry in the input seqstream n write_seq -- writes to a seqstream n there is more n Refer to the web site for documentation n Example: format conversion

16 UniProtKB/SwissPro format Each sequence entry is composed of lines. Each line begins with a two-character code, which indicates the type of data contained in the line ID AC DT DE GN OS OG OC RN RP RC - Identification. - Accession number(s). - Date. - Description. - Gene name(s). - Organism species. - Organelle. - Organism classification. - Reference number. - Reference position. - Reference comments. RX - Reference cross-references. RA - Reference authors. RL - Reference location. CC - Comments or notes. DR - Database cross-references. KW - Keywords. FT - Feature table data. SQ - Sequence header. - (blanks) sequence data. // - Termination line.

17 swisspro to fasta format conversion n swiss2fasta.pl #!/usr/bin/perl -w use strict; use Bio::SeqIO; # create a SeqIO object for the input stream my $in = Bio::SeqIO->new('-file' => "sprot.txt", '-format' => 'swiss ); # create a SeqIO object for the input stream my $out = Bio::SeqIO->new('-file' => ">sprot.fasta", '-format' => 'fasta ); # read the the input stream and write to the output stream # one record at a time while ( my $seq = $in->next_seq() ) { $out->write_seq($seq); }

18 Example: Remote database query #!/usr/bin/perl -w use strict; use Bio::DB::GenBank; my ($gb, $seq1, $seq2, $seq_id); # use eval to test for success of code block eval { $gb = new Bio::DB::GenBank() }; if ($@) { die "Warning: Couldn't connect to Genbank";} # get by sequence id $seq1 = $gb->get_seq_by_id('musighba1'); $seq_id = $seq1->display_id(); print "got seq1 display id is $seq_id \n"; # get by accession number $seq2 = $gb->get_seq_by_acc('af303112'); $seq_id = $seq2->display_id(); print "got seq2 display id is $seq_id \n"; # get a bunch of sequences by accession number my $seqio = $gb->get_stream_by_id([ qw( J00522 AF303112)]); while( my $seqobj = $seqio->next_seq()) { print $seqobj->display_id(),"\n"; print $seqobj->seq()."\n\n"; }