Merge Conflicts p. 92 More GitHub Workflows: Forking and Pull Requests p. 97 Using Git to Make Life Easier: Working with Past Commits p.

Size: px

Start display at page:

Download "Merge Conflicts p. 92 More GitHub Workflows: Forking and Pull Requests p. 97 Using Git to Make Life Easier: Working with Past Commits p."

Byron White
5 years ago
Views:

1 Preface p. xiii Ideology: Data Skills for Robust and Reproducible Bioinformatics How to Learn Bioinformatics p. 1 Why Bioinformatics? Biology's Growing Data p. 1 Learning Data Skills to Learn Bioinformatics p. 4 New Challenges for Reproducible and Robust Research p. 5 Reproducible Research p. 6 Robust Research and the Golden Rule of Bioinformatics p. 8 Adopting Robust and Reproducible Practices Will Make Your Life Easier, Too p. 9 Recommendations for Robust Research p. 10 Pay Attention to Experimental Design p. 10 Write Code for Humans, Write Data for Computers p. 11 Let Your Computer Do the Work For You p. 12 Make Assertions and Be Loud, in Code and in Your Methods p. 12 Test Code, or Better Yet, Let Code Test Code p. 13 Use Existing Libraries Whenever Possible p. 14 Treat Data as Read-Only p. 14 Spend Time Developing Frequently Used Scripts into Tools p. 15 Let Data Prove That It's High Quality p. 15 Recommendations for Reproducible Research p. 16 Release Your Code and Data p. 16 Document Everything p. 16 Make Figures and Statistics the Results of Scripts p. 17 Use Code as Documentation p. 17 Continually Improving Your Bioinformatics Data Skills p. 17 Prerequisites: Essential Skills for Getting Started with a Bioinformatics Project Setting Up and Managing a Bioinformatics Project p. 21 Project Directories and Directory Structures p. 21 Project Documentation p. 24 Use Directories to Divide Up Your Project into Subprojects p. 26 Organizing Data to Automate File Processing Tasks p. 26 Markdown for Project Notebooks p. 31 Markdown Formatting Basics p. 31 Using Pandoc to Render Markdown to HTML p. 35 Remedial Unix Shell p. 37 Why Do We Use Unix in Bioinformatics? Modularity and the Unix Philosophy p. 37 Working with Streams and Redirection p. 41 Redirecting Standard Out to a File p. 41 Redirecting Standard Error p. 43 Using Standard Input Redirection p. 45 The Almighty Unix Pipe: Speed and Beauty in One p. 45

2 Pipes in Action: Creating Simple Programs with Grep and Pipes p. 47 Combining Pipes and Redirection p. 48 Even More Redirection: A tee in Your Pipe p. 49 Managing and Interacting with Processes p. 50 Background Processes p. 50 Killing Processes p. 51 Exit Status: How to Programmatically Tell Whether Your Command Worked p. 52 Command Substitution p. 54 Working with Remote Machines p. 57 Connecting to Remote Machines with SSH p. 57 Quick Authentication with SSH Keys p. 59 Maintaining Long-Running Jobs with nohup and tmux p. 61 Nohup p. 61 Working with Remote Machines Through Tmux p. 61 Installing and Configuring Tmux p. 62 Creating, Detaching, and Attaching Tmux Sessions p. 62 Working with Tmux Windows p. 64 Git for Scientists p. 67 Why Git Is Necessary in Bioinformatics Projects p. 68 Git Allows You to Keep Snapshots of Your Project p. 68 Git Helps You Keep Track of Important Changes to Code p. 69 Git Helps Keep Software Organized and Available After People Leave p. 69 Installing Git p. 70 Basic Git: Creating Repositories, Tracking Files, and Staging and Committing Changes p. 70 Git Setup: Telling Git Who You Are p. 70 Git init and git clone: Creating Repositories p. 70 Tracking Files in Git: git add and git status Part I p. 72 Staging Files in Git: git add and git status Part II p. 73 Git commit: Taking a Snapshot of Your Project p. 76 Seeing File Differences: git diff p. 77 Seeing Your Commit History: git log p. 79 Moving and Removing Files; git mv and git rm p. 80 Telling Git What to Ignore:.gitignore p. 81 Undoing a Stage: git reset p. 83 Collaborating with Git: Git Remotes, git push, and git pull p. 83 Creating a Shared Central Repository with GitHub p. 86 Authenticating with Git Remotes p. 87 Connecting with Git Remotes: git remote p. 87 Pushing Commits to a Remote Repository with git push p. 88 Pulling Commits from a Remote Repository with git pull p. 89 Working with Your Collaborators: Pushing and Pulling p. 90

3 Merge Conflicts p. 92 More GitHub Workflows: Forking and Pull Requests p. 97 Using Git to Make Life Easier: Working with Past Commits p. 97 Getting Files from the Past: git checkout p. 97 Stashing Your Changes: git stash p. 99 More git diff: Comparing Commits and Files p. 100 Undoing and Editing Commits: git commit -amend p. 102 Working with Branches p. 102 Creating and Working with Branches; git branch and git checkout p. 103 Merging Branches: git merge p. 105 Branches and Remotes p. 106 Continuing Your Git Education p. 108 Bioinformatics Data p. 109 Retrieving Bioinformatics Data p. 110 Downloading Data with wget and curl p. 110 Rsync and Secure Copy (scp) p. 113 Data Integrity p. 114 SHA and MD5 Checksums p. 115 Looking at Differences Between Data p. 116 Compressing Data and Working with Compressed Data p. 118 Gzip p. 119 Working with Gzipped Compressed Files p. 120 Case Study: Reproducibly Downloading Data p. 120 Practice: Bioinformatics Data Skills Unix Data Tools p. 125 Unix Data Tools and the Unix One-Liner Approach: Lessons from Programming Pearls p. 125 When to Use the Unix Pipeline Approach and How to Use It Safely p. 127 Inspecting and Manipulating Text Data with Unix Tools p. 128 Inspecting Data with Head and Tail p. 129 Less p. 131 Plain-Text Data Summary Information with wc, Is, and awk p. 134 Working with Column Data with cut and Columns p. 138 Formatting Tabular Data with column p. 139 The All-powerful Grep p. 140 Decoding Plain-Text Data: hexdump p. 145 Sorting Plain-Text Data with Sort p. 147 Finding Unique Values in Uniq p. 152 Join p. 155 Text Processing with Awk p. 157 Bioawk: An Awk for Biological Formats p. 163 Stream Editing with Sed p. 165

4 Advanced Shell Tricks p. 169 Subshells p. 169 Named Pipes and Process Substitution p. 171 The Unix Philosophy Revisited p. 173 A Rapid Introduction to the R Language p. 175 Getting Started with R and RStudio p. 176 R Language Basics p. 178 Simple Calculations in R, Calling Functions, and Getting Help in R p. 178 Variables and Assignment p. 182 Vectors, Vectorization, and Indexing p. 183 Working with and Visualizing Data in R p. 193 Loading Data into R p. 194 Exploring and Transforming Dataframes p. 199 Exploring Data Through Slicing and Dicing: Subsetting Dataframes p. 203 Exploring Data Visually with ggplot2 I: Scatterplots and Densities p. 207 Exploring Data Visually with ggplot2 II: Smoothing p. 213 Binning Data with cut() and Bar Plots with ggplot2 p. 215 Merging and Combining Data: Matching Vectors and Merging Dataframes p. 219 Using ggplot2 Facets p. 224 More R Data Structures: Lists p. 228 Writing and Applying Functions to Lists with lapply() and sapply() p. 231 Working with the Split-Apply-Combine Pattern p. 239 Exploring Dataframes with dplyr p. 243 Working with Strings p. 248 Developing Workflows with R Scripts p. 253 Control Flow: if, for, and while p. 253 Working with R Scripts p. 254 Workflows for Loading and Combining Multiple Files p. 257 Exporting Data p. 260 Further R Directions and Resources p. 261 Working with Range Data p. 263 A Crash Course in Genomic Ranges and Coordinate Systems p. 264 An Interactive Introduction to Range Data with GenomicRanges p. 269 Installing and Working with Bioconductor Packages p. 269 Storing Generic Ranges with IRanges p. 270 Basic Range Operations: Arithmetic, Transformations, and Set Operations p. 275 Finding Overlapping Ranges p. 281 Finding Nearest Ranges and Calculating Distance p. 290 Run Length Encoding and Views p. 292 Storing Genomic Ranges with GenomicRanges p. 299 Grouping Data with GRangesList p. 303

5 Working with Annotation Data: GenomicFeatures and rtracklayer p. 308 Retrieving Promoter Regions: Flank and Promoters p. 314 Retrieving Promoter Sequence: Connection GenomicRanges with Sequence Data p. 316 Getting Intergenic and Intronic Regions: Gaps, Reduce, and Setdiffs in Practice p. 319 Finding and Working with Overlapping Ranges p. 324 Calculating Coverage of GRanges Objects p. 328 Working with Ranges Data on the Command Line with BEDTools p. 329 Computing Overlaps with BEDTools Intersect p. 330 BEDTools Slop and Flank p. 333 Coverage with BEDTools p. 335 Other BEDTools Subcommands and pybedtools p. 336 Working with Sequence Data p. 339 The FASTA Format p. 339 The FASTQ Format p. 341 Nucleotide Codes p. 343 Base Qualities p. 344 Example: Inspecting and Trimming Low-Quality Bases p. 346 A FASTA/FASTQ Parsing Example-. Counting Nucleotides p. 349 Indexed FASTA Files p. 352 Working with Alignment Data p. 355 Getting to Know Alignment Formats: SAM and BAM p. 356 The SAM Header p. 356 The SAM Alignment Section p. 359 Bitwise Flags p. 360 CIGAR Strings p. 363 Mapping Qualities p. 365 Command-Line Tools for Working with Alignments in the SAM Format p. 365 Using samtools view to Convert between SAM and BAM p. 365 Samtools Sort and Index p. 367 Extracting and Filtering Alignments with samtools view p. 368 Visualizing Alignments with samtools tview and the Integrated Genomics Viewer p. 372 Pileups with samtools pileup, Variant Calling, and Base Alignment Quality p. 378 Creating Your Own SAM/BAM Processing Tools with Pysam p. 384 Opening BAM Files, Fetching Alignments from a Region, and Iterating Across Reads p. 384 Extracting SAM/BAM Header Information from an AlignmentFile Object p. 387 Working with AlignedSegment Objects p. 388 Writing a Program to Record Alignment Statistics p. 391 Additional Pysam Features and Other SAM/BAM APIs p. 394 Bioinformatics Shell Scripting, Writing Pipelines, and Parallelizing Tasks p. 395 Basic Bash Scripting p. 396 Writing and Running Robust Bash Scripts p. 396

6 Variables and Command Arguments p. 398 Conditionals in a Bash Script: if Statements p. 401 Processing Files with Bash Using for Loops and Globbing p. 405 Automating File-Processing with find and xargs p. 411 Using find and xargs p. 411 Finding Files with find p. 412 Find's Expressions p. 413 Find's -exec: Running Commands on finds Results p. 415 Xargs: A Unix Powertool p. 416 Using xargs with Replacement Strings to Apply Commands to Files p. 418 Xargs and Parallelization p. 419 Make and Makefiles: Another Option for Pipelines p. 421 Out-of-Memory Approaches: Tabix and SQLite p. 425 Fast Access to Indexed Tab-Delimited Files with BGZF and Tabix p. 425 Compressing Files for Tabix with Bgzip p. 426 Indexing Files with Tabix p. 427 Using Tabix p. 427 Introducing Relational Databases Through SQLite p. 428 When to Use Relational Databases in Bioinformatics p. 429 Installing SQLite p. 431 Exploring SQLite Databases with the Command-Line Interface p. 431 Querying Out Data: The Almighty SELECT Command p. 434 SQLite Functions p. 441 SQLite Aggregate Functions p. 442 Subqueries p. 447 Organizing Relational Databases and Joins p. 448 Writing to Databases p. 455 Dropping Tables and Deleting Databases p. 458 Interacting with SQLite from Python p. 459 Dumping Databases p. 465 Conclusion p. 467 Where to Go From Here? p. 468 Glossary p. 471 Bibliography p. 479 Index p. 483 Table of Contents provided by Blackwell's Book Services and R.R. Bowker. Used with permission.

Lecture 3. Essential skills for bioinformatics: Unix/Linux

Lecture 3. Essential skills for bioinformatics: Unix/Linux Lecture 3 Essential skills for bioinformatics: Unix/Linux RETRIEVING DATA Overview Whether downloading large sequencing datasets or accessing a web application hundreds of times to download specific files,