How to version control like a pro: a roadmap to your reproducible & collaborative research

Transcription

1 How to version control like a pro: a roadmap to your reproducible & collaborative research The material in this tutorial is inspired by & adapted from the Software Carpentry lesson on version control & Dany Vohl s tutorial from the ANITA astroinformatics school in Depending on how much we can work through today you can watch Dany s git tutorial on the ADACS LMS CC BY-SA 4.0 Slides -

2 HELLO my name is Rebecca Slides - CC BY-SA RStudio

3 HELLO my name is Kelle Slides -

4 Polling results from HWSA presentation, 23 June 2018, 87 responses total (duplicates due to slow internet?) Note poll is no longer active

5 The Problem Slides -

6 Why Version Control? As astronomers, we spend much of our time writing code, whether it be a simulation code or an observational reduction pipeline. As such, our codes are often constantly evolving. By putting all of our code under version control we can: tag code versions for later reference (via tags). record a unique identifier for the exact code version used to produce a particular plot or result (via commit identifiers). roll back our code to previous states (via checkout). identify when/how bugs were introduced (via diff/blame). keep multiple versions of the same code in sync with each other (via branches/merging). efficiently share and collaborate on our codes with others (via remotes/online hosting).

7 Why Version Control? It's important to also realise that many of the advantages of version control are not limited to just managing code. For example, it can also be useful when writing papers. Here we can use version control to: bring back that paragraph we accidentally deleted last week. try out a different structure and simply disregard it if we don't like it. concurrently work on a paper with a collaborator and then automatically merge all of our changes together. The upshot is you should use version control for almost everything. The benefits are well worth it...

8 Introducing Git In this tutorial we will be using Git for version control. Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. Git is easy to learn and has a tiny footprint with lightning fast performance. It outclasses SCM tools like Subversion, CVS, Perforce, and ClearCase with features like cheap local branching, convenient staging areas, and multiple workflows.git website Distributed -> everyone has their own complete copy of the entire repository and can make changes as they like Commiting to a central repository can be done once happy with the changes As opposed to Suberversion access to the central repo is not required to make changes Git is fast (primarily written in C & shell script) and lightweight (as you only track changes) Written by Linus Torvalds (creator of Linux)

9 Why GitHub Slides - Version Control Visible code and reproducibility Open code and reuse Collaborative code development Open code development

10 Set Up Open your bash terminal (git bash for Windows) Using git bash (on windows) you will already have git installed On Linux and Mac try: git --version (if not installed you will get an error) Installation instructions at:

11 The Premise

12 Getting started with Git Git commands are written as git verb, where verb is what we actually want to do: $ git config --global user.name "Vlad Dracula" $ git config --global user. "vlad@tran.sylvan.ia" The flag --global tells Git to use the settings for every project, in your user account, on this computer. You can check your settings at any time: $ git config --list

13 Getting started with Git

14 Creating a repository First, let s create a directory in Desktop folder for our work and then move into that directory: $ cd ~/Desktop $ mkdir planets $ cd planets Then we tell Git to make planets a repository a place where Git can store versions of our files: $ git init To check if everything was successful: $ ls -a You should see a.git folder. Warning: Unless you are familiar with Git it is generally best to avoid touching the.git folder or it s contents. We can check that everything is set up correctly by asking Git to tell us the status of our project: $ git status

15 Tracking changes - creating new files Let s create a file called mars.txt that contains some notes about the Red Planet s suitability as a base. I ll use nano to edit the file; you can use whatever editor you like. $ nano mars.txt Type the text below into the mars.txt file, save and close it: Cold and dry, but everything is my favorite color To check all was saved correctly: $ cat mars.txt If we check the status of our project again, Git tells us that it s noticed the new file: $ git status

16 Tracking changes - adding files We can add our file to the staging area: $ git add mars.txt You can add more than one file by listing the ones you want to add You can also use -A flag to add all files to the staging area Note: all files staged before a commit will have the same commit message If we check the status of our project again, Git tells us that it s noticed the new file: $ git status

17 Tracking changes - commiting files To add our changes to the repository we need to run one more command: $ git commit -m "Start notes on Mars as a base" Git is always a two-step process to track changes: $ git add [filename] $ git commit -m "brief statement (<50 char) about the changes"

18 Tracking changes - your go Edit the mars.txt file so that it reads: Cold and dry, but everything is my favorite color The two moons may be a problem for Wolfman But the Mummy will appreciate the lack of humidity then add and commit it to your repository

19 Tracking changes - your go $ nano mars.txt [add text] > The two moons may be a problem for Wolfman > But the Mummy will appreciate the lack of humidity [save and exit] [add file and commit changes] $ git add mars.txt $ git commit -m adding consideration about base suitability for team

20 Other useful commands If we forgot to stage a file for the last commit, or we had a typo in the commit message $ git commit --amend To delete a file in your repository and file system (this will also stage this deletion action for your next commit) $ git rm [filename] To stop tracking a file in the repository without actually deleting it from the file system $ git rm --cached [filename]

21 Recap $ git init initiate the repository $ git status check the current state of the repository $ git add [filename] stage a file / changes made to a file $ git commit -m useful message commit staged files / changes $ git commit --amend amend last commit $ git rm [filename] delete a file and stage the change $ git rm --cached [filename] remove a file from tracking

22 Ignoring stuff Say we have intermittent output files from our program that we do not want to track, as they a) b) change frequently, and can easily be reproduced tracking them would waste disk space $ mkdir results $ touch a.dat b.dat c.dat results/a.out results/b.out To ignore files in your repository create a.gitignore file listing all files to skip: $ nano.gitignore > *.dat > results/ [save and exit] Don t forget to track your.gitignore file! $ git add.gitignore $ git commit -m "Ignore data files and the results folder."

23 Exploring our changes If we want to know what we ve done recently, we can ask Git to show us the project s history: $ git log git log lists: all commits made to a repository in reverse chronological order. the commit s full identifier, the commit s author, when it was created, and the log message Git was given when the commit was created.

24 Exploring our changes You can see the changes made to an unstaged file compared to the last commit with $ git diff [filename] Other variations of this: $ git diff --staged comparing the staged file(s) to the last commit $ git diff [commit] [commit] comparing two commits using unique identifiers $ git diff HEAD~1 diff --git a/mars.txt b/mars.txt index df0654a..315bf3a a/mars.txt Cold and dry, but everything is my favorite color +The two moons may be a problem for Wolfman

25 An ill considered change Let s start a new file on Venus as a base. $ nano venus.txt Add some info on Venus and save the file. Next we will also add a new line to our mars.txt: $ nano mars.txt > An ill considered change Save and exit. Now lets stage and commit those files $ git add -A $ git commit -m started notes on Venus

26 An ill considered change Let s check everything is up to date and committed $ git status We have realised that our changes to mars.txt were incorrect, so let s roll them back We want to roll back to the commit before the last, so let s find out its identifier $ git log --oneline To roll back our changes we will checkout the mars.txt file from HEAD~1: $ git checkout HEAD~1 mars.txt Check that the file has changed as expected $ cat mars.txt or $ git diff mars.txt Note, mars.txt has been modified so you will need to stage and commit it again! $ git add mars.txt $ git commit -m reverted changes back to commit [number]

27 Recap $.gitignore files and folders mentioned in here will not be tracked $ git log show a log of the commit history $ git log --oneline show the commit history with one line per commit $ git diff [filename] compare current (unstaged) file to latest commit $ git diff --staged comparing the staged file(s) to the last commit $ git diff [commit] [commit] comparing two commits using unique identifiers $ git checkout [commit] [file] roll (specified file) back to specific commit $ HEAD denotes the latest commit $ HEAD~i denotes the ith commit before the latest commit

28 Remote repositories on GitHub Let s start by sharing the changes we ve made to our current project with the world. Log in to GitHub, create a new repository called planets

29 Remote repositories on GitHub

30 Remote repositories on GitHub Our remote is still empty, so let s populate it Copy the URL and go back to your terminal $ git remote add origin We can check the remote was set up correctly with $ git remote -v

31 Remote repositories on GitHub The name origin is a local nickname for your remote repository. To push your local changes to GitHub use: $ git push origin master Note: the -u option is synonymous with the --set-upstream-to option for the git branch command, and is used to associate the current branch with a remote branch so that the git pull command can be used without any arguments. We can pull changes from the remote repository to the local one as well: $ git pull origin master

32 Recap $ git remote add origin [URL] link an empty remote repo to your local repo $ git push origin master push your local changes to the remote repo $ git pull origin master pull changes from the remote repo $ git remote -v show nickname and url of the associated remote repo $ git clone [URL] [location] clone a remote repository to your computer $ git push -u origin master push your local changes to the remote repo and set specified remote as you upstream think of it as setting up the default so now you can update without specifying the remote and branch using $ git push or $ git pull

33 Useful links Branching interactive tutorial Collaboration and Conflicts Git/Github Cheatsheets

34 What future training would you like to see? Please fill in the survey below (3 short questions). Feel free to share with others at your institute, the survey will be open for 2 weeks until 8 July