Reproducibility and Extensibility in Scientific Research. Jessica Forde

Transcription

1 Reproducibility and Extensibility in Scientific Research Jessica Forde

2 Project Jupyter IPython Jupyter Notebook Architecture of JupyterHub Overview The problem of reproducibility in science Repo2docker Binder Extending research via interactive computing

3 Who are we?

4

5

6 ~2.1 Million Notebooks on GitHub jupyter-notebook? since=monthly

7

8

9 Project Jupyter Mission

10 We re not-for-profit

11 Jupyter Notebook

12 Rule et al., 2018

13 Carol Willing

14 More than Python

15

16

17

18 Yu Watanabe

19 jupyter/wiki/jupyter-kernels

20 jupyter/wiki/jupyter-kernels

21 More than Notebooks

22 HUB photo: xkcd, Jupyter, C. Willing

23 HUB JupyterHub provides a single user Jupyter Notebook server for each person in a group Similar to SWAN photo: xkcd, Jupyter, C. Willing

24 Hub: manages user accounts, authentication, and coordinates Single User Notebook Servers using a Spawner Proxy: routes HTTP requests to the Hub and Single User Notebook Servers. Spawner: starts a single-user notebook servers when a user logs in

25 Hub launches a proxy Proxy forwards all requests to Hub by default Hub handles login, and spawns single-user servers on demand Hub configures proxy to forward url prefixes to the single-user notebook servers

26 Kubernetes as Spawner

27 Kubernetes as Spawner

28 Scaling JupyterHub

29 Scaling JupyterHub

30 Scaling JupyterHub

31

32 Reproducibility

33 Atlantic Monthly, April

34 The more sophisticated science becomes, the harder it is to communicate results. Papers today are longer than ever and full of jargon and symbols. They depend on chains of computer programs that generate data, and clean up data, and plot data, and run statistical models on data. These programs tend to be both so sloppily written and so central to the results that it s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you ve actually discovered, clearly enough that someone else can discover it for themselves. - James Somers

35 Reproducibility: Producing similar results with the same data

36 Reproducibility: A software problem

37 GitHub Open Science Framework Technical Solutions CodaLab RunMyCode Research Journals

38 This week in xkcd

39 Reproducible scientific software pipelines Repository would contain: Data Dependencies Hyperparameters Scripts to run the jobs on similar hardware Analysis code

40 Reproducible scientific software pipelines Repository would contain: Data Dependencies No mention of OS or lower-level software Hyperparameters Scripts to run the jobs on similar hardware Analysis code

41 Docker for reproducible software source: Docker

42 Dockerfiles from GitHub Repos

43 Using repo2docker

44 repo2docker as reproducibility unit-test repo2docker looks for configuration files to determine how to build the docker image Typically describe dependencies and other instructions to create the environment configs in either root or folder called binder

45 Supported config files Dockerfile: full environment setup environment.yml: conda requirements.txt: pip REQUIRE: Julia apt.txt: Debian packages postbuild: custom install script runtime.txt: Python runtime

46 Example repo

47 Installing LaTeX

48 Installing LaTeX

49 Using binder folder and postbuild

50 Future: repo2docker with cloud services

51 repo2docker as reproducibility unit-test making an image from a repository requires: 1. version of the base docker image 2. version of repo2docker itself 3. versions of the libraries installed by the repository

52 repo2docker as reproducibility unit-test making an image from a repository requires: 1. version of the base docker image 2. version of repo2docker itself 3. versions of the libraries installed by the repository repo2docker deterministically controls 1 & 2 user controls 3

53 repo2docker as reproducibility unit-test Because repo2docker is deterministic, we can determine whether researchers have fully described the software environment used to load, analyze, and visualize their data Version numbers especially important If the configuration files don t create an environment that can run the code provided, the repository is not reproducible as currently configured

54 Reproducibility: Producing similar results with the same data

55 Author photo credit: xkcd

56 Author Other scientist photo credit: xkcd

57 Reproducibility: Producing similar results with the same data independently

58 repo2docker: Produces the same environment as the scientist

59 docker opens the doors to the laboratory itself

60 At a minimum, we should specify our environment

61 conda env export > environment.yml

62 pip freeze > requirements.txt

63 Docker for pods

64 JupyterHub, repo2docker as a service

65 Binder: Public repositories using JupyterHub and repo2docker

66

67

68

69 mybinder.org

70 mybinder.org

71 username/repo/branch

72 GitHub (GitLab and others also available) username/repo/branch

73 username/repo/branch? urlpath=lab

74

75

76 master?urlpath=tree/examples/notebooks/ pytorch_tests_onoff.ipynb

77 examples/notebooks/binderexample/statisticalanalysis.ipynb

78 JupyterHub

79 JupyterHub + repo2docker = binder photocredit: binder team

80

81 Create your own JupyterHub

82 Create your own BinderHub

83

84 The scope of the problem

85 The scope of the problem

86 We ed corresponding authors in our sample to request the data and code associated with their articles and attempted to replicate the findings from a randomly chosen subset of the articles for which we received artifacts. We estimate the artifact recovery rate to be 44% with a 95% bootstrap confidence interval of the proportion [0.36, 0.50], and we estimate the replication rate to be 26% with a 95% bootstrap confidence interval [0.20, 0.32]. - Stodden et al.

87 Moyna Baker, 2016 Nature (n=1,500)

88 Moyna Baker, 2016 Nature

89 GitHub repo s aren t enough

90 Report what you ve actually discovered, clearly enough that someone else can discover it for themselves. - James Somers

91 The maintainer s mind

92 The maintainer s mind

93 The scope of the problem

94 Scientific communication has become software engineering

95 Always code and comment in such a way that if someone a few notches junior picks up the code, they will take pleasure in reading and learning from it. - Code for the Maintainer

96 As scientists, we are now all becoming open source maintainers

97 As scientists, we are now all becoming open source maintainers Welcome to the club :D

98 Atlantic Monthly, April

99 Here s what s next

100

101 Create your own BinderHub

102

103 Show, don t tell

104

105

106

107 What if?

108

109 Contribute, get involved

110 Contribute, get involved GitHub: github.com/jupyterhub github.com/jzf2101 We read issues! We mark issues for new contributors as good first issue or help wanted github.com/jupyter/governance for code of conduct

111 Contribute, get involved Gitter gitter.im/juptyerhub/jupyterhub gitter.im/juptyerhub/binder gitter.im/juptyer/jupyter

112 Thanks to all the maintainers jupyter.org

113 And the contributors too! jupyter.org

114 Acknowledgements