Scientific Filesystem. Vanessa Sochat

Transcription

1 Scientific Filesystem Vanessa Sochat 1

2 Welcome! 2

3 Welcome! 3

4

5 Let s all be friends! 5

6 Once upon a time There was a scientist

7

8

9

10 Foo Bar foo bar

11

12 e c n e sci + 12

13 Pipeline foobar foo bar?

14 Pipeline foobar foo bar

15 Pipeline foobar foo bar

16 Pipeline foobar foo bar reproducible

17 Give it a try?

18 foo bar

19 ENTRYPOINT foo bar

20 What about the human component of reproducibility? 20

21 metadata foo bar content

22 control metadata foo bar content

23 ENTRYPOINT control Entrypoint Environment Metadata metadata foo bar content Software Libraries Scripts

24 Where are foo and bar? 24

25 ENTRYPOINT control metadata foo bar content

26 ENTRYPOINT control metadata foo bar exec./foobar content

27 ENTRYPOINT control metadata foo bar exec./foobar./bar./foo content

28 ENTRYPOINT control metadata foo bar exec./foobar./bar./foo content fo o bar

29 I just want to run bar. 29

30 Solution 1: More containers foo bar

31 Solution 1: More containers foo bar

32 Solution 2: Custom entrypoint If foo: exec foo else If bar: exec bar else: exec foobar

33 Solution 2: Custom entrypoint If foo: exec foo else If bar: exec bar else: exec foobar BESTAPP=FOO BESTAPP=BAR EXPORT BESTAPP

34 fo bar o

35 The Linux Filesystem Hierarchy was not optimized for scientific applications. 35

36 control foo

37 Linux Filesystem Hierarchy

38 control foo metadata bar metadata foo bar global content

39 Let s make a Scientific Filesystem! 39

40 What is SCIF? The specification defines a filesystem structure and environment variable namespace that interact with functions to produce discoverable software applications. 40

41 Client A controller for a SCIF, either for a developer or a user Integration A third party software or tool that understands the SCIF structure and interacts with all or some portion of it 41

42 Goals Enable users to easily generate predictable, discoverable applications A recipe format that maps to a SCIF, and vice versa Software to generate and manage SCIF 42

43 Non-Goals The Scientific Filesystem (SCIF) explicitly does not implement, define, or provide in v0.2: Package Management Workflow Management A mechanism for authentication and authorization or management of file permissions. Support or integration with non-linux operating systems. 43

44 Metadata Entry Points BESTAPP=foo VERSION=1.0./run-foo.sh./run-bar.sh The SCIF Recipe Content Files Install Routine 44

45 The SCIF Recipe Metadata Entry Points BESTAPP=foo VERSION=1.0./run-foo.sh./run-bar.sh Content Files Install Routine 45

46 %apprun hello-world /bin/bash hello-world.sh %appinstall hello-world echo "echo 'Hello World!'" >> $SCIF_APPBIN/hello-world.sh %app[action] [name] chmod u+x $SCIF_APPBIN/hello-world.sh %appenv hello-world THEBESTAPP=$SCIF_APPNAME export THEBESTAPP %applabels hello-world MAINTAINER Vanessasaur %apphelp hello-world This is an example "Hello World" application. You can install it to a Scientific Filesystem (scif) with the command: scif install hello-world.scif he -wo d.s i

47 Developer Interaction recipe.scif scif install recipe.scif scif run foo write recipe install interact

48 User Interaction recipe.scif scif install recipe.scif scif run foo write recipe install interact

49 User Interaction./container scif run foo interact apps run exec help inspect shell pyshell

50 ./scif apps docker run vanessa/scif:hw apps./scif-cli apps

51 ./scif apps docker run vanessa/scif:hw apps./scif-cli apps

52 ./scif run hello-world docker run vanessa/scif:hw run hello-world-echo./scif-cli run hello-world-echo %apprun hello-world-echo exec echo Hello World!

53 ./scif run hello-world docker run vanessa/scif:hw run hello-world-echo./scif-cli run hello-world-echo %apprun hello-world-echo exec echo Hello World!

54 ./scif exec hello-world docker run vanessa/scif:hw exec hello-world echo [e]omg./scif-cli exec hello-world echo [e]omg %appenv hello-world OMG=TACOS export OMG

55 ./scif exec hello-world docker run vanessa/scif:hw exec hello-world echo [e]omg./scif-cli exec hello-world echo [e]omg %appenv hello-world OMG=TACOS export OMG

56

57 ./scif inspect hello-world

58 ./scif help hello-world docker run vanessa/scif:hw help hello-world./scif-cli help hello-world %apphelp hello-world This is an example "Hello World" application. You can install it to a Scientific Filesystem (scif) with the command: scif install hello-world.scif

59 ./scif shell hello-world %appenv hello-world OMG=TACOS export OMG

60 ./scif shell hello-world

61 ./scif pyshell

62

63 Architecture The Scientific Filesystem has a default root on the host to intentionally be separate from standard linux folders, and to not interfere with likely existing folders (e.g., just /data or /apps). /scif /apps /data

64 Organization scif entrypoint foo runtime /scif /apps foo /foo /bin /lib /scif runscript runscript.help labels.json environment.sh

65 Organization scif entrypoint foo runtime /scif /apps foo /foo $PATH /bin $LD_LIBRARY_PATH /lib /scif runscript runscript.help labels.json environment.sh

66 Interaction./container apps foo bar /scif /apps /bar /foo

67 Interaction /scif /apps./container exec bar [command] /foo /bin /lib /scif runscript runscript.help labels.json environment.sh

68 Interaction /scif /apps./container run bar [args] /foo /bin /lib /scif runscript runscript.help labels.json environment.sh

69 Interaction /scif /apps./container shell bar $SCIF_SHELL./container pyshell bar /foo /bin /lib /scif runscript runscript.help labels.json environment.sh

70 Environment Namespace 70

71

72 ./container run foo foo active bar sleeping

73 ./container run foo BESTAPP=FOO! foo active bar sleeping

74 ./container run foo BESTAPP=FOO! foo active bar sleeping

75 foo active

76

77 What can I do with a SCIF? 77

78 - expose multiple container entrypoints - package metrics/evaluation alongside the analysis - SCIF apps as different working environments, runtime - SCIF apps for different runtime contexts

79 What are we going to do today? 79

80 What are we going to do today? Build Containers Run a Workflow Evaluate Across different container technologies, all with the same scientific filesystem (SCIF) The same snakemake workflow made modular by SCIF applications How do the different container technologies compare? 80

81

82 Scientific Pipeline - Felix Bartusch - variant calling - Snakemake 82

83 What level of modularity? 83

84 - Level 1: Container Modularity - One entrypoint Run the entire thing, or don t./container

85 - Level 2: Pipeline Modularity - Multiple entrypoints Correspond to pipeline steps Step collection of commands./container apps mapping sorting./container run mapping mapping sorting

86 - Level 3: Developer Modularity - Expose lowest level executables Ideal for building pipelines Step core tools./container apps bwa samtools./container run samtools bwa samtools

87 A Scientific Filesystem Design for Variant Calling What level of modularity should be exposed to interact with the container? bwa samtools graphviz_create_dag snakemake./container apps bwa samtools graphviz_create_dag snakemake

88 What did I do with the container? Build Containers Run a Workflow Evaluate 88

89 Container Comparisons What steps did we take across container technologies? 0. install the software 1. build the container 2. run components of the workflow 3. run the entire workflow

90 What did we learn? What did I learn about user interaction? Good software without documentation and tutorial for several use cases is akin to a treasure chest on a remote planet.

91 Container Comparisons What steps did we take across container technologies? 0. install the software

92 Container Comparisons What steps did we take across container technologies? 0. install the software 1. Documentation is the first interaction a. b. c. d. Finding it Is it up to date? Is there too much detail? Is there a clear avenue to get more help?

93 Container Comparisons What steps did we take across container technologies? 0. install the software 1. build the container

94 Container Comparisons What steps did we take across container technologies? 0. install the software 1. build the container 1. Examples are needed for learning a. Does it show me all the things I might need? i. binds ii. options iii. permissions

95 Container Comparisons What steps did we take across container technologies? 0. install the software 1. build the container 2. run components of the workflow

96 Container Comparisons What steps did we take across container technologies? 0. install the software 1. build the container 2. run components of the workflow 1. Establish boundaries a. show inside vs. outside

97 Container Comparisons What steps did we take across container technologies? 0. install the software 1. build the container 2. run components of the workflow 3. run the entire workflow

98 Run the entire workflow! 98

99 runc --root /tmp/runc run --bundle /tmp/runc snakemake.runc docker run -v $PWD/data:/scif/data:z -v $PWD:/scif/data/snakemake:z -it vanessa/snakemake.scif run snakemake all singularity run --bind data/:/scif/data snakemake.simg run snakemake all ch-run --cd $PWD -b data:/scif/data /var/tmp/snakemake.ch -- /opt/conda/bin/scif run snakemake all shifter --image=vanessa/snakemake.scif --volume `pwd`/data:/scif/data --workdir /scif/data /opt/conda/bin/scif run snakemake all

100 command runc --root /tmp/runc run --bundle /tmp/runc snakmake.runc docker run -v $PWD/data:/scif/data:z -v $PWD:/scif/data/snakemake:z -it vanessa/snakemake.scif run snakemake all singularity run --bind data/:/scif/data snakemake.simg run snakemake all ch-run --cd $PWD -b data:/scif/data /var/tmp/snakemake.ch -- /opt/conda/bin/scif run snakemake all shifter --image=vanessa/snakemake.scif --volume `pwd`/data:/scif/data --workdir /scif/data /opt/conda/bin/scif run snakemake all

101 executable runc --root /tmp/runc run --bundle /tmp/runc snakemake.runc docker run -v $PWD/data:/scif/data -v $PWD:/scif/data/snakemake -it vanessa/snakemake.scif run snakemake all singularity run --bind data/:/scif/data snakemake.simg run snakemake all ch-run --cd $PWD -b data:/scif/data /var/tmp/snakemake.ch -- /opt/conda/bin/scif run snakemake all shifter --image=vanessa/snakemake.scif --volume `pwd`/data:/scif/data --workdir /scif/data /opt/conda/bin/scif run snakemake all

102 executable + action runc --root /tmp/runc run --bundle /tmp/runc snakmake.runc docker run -v $PWD/data:/scif/data -v $PWD:/scif/data/snakemake -it vanessa/snakemake.scif run snakemake all singularity run --bind data/:/scif/data snakemake.simg run snakemake all ch-run --cd $PWD -b data:/scif/data /var/tmp/snakemake.ch -- /opt/conda/bin/scif run snakemake all shifter --image=vanessa/snakemake.scif --volume `pwd`/data:/scif/data --workdir /scif/data /opt/conda/bin/scif run snakemake all

103 container runc --root /tmp/runc run --bundle /tmp/runc snakemake.runc docker run -v $PWD/data:/scif/data -v $PWD:/scif/data/snakemake -it vanessa/snakemake.scif run snakemake all singularity run --bind data/:/scif/data snakemake.simg run snakemake all ch-run --cd $PWD -b data:/scif/data /var/tmp/snakemake.ch -- /opt/conda/bin/scif run snakemake all shifter --image=vanessa/snakemake.scif --volume `pwd`/data:/scif/data --workdir /scif/data /opt/conda/bin/scif run snakemake all

104 volumes / mounts runc --root /tmp/runc run --bundle /tmp/runc snakmake.runc docker run -v $PWD/data:/scif/data -v $PWD:/scif/data/snakemake -it vanessa/snakemake.scif run snakemake all singularity run --bind data/:/scif/data snakemake.simg run snakemake all ch-run --cd $PWD -b data:/scif/data /var/tmp/snakemake.ch -- /opt/conda/bin/scif run snakemake all shifter --image=vanessa/snakemake.scif --volume `pwd`/data:/scif/data --workdir /scif/data /opt/conda/bin/scif run snakemake all

105 executable action options container options command

106 106

107 Are the outputs comparable? 107

108 Results are the same $ diff results/docker/calls.vcf results/singularity/calls.vcf 28c28 < ##bcftools_callcommand=call -mv -; Date=Sat Mar 10 01:07: > ##bcftools_callcommand=call -mv -; Date=Fri Mar 9 20:07: $ diff results/docker/calls.vcf results/runc/calls.vcf 28c28 < ##bcftools_callcommand=call -mv -; Date=Sat Mar 10 01:07: > ##bcftools_callcommand=call -mv -; Date=Sat Mar 10 01:08: $ diff results/docker/calls.vcf results/shifter/calls.vcf 28c28 < ##bcftools_callcommand=call -mv -; Date=Sat Mar 10 01:07: > ##bcftools_callcommand=call -mv -; Date=Mon Mar 12 20:45: $ diff results/docker/calls.vcf results/charliecloud/calls.vcf 28c28 < ##bcftools_callcommand=call -mv -; Date=Sat Mar 10 01:07: > ##bcftools_callcommand=call -mv -; Date=Sat Mar 10 01:07: vs

109 What did we learn? Do the technologies produce the same result? From the perspective of the scientist, we have comparable resource usage and produce the same output.

110 What about resource usage? 110

111 singularity run --bind data/:/scif/data snakemake.simg run valgrind singularity run --bind data/:/scif/data snakemake.simg run timeit

112

113

114

115

116

117 ...but how do my choices of the content, and host, influence my result?

118 Does Heap Consumption Vary Across Machine Type?

119

120 Local machine sregistry build build template cloud builder cloud storage sregistry pull sregistry pull vsoch/hello-world:latest...

121 n1-standard-1 n1-standard-2 n1-standard-4 n1-standard-8

122 n1-standard-1 n1-standard-2 n1-standard-4 n1-standard-8

123 n1-standard-1 n1-standard-2 n1-standard-4 n1-standard-8

124 Does Heap Consumption Vary Across Machine Type? Variant calling memory usage is comparable from 1 to 64 cores

125 What is the optimal design?

126 tensorflow 17.0 tensorflow 17.1 theano theano 1.0.0

127 tensorflow 17.0 tensorflow 17.1 theano theano 1.0.0

128 TLDR 128

129 ! g n i h t e m a t does the s I 129

130 Containerization seems to produce a reproducible workflow.

131 We need to make tools that are usable across technologies.

132 Transparency, ease of use, and ability to contribute are important.

133 It s our job to encourage introspection and evaluation of variation.

134 Once upon a time There was a scientist

135 There was a reproducibility crisis

136 ...but friends were there to help :)

137

138 PRESENTER: VANESSASAURUS THIS IS THE END