Slacker: Fast Distribution with Lazy Docker Containers. Tyler Harter, Brandon Salmon, Rose Liu, Andrea C. Arpaci-Dusseau, Remzi H.

Size: px
Start display at page:

Download "Slacker: Fast Distribution with Lazy Docker Containers. Tyler Harter, Brandon Salmon, Rose Liu, Andrea C. Arpaci-Dusseau, Remzi H."

Transcription

1 Slacker: Fast Distribution with Lazy Docker Containers Tyler Harter, Brandon Salmon, Rose Liu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau

2 Container Popularity spoon.net

3 Theory and Practice Theory: containers are lightweight just like starting a process!

4 Theory and Practice Theory: containers are lightweight just like starting a process! Practice: container startup is slow 25 second startup time [1] task startup latency (the time from job submission to a task running) is an area that has received and continues to receive significant attention. It is highly variable, with the median typically about 25 s. Package installation takes about 80% of the total: one of the known bottlenecks is contention for the local disk where packages are written. [1] Large-scale cluster management at Google with Borg.

5 Theory and Practice Theory: containers are lightweight just like starting a process! Practice: container startup is slow 25 second startup time [1] task startup latency (the time from job submission to a task running) is an area that has received and continues to receive significant attention. It is highly variable, with the median typically about 25 s. Package installation takes about 80% of the total: one of the known bottlenecks is contention for the local disk where packages are written. [1] Large-scale cluster management at Google with Borg.

6 Theory and Practice Theory: containers are lightweight just like starting a process! Practice: container startup is slow 25 second startup time [1] Startup time matters flash crowds load balance interactive development [1] Large-scale cluster management at Google with Borg.

7 Contributions HelloBench Docker benchmark for stressing startup based on 57 container workloads Startup analysis 76% of startup time spent copying/installing images startup requires only 6% of that image data Slacker: Docker storage driver lazily pull only needed data leverage extensions to Linux kernel and NFS server 5-20x startup speedups

8 Slacker Outline Background Containers: lightweight isolation Docker: file-system provisioning Container Workloads Default Driver: AUFS Our Driver: Slacker Evaluation Conclusion

9 Why use containers?

10 Why use containers? (it s trendy)

11 Why use containers? (it s trendy) (efficient solution to classic problem)

12 Big Goal: Sharing and Isolation App A App B Physical Machine want: multitenancy

13 Big Goal: Sharing and Isolation App A App B Physical Machine don t want: crashes

14 Big Goal: Sharing and Isolation App A App B Physical Machine don t want: crashes

15 Big Goal: Sharing and Isolation App A App B Physical Machine don t want: unfairness

16 Big Goal: Sharing and Isolation App A sensitive data App B Physical Machine don t want: leaks

17 Solution: Virtualization namespaces and scheduling provide illusion of private resources

18 Evolution of Virtualization 1st generation: process virtualization isolate within OS (e.g., virtual memory) fast, but incomplete (missing ports, file system, etc.) App A App B process process Operating System process virtualization

19 Evolution of Virtualization 1st generation: process virtualization isolate within OS (e.g., virtual memory) fast, but incomplete (missing ports, file system, etc.) 2nd generation: machine virtualization isolate around OS complete, but slow (redundancy, emulation) App A App B App A App B process process OS OS Operating System VM VM process virtualization machine virtualization

20 Evolution of Virtualization 1st generation: process virtualization isolate within OS (e.g., virtual memory) fast, but incomplete (missing ports, file system, etc.) 2nd generation: machine virtualization isolate around OS complete, but slow (redundancy, emulation) App A App B App A App B process process OS OS Operating System VM VM process virtualization machine virtualization

21 Evolution of Virtualization 1st generation: process virtualization isolate within OS (e.g., virtual memory) fast, but incomplete (missing ports, file system, etc.) 2nd generation: machine virtualization isolate around OS complete, but slow (redundancy, emulation) 3rd generation: container virtualization extend process virtualization: ports, file system, etc. fast and complete

22 Evolution of Virtualization 1st generation: process virtualization isolate within OS (e.g., virtual memory) fast, but incomplete (missing ports, file system, etc.) 2nd generation: machine virtualization isolate around OS complete, but slow (redundancy, emulation) 3rd generation: container virtualization extend process virtualization: ports, file system, etc. fast and complete??? many storage challenges

23 New Storage Challenges Crash isolation Physical Disentanglement in a Container-Based File System. Lanyue Lu, Yupu Zhang, Thanh Do, Samer Al-Kiswany, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau. OSDI 14. Performance isolation Split-level I/O Scheduling For Virtualized Environments. Suli Yang, Tyler Harter, Nishant Agrawal, Salini Selvaraj Kowsalya, Anand Krishnamurthy, Samer Al-Kiswany, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau. SOSP 15. File-system provisioning Slacker: Fast Distribution with Lazy Docker Containers. Tyler Harter, Brandon Salmon, Rose Liu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau. FAST 16. today

24 Slacker Outline Background Containers: lightweight isolation Docker: file-system provisioning Container Workloads Default Driver: AUFS Our Driver: Slacker Evaluation Conclusion

25 Docker Background Deployment tool built on containers An application is defined by a file-system image application binary shared libraries etc. Version-control model extend images by committing additional files deploy applications by pushing/pulling images

26 Containers as Repos LAMP stack example commit 1: Linux packages (e.g., Ubuntu) commit 2: Apache commit 3: MySQL commit 4: PHP Central registries Docker HUB private registries Docker layer commit container scratch space

27 Push, Pull, Run registry worker worker worker

28 Push, Pull, Run registry push worker worker worker

29 Push, Pull, Run registry worker worker worker

30 Push, Pull, Run registry pull pull worker worker worker

31 Push, Pull, Run registry worker worker worker

32 Push, Pull, Run registry C run C run C run worker worker worker

33 Push, Pull, Run registry need a new benchmark to measure Docker push, pull, and run operations. C run C run C run worker worker worker

34 Slacker Outline Background Container Workloads HelloBench Analysis Default Driver: AUFS Our Driver: Slacker Evaluation Conclusion

35 HelloBench Goal: stress container startup including push/pull 57 container images from Docker HUB run simple hello world -like task wait until it s done/ready push pull run

36 HelloBench Goal: stress container startup including push/pull 57 container images from Docker HUB run simple hello world -like task wait until it s done/ready push pull run

37 HelloBench Goal: stress container startup including push/pull 57 container images from Docker HUB run simple hello world -like task wait until it s done/ready ready push pull run

38 HelloBench Goal: stress container startup including push/pull 57 container images from Docker HUB run simple hello world -like task wait until it s done/ready Development cycle distributed programming/testing ready push pull run development cycle

39 HelloBench Goal: stress container startup including push/pull 57 container images from Docker HUB run simple hello world -like task wait until it s done/ready Development cycle distributed programming/testing Deployment cycle flash crowds, rebalance ready push pull run deployment cycle

40 Workload Categories Linux Distro alpine busybox centos cirros crux debian fedora mageia opensuse oraclelinux ubuntu ubuntudebootstrap ubuntu-upstart Database cassandra crate elas6csearch mariadb mongo mysql percona postgres redis rethinkdb! Web Framework django iojs node rails Language clojure gcc golang haskell hylang java jruby julia mono perl php pypy python r-base rakudo-star ruby thri< Web Server glassfish h>pd je>y nginx php-zendserver tomcat! Other drupal ghost hello-world jenkins rabbitmq registry sonarqube

41 Slacker Outline Background Container Workloads HelloBench Analysis Default Driver: AUFS Our Driver: Slacker Evaluation Conclusion

42 Questions How is data distributed across Docker layers?! How much image data is needed for container startup?! How similar are reads between runs?

43 Questions How is data distributed across Docker layers?! How much image data is needed for container startup?! How similar are reads between runs?

44 HelloBench images circle: commit red: image

45 Image Data Depth

46 Image Data Depth half of data is at depth 9+

47 Questions How is data distributed across Docker layers? half of data is at depth 9+ design implication: flatten layers at runtime How much image data is needed for container startup? How similar are reads between runs?

48 Questions How is data distributed across Docker layers? half of data is at depth 9+ design implication: flatten layers at runtime How much image data is needed for container startup? How similar are reads between runs?

49 Container Amplification

50 Container Amplification

51 Container Amplification only 6.4% of data needed during startup

52 Questions How is data distributed across Docker layers? half of data is at depth 9+ design implication: flatten layers at runtime How much image data is needed for container startup? 6.4% of data is needed design implication: lazily fetch data How similar are reads between runs?

53 Questions How is data distributed across Docker layers? half of data is at depth 9+ design implication: flatten layers at runtime How much image data is needed for container startup? 6.4% of data is needed design implication: lazily fetch data How similar are reads between runs?

54 Repeat Runs measure hits/misses for second of two runs

55 Repeat Runs measure hits/misses for second of two runs up to 99% of reads could be serviced by a cache

56 Questions How is data distributed across Docker layers? half of data is at depth 9+ design implication: flatten layers at runtime How much image data is needed for container startup? 6.4% of data is needed design implication: lazily fetch data How similar are reads between runs? containers from same image have similar read patterns design implication: share cache state between containers

57 Slacker Outline Background Container Workloads Default Driver: AUFS Design Performance Our Driver: Slacker Evaluation Conclusion

58 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) layer directory in underlying FS root FS union of layer directories

59 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) layer directory in underlying FS root FS union of layer directories Operations push pull run

60 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS AUFS Driver layers: A B C

61 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS AUFS Driver directories: A B C

62 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS AUFS Driver PUSH directories: A B C

63 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS AUFS Driver PUSH directories: tar.gz A B C

64 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS AUFS Driver PUSH directories: tar.gz A B C

65 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS AUFS Driver PULL directories: A B C

66 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS AUFS Driver PULL directories: A B C tar.gz

67 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS AUFS Driver PULL directories: A B C tar.gz X Y Z

68 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS AUFS Driver PULL directories: A B C X Y Z

69 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS AUFS Driver RUN directories: A B C X Y Z

70 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS AUFS Driver RUN scratch dir: A B C X Y Z

71 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS root FS RUN AUFS Driver AUFS A B C X Y Z

72 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS read B RUN AUFS Driver AUFS A B C X Y Z

73 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS read B RUN AUFS Driver AUFS A B C X Y Z

74 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS RUN AUFS Driver AUFS A B C X Y Z

75 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS read X RUN AUFS Driver AUFS A B C X Y Z

76 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS read X RUN AUFS Driver AUFS A B C X Y Z

77 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS RUN AUFS Driver AUFS A B C X Y Z

78 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS append Z RUN AUFS Driver AUFS A B C X Y Z

79 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS append Z RUN AUFS Driver Z A B C copy AUFS X Y Z

80 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS append Z RUN AUFS Driver Z AUFS A B C X Y Z

81 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS append Z RUN AUFS Driver Z AUFS A B C X Y Z

82 AUFS Storage Driver Uses AUFS file system (Another Union FS) stores data in an underlying FS (e.g., ext4) each layer Docker directory layer in is underlying a directory FS in underlying FS union root FS these union directories of layer to directories create complete view of FS RUN AUFS Driver Z AUFS A B C X Y Z

83 Slacker Outline Background Container Workloads Default Driver: AUFS Design Performance Our Driver: Slacker Evaluation Conclusion

84 AUFS File System

85 AUFS File System Deep data is slow

86 AUFS Storage Driver

87 AUFS Storage Driver 76% of deployment cycle spent on pull

88 Slacker Outline Background Container Workloads Default Driver: AUFS Our Driver: Slacker Evaluation Conclusion

89 Slacker Driver Goals make push+pull very fast utilize powerful primitives of a modern storage server (Tintri VMstore) create drop-in replacement; don t change Docker framework itself Design lazy pull layer flattening cache sharing

90 Slacker Driver Goals make push+pull very fast utilize powerful primitives of a modern storage server (Tintri VMstore) create drop-in replacement; don t change Docker framework itself Design lazy pull layer flattening cache sharing

91 Prefetch vs. Lazy Fetch AUFS Slacker registry worker registry worker images containers images and containers

92 Prefetch vs. Lazy Fetch AUFS Slacker registry worker registry worker images containers images and containers significant copying over network to/from disk centralized storage easy sharing

93 Prefetch vs. Lazy Fetch AUFS Slacker registry worker registry worker images containers images and containers

94 Prefetch vs. Lazy Fetch Slacker registry container ext4 loopback images and containers NFS File

95 Prefetch vs. Lazy Fetch Slacker registry container ext4 loopback NFS File

96 Prefetch vs. Lazy Fetch Slacker registry container ext4 loopback NFS File VMstore abstractions

97 VMstore Abstractions Copy-on-Write VMstore provides snapshot() and clone() block granularity avoids AUFS s problems with file granularity snapshot(nfs_path)! create read-only copy of NFS file return snapshot ID clone(snapshot_id)! create r/w NFS file from snapshot Slacker Usage NFS files container storage snapshots image storage clone() provision container from image snapshot() create image from container

98 Snapshot and Clone worker A container NFS file Tintri VMstore

99 Snapshot and Clone worker A snapshot NFS file Tintri VMstore Worker A: push

100 Snapshot and Clone worker A snapshot NFS file COW snap N Tintri VMstore Worker A: push

101 Snapshot and Clone worker A N NFS file snap N Tintri VMstore Worker A: push

102 Snapshot and Clone worker A N registry NFS file snap N Tintri VMstore Worker A: push

103 Snapshot and Clone worker A registry img N NFS file snap N Tintri VMstore Worker A: push

104 Snapshot and Clone Note: registry is only a name server. Maps layer metadata snapshot ID worker A registry img N NFS file snap N Tintri VMstore

105 Snapshot and Clone worker A registry img N NFS file snap N Tintri VMstore

106 Snapshot and Clone worker A registry worker B img N NFS file snap N Tintri VMstore Worker B: pull and run

107 Snapshot and Clone worker A registry worker B img N N NFS file snap N Tintri VMstore Worker B: pull and run

108 Snapshot and Clone worker A registry worker B img N clone N NFS file snap N Tintri VMstore Worker B: pull and run

109 Snapshot and Clone worker A registry worker B img N clone N NFS file snap N COW NFS file Tintri VMstore Worker B: pull and run

110 Snapshot and Clone worker A registry worker B img N NFS file snap N NFS file Tintri VMstore Worker B: pull and run

111 Snapshot and Clone worker A registry worker B img container NFS file snap N NFS file Tintri VMstore Worker B: pull and run

112 Slacker Driver Goals make push+pull very fast utilize powerful primitives of a modern storage server (Tintri VMstore) create drop-in replacement; don t change Docker framework itself Design lazy pull layer flattening cache sharing

113 Slacker Driver Goals make push+pull very fast utilize powerful primitives of a modern storage server (Tintri VMstore) create drop-in replacement; don t change Docker framework itself Design lazy pull layer flattening cache sharing

114 Slacker Flattening File Namespace Level flatten layers if B is child of A, then copy A to B to start. Don t make B empty Block Level do COW+dedup beneath NFS files, inside VMstore namespace copy-on-write dir dir dir dir ext4 ext4 ext4 ext4 ext4 NFS NFS NFS NFS copy-on-write block AUFS Slacker

115 Slacker Flattening File Namespace Level flatten layers if B is child of A, then copy A to B to start. Don t make B empty Block Level do COW+dedup beneath NFS files, inside VMstore namespace copy-on-write A B C D ext4 A ext4 ext4 ext4 AB ABC ABCD ext4 copy-on-write block AUFS Slacker

116 Challenge: Framework Assumptions Assumed Layout Actual Layout D A B C D runnable Layers B C runnable Layers A B C A B A A

117 Challenge: Framework Assumptions Assumed Layout D Actual Layout A B C D Layers B C pull Layers A B C A B pull A A

118 Challenge: Framework Assumptions Strategy: lazy cloning. Don t clone non-top layers until Docker tries to mount them. Assumed Layout D Actual Layout A B C D Layers B C Layers A B C A B optimize A A

119 Slacker Driver Goals make push+pull very fast utilize powerful primitives of a modern storage server (Tintri VMstore) create drop-in replacement; don t change Docker framework itself Design lazy pull layer flattening cache sharing

120 Slacker Driver Goals make push+pull very fast utilize powerful primitives of a modern storage server (Tintri VMstore) create drop-in replacement; don t change Docker framework itself Design lazy pull layer flattening cache sharing

121 Challenge: Cache Sharing NFS Client: cache: storage for 2 containers started from same image ABC image ABX ABY

122 Challenge: Cache Sharing NFS Client: cache: ABX ABY

123 Challenge: Cache Sharing NFS Client: cache: read ABX ABY

124 Challenge: Cache Sharing NFS Client: cache: AABX ABY

125 Challenge: Cache Sharing NFS Client: cache: A ABX ABY

126 Challenge: Cache Sharing NFS Client: cache: A read ABX ABY

127 Challenge: Cache Sharing NFS Client: cache: A ABX ABY A

128 Challenge: Cache Sharing NFS Client: cache: A A ABX ABY

129 Challenge: Cache Sharing NFS Client: cache: A A Challenge: how to avoid space and I/O waste? ABX ABY

130 Challenge: Cache Sharing NFS Client: cache: A A Strategy: track differences and deduplicate I/O (more in paper) ABX ABY

131 Slacker Outline Background Container Workloads Default Driver: AUFS Our Driver: Slacker Evaluation Conclusion

132 Questions What are deployment and development speedups? How is long-term performance?!

133 Questions What are deployment and development speedups? How is long-term performance?

134 HelloBench Performance deployment: pull+run development: push+pull+run

135 Questions What are deployment and development speedups? 5x and 20x faster respectively (median speedup) How is long-term performance?

136 Questions What are deployment and development speedups? 5x and 20x faster respectively (median speedup) How is long-term performance?

137 Server Benchmarks Databases and Web Servers PostgreSQL Redis Apache web server (static) io.js Javascript server (dynamic) Experiment measure throughput (after startup) run 5 minutes

138 Server Benchmarks Databases and Web Servers PostgreSQL Redis Apache web server (static) io.js Javascript server (dynamic) Experiment measure throughput (after startup) run 5 minutes Result: Slacker is always at least as fast as AUFS

139 Questions What are deployment and development speedups? 5x and 20x faster respectively (median speedup) How is long-term performance? there is no long-term penalty for being lazy

140 Slacker Outline Background Container Workloads Default Driver: AUFS Our Driver: Slacker Evaluation Conclusion

141 Conclusion Containers are inherently lightweight but existing frameworks are not COW between workers is necessary for fast startup use shared storage utilize VMstore snapshot and clone Slacker driver 5x deployment speedup 20x development speedup HelloBench:

Emergent Properties in Modular Storage A Study of Apple Desktop Applications, Facebook Messages, and Docker Containers

Emergent Properties in Modular Storage A Study of Apple Desktop Applications, Facebook Messages, and Docker Containers Emergent Properties in Modular Storage A Study of Apple Desktop Applications, Facebook Messages, and Docker Containers Tyler Harter How are complex applications built? (for example, Facebook Messages)

More information

Supporting Docker in Emulab-Based Network Testbeds. David Johnson, Elijah Grubb, Eric Eide University of Utah

Supporting Docker in Emulab-Based Network Testbeds. David Johnson, Elijah Grubb, Eric Eide University of Utah Supporting Docker in Emulab-Based Network Testbeds David Johnson, Elijah Grubb, Eric Eide University of Utah 2 2 2 2 over the course of a study prototype on laptop network testbed commercial cloud need

More information

Physical Disentanglement in a Container-Based File System

Physical Disentanglement in a Container-Based File System Physical Disentanglement in a Container-Based File System Lanyue Lu, Yupu Zhang, Thanh Do, Samer AI-Kiswany Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau University of Wisconsin Madison Motivation

More information

CS-580K/480K Advanced Topics in Cloud Computing. Container III

CS-580K/480K Advanced Topics in Cloud Computing. Container III CS-580/480 Advanced Topics in Cloud Computing Container III 1 Docker Container https://www.docker.com/ Docker is a platform for developers and sysadmins to develop, deploy, and run applications with containers.

More information

[Docker] Containerization

[Docker] Containerization [Docker] Containerization ABCD-LMA Working Group Will Kinard October 12, 2017 WILL Kinard Infrastructure Architect Software Developer Startup Venture IC Husband Father Clemson University That s me. 2 The

More information

Docker 101 Workshop. Eric Smalling - Solution Architect, Docker

Docker 101 Workshop. Eric Smalling - Solution Architect, Docker Docker 101 Workshop Eric Smalling - Solution Architect, Docker Inc. @ericsmalling Who Am I? Eric Smalling Solution Architect Docker Customer Success Team ~25 years in software development, architecture,

More information

Developing and Testing Java Microservices on Docker. Todd Fasullo Dir. Engineering

Developing and Testing Java Microservices on Docker. Todd Fasullo Dir. Engineering Developing and Testing Java Microservices on Docker Todd Fasullo Dir. Engineering Agenda Who is Smartsheet + why we started using Docker Docker fundamentals Demo - creating a service Demo - building service

More information

Let's Play... Try to name the databases described on the following slides...

Let's Play... Try to name the databases described on the following slides... Database Software Let's Play... Try to name the databases described on the following slides... "World's most popular" Free relational database system (RDBMS) that... the "M" in "LAMP" and "XAMP" stacks

More information

Microservices. Chaos Kontrolle mit Kubernetes. Robert Kubis - Developer Advocate,

Microservices. Chaos Kontrolle mit Kubernetes. Robert Kubis - Developer Advocate, Microservices Chaos Kontrolle mit Kubernetes Robert Kubis - Developer Advocate, Google @hostirosti About me Robert Kubis Developer Advocate Google Cloud Platform London, UK hostirosti github.com/hostirosti

More information

Designing a True Direct-Access File System with DevFS

Designing a True Direct-Access File System with DevFS Designing a True Direct-Access File System with DevFS Sudarsun Kannan, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University of Wisconsin-Madison Yuangang Wang, Jun Xu, Gopinath Palani Huawei Technologies

More information

ROBIN SYSTEMS. Containerizing Oracle: Not Thinking About It Yet? You Should Be!!!

ROBIN SYSTEMS. Containerizing Oracle: Not Thinking About It Yet? You Should Be!!! ROBIN SYSTEMS Containerizing Oracle: Not Thinking About It Yet? You Should Be!!! ABOUT ME Over 19 years of experience across Databases, & big data applications Director of Products, Robin Systems Virtualizing

More information

Improve Web Application Performance with Zend Platform

Improve Web Application Performance with Zend Platform Improve Web Application Performance with Zend Platform Shahar Evron Zend Sr. PHP Specialist Copyright 2007, Zend Technologies Inc. Agenda Benchmark Setup Comprehensive Performance Multilayered Caching

More information

Beyond 1001 Dedicated Data Service Instances

Beyond 1001 Dedicated Data Service Instances Beyond 1001 Dedicated Data Service Instances Introduction The Challenge Given: Application platform based on Cloud Foundry to serve thousands of apps Application Runtime Many platform users - who don

More information

Travis Cardwell Technical Meeting

Travis Cardwell Technical Meeting .. Introduction to Docker Travis Cardwell Tokyo Linux Users Group 2014-01-18 Technical Meeting Presentation Motivation OS-level virtualization is becoming accessible Docker makes it very easy to experiment

More information

Index. Bessel function, 51 Big data, 1. Cloud-based version-control system, 226 Containerization, 30 application, 32 virtualize processes, 30 31

Index. Bessel function, 51 Big data, 1. Cloud-based version-control system, 226 Containerization, 30 application, 32 virtualize processes, 30 31 Index A Amazon Web Services (AWS), 2 account creation, 2 EC2 instance creation, 9 Docker, 13 IP address, 12 key pair, 12 launch button, 11 security group, 11 stable Ubuntu server, 9 t2.micro type, 9 10

More information

An introduction to Docker

An introduction to Docker An introduction to Docker Ing. Vincenzo Maffione Operating Systems Security Container technologies on Linux Several light virtualization technologies are available for Linux They build on cgroups, namespaces

More information

Presented By: Gregory M. Kurtzer HPC Systems Architect Lawrence Berkeley National Laboratory CONTAINERS IN HPC WITH SINGULARITY

Presented By: Gregory M. Kurtzer HPC Systems Architect Lawrence Berkeley National Laboratory CONTAINERS IN HPC WITH SINGULARITY Presented By: Gregory M. Kurtzer HPC Systems Architect Lawrence Berkeley National Laboratory gmkurtzer@lbl.gov CONTAINERS IN HPC WITH SINGULARITY A QUICK REVIEW OF THE LANDSCAPE Many types of virtualization

More information

Cider: a Rapid Docker Container Deployment System Through Sharing Network Storage

Cider: a Rapid Docker Container Deployment System Through Sharing Network Storage Cider: a Rapid Docker Container Deployment System Through Sharing Network Storage Lian Du, Tianyu Wo, Renyu Yang *, Chunming Hu State Key Laboratory of Software Development Environment BDBC, Beihang University,

More information

STATE OF MODERN APPLICATIONS IN THE CLOUD

STATE OF MODERN APPLICATIONS IN THE CLOUD STATE OF MODERN APPLICATIONS IN THE CLOUD 2017 Introduction The Rise of Modern Applications What is the Modern Application? Today s leading enterprises are striving to deliver high performance, highly

More information

Introduction to Docker. Antonis Kalipetis Docker Athens Meetup

Introduction to Docker. Antonis Kalipetis Docker Athens Meetup Introduction to Docker Antonis Kalipetis - @akalipetis Docker Athens Meetup Contents Introduction to Docker, Containers, and the Matrix from Hell Why people care: Separation of Concerns Technical Discussion

More information

Global Software Distribution with CernVM-FS

Global Software Distribution with CernVM-FS Global Software Distribution with CernVM-FS Jakob Blomer CERN 2016 CCL Workshop on Scalable Computing October 19th, 2016 jblomer@cern.ch CernVM-FS 1 / 15 The Anatomy of a Scientific Software Stack (In

More information

Midterm Presentation Schedule

Midterm Presentation Schedule Midterm Presentation Schedule October 18 th Aurora, Bash, Sangam October 20 th Flash, Omega, CodeRing October 25th Omni, Aviato, NPComplete Mid Term Presentation Format 25 minutes Be prepared to use the

More information

Engineering Robust Server Software

Engineering Robust Server Software Engineering Robust Server Software Containers Isolation Isolation: keep different programs separate Good for security Might also consider performance isolation Also has security implications (side channel

More information

Shifter at CSCS Docker Containers for HPC

Shifter at CSCS Docker Containers for HPC Shifter at CSCS Docker Containers for HPC HPC Advisory Council Swiss Conference Alberto Madonna, Lucas Benedicic, Felipe A. Cruz, Kean Mariotti - CSCS April 9 th, 2018 Table of Contents 1. Introduction

More information

利用 Mesos 打造高延展性 Container 環境. Frank, Microsoft MTC

利用 Mesos 打造高延展性 Container 環境. Frank, Microsoft MTC 利用 Mesos 打造高延展性 Container 環境 Frank, Microsoft MTC About Me Developer @ Yahoo! DevOps @ HTC Technical Architect @ MSFT Agenda About Docker Manage containers Apache Mesos Mesosphere DC/OS application = application

More information

Table of Contents 1.1. Introduction. Overview of vsphere Integrated Containers 1.2

Table of Contents 1.1. Introduction. Overview of vsphere Integrated Containers 1.2 Table of Contents Introduction Overview of vsphere Integrated Containers 1.1 1.2 2 Overview of vsphere Integrated Containers This document provides an overview of VMware vsphere Integrated Containers.

More information

What s Up Docker. Presented by Robert Sordillo Avada Software

What s Up Docker. Presented by Robert Sordillo Avada Software What s Up Docker Presented by Robert Sordillo (rsordillo@avadasoftware.com) Avada Software What is Docker? Is a open source software Container platform. It s benefits are eliminating works on my machine

More information

Cloud providers, tools and best practices in running Magento on Kubernetes. Adrian Balcan MindMagnet Software

Cloud providers, tools and best practices in running Magento on Kubernetes. Adrian Balcan MindMagnet Software Cloud providers, tools and best practices in running Magento on Kubernetes Adrian Balcan DevOps @ MindMagnet Software About Me Companies Projects Adrian Balcan contact@adrianbalcan.com Agenda Magento on

More information

FS Consistency & Journaling

FS Consistency & Journaling FS Consistency & Journaling Nima Honarmand (Based on slides by Prof. Andrea Arpaci-Dusseau) Why Is Consistency Challenging? File system may perform several disk writes to serve a single request Caching

More information

SLIPSTREAM: AUTOMATIC INTERPROCESS COMMUNICATION OPTIMIZATION. Will Dietz, Joshua Cranmer, Nathan Dautenhahn, Vikram Adve

SLIPSTREAM: AUTOMATIC INTERPROCESS COMMUNICATION OPTIMIZATION. Will Dietz, Joshua Cranmer, Nathan Dautenhahn, Vikram Adve SLIPSTREAM: AUTOMATIC INTERPROCESS COMMUNICATION OPTIMIZATION Will Dietz, Joshua Cranmer, Nathan Dautenhahn, Vikram Adve Introduction 2 Use of TCP is ubiquitous Widely Supported Location Transparency Programmer-friendly

More information

Building A Better Test Platform:

Building A Better Test Platform: Building A Better Test Platform: A Case Study of Improving Apache HBase Testing with Docker Aleks Shulman, Dima Spivak Outline About Cloudera Apache HBase Overview API compatibility API compatibility testing

More information

/ Cloud Computing. Recitation 5 February 14th, 2017

/ Cloud Computing. Recitation 5 February 14th, 2017 15-319 / 15-619 Cloud Computing Recitation 5 February 14th, 2017 1 Overview Administrative issues Office Hours, Piazza guidelines Last week s reflection Project 2.1, OLI Unit 2 modules 5 and 6 This week

More information

Current Topics in OS Research. So, what s hot?

Current Topics in OS Research. So, what s hot? Current Topics in OS Research COMP7840 OSDI Current OS Research 0 So, what s hot? Operating systems have been around for a long time in many forms for different types of devices It is normally general

More information

CA485 Ray Walshe Google File System

CA485 Ray Walshe Google File System Google File System Overview Google File System is scalable, distributed file system on inexpensive commodity hardware that provides: Fault Tolerance File system runs on hundreds or thousands of storage

More information

Getting Started with Hadoop

Getting Started with Hadoop Getting Started with Hadoop May 28, 2018 Michael Völske, Shahbaz Syed Web Technology & Information Systems Bauhaus-Universität Weimar 1 webis 2018 What is Hadoop Started in 2004 by Yahoo Open-Source implementation

More information

OS Virtualization. Linux Containers (LXC)

OS Virtualization. Linux Containers (LXC) OS Virtualization Emulate OS-level interface with native interface Lightweight virtual machines No hypervisor, OS provides necessary support Referred to as containers Solaris containers, BSD jails, Linux

More information

Red Hat Software Collections. Ryan Hennessy Sr. Solutions Architect

Red Hat Software Collections. Ryan Hennessy Sr. Solutions Architect Red Hat Software Collections Ryan Hennessy Sr. Solutions Architect hennessy@redhat.com Hello Everybody... 2 Solutions Architect based out of the FAR west suburbs of Chicago (Iowa Adjacent) Husband, father

More information

Simplified CICD with Jenkins and Git on the ZeroStack Platform

Simplified CICD with Jenkins and Git on the ZeroStack Platform DATA SHEET Simplified CICD with Jenkins and Git on the ZeroStack Platform In the technical article we will walk through an end to end workflow of starting from virtually nothing and establishing a CICD

More information

Using the SDACK Architecture to Build a Big Data Product. Yu-hsin Yeh (Evans Ye) Apache Big Data NA 2016 Vancouver

Using the SDACK Architecture to Build a Big Data Product. Yu-hsin Yeh (Evans Ye) Apache Big Data NA 2016 Vancouver Using the SDACK Architecture to Build a Big Data Product Yu-hsin Yeh (Evans Ye) Apache Big Data NA 2016 Vancouver Outline A Threat Analytic Big Data product The SDACK Architecture Akka Streams and data

More information

Think Small to Scale Big

Think Small to Scale Big Think Small to Scale Big Intro to Containers for the Datacenter Admin Pete Zerger Principal Program Manager, MVP pete.zerger@cireson.com Cireson Lee Berg Blog, e-mail address, title Company Pete Zerger

More information

Generating Realistic Impressions for File-System Benchmarking. Nitin Agrawal Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau

Generating Realistic Impressions for File-System Benchmarking. Nitin Agrawal Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau Generating Realistic Impressions for File-System Benchmarking Nitin Agrawal Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau For better or for worse," benchmarks shape a field " David Patterson 2 Inputs

More information

Arrakis: The Operating System is the Control Plane

Arrakis: The Operating System is the Control Plane Arrakis: The Operating System is the Control Plane Simon Peter, Jialin Li, Irene Zhang, Dan Ports, Doug Woos, Arvind Krishnamurthy, Tom Anderson University of Washington Timothy Roscoe ETH Zurich Building

More information

PERSISTENCE: FSCK, JOURNALING. Shivaram Venkataraman CS 537, Spring 2019

PERSISTENCE: FSCK, JOURNALING. Shivaram Venkataraman CS 537, Spring 2019 PERSISTENCE: FSCK, JOURNALING Shivaram Venkataraman CS 537, Spring 2019 ADMINISTRIVIA Project 4b: Due today! Project 5: Out by tomorrow Discussion this week: Project 5 AGENDA / LEARNING OUTCOMES How does

More information

Continuous delivery of Java applications. Marek Kratky Principal Sales Consultant Oracle Cloud Platform. May, 2016

Continuous delivery of Java applications. Marek Kratky Principal Sales Consultant Oracle Cloud Platform. May, 2016 Continuous delivery of Java applications using Oracle Cloud Platform Services Marek Kratky Principal Sales Consultant Oracle Cloud Platform May, 2016 Safe Harbor Statement The following is intended to

More information

ViewBox. Integrating Local File Systems with Cloud Storage Services. Yupu Zhang +, Chris Dragga + *, Andrea Arpaci-Dusseau +, Remzi Arpaci-Dusseau +

ViewBox. Integrating Local File Systems with Cloud Storage Services. Yupu Zhang +, Chris Dragga + *, Andrea Arpaci-Dusseau +, Remzi Arpaci-Dusseau + ViewBox Integrating Local File Systems with Cloud Storage Services Yupu Zhang +, Chris Dragga + *, Andrea Arpaci-Dusseau +, Remzi Arpaci-Dusseau + + University of Wisconsin Madison *NetApp, Inc. 5/16/2014

More information

vsansparse Tech Note First Published On: Last Updated On:

vsansparse Tech Note First Published On: Last Updated On: First Published On: 02-09-2017 Last Updated On: 02-09-2017 1 Table of Contents 1. Introduction 1.1.Introduction 1.2.Virtual Machine Snapshot Overview 2. Introducing vsansparse Snapshots 2.1.Introducing

More information

Copyright 2014 Fusion-io, Inc. All rights reserved.

Copyright 2014 Fusion-io, Inc. All rights reserved. Snapshots in a Flash with iosnap TM Sriram Subramanian, Swami Sundararaman, Nisha Talagala, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau Presented By: Samer Al-Kiswany Snapshots Overview Point-in-time representation

More information

Table of Contents 1.1. Overview. Containers, Docker, Registries vsphere Integrated Containers Engine

Table of Contents 1.1. Overview. Containers, Docker, Registries vsphere Integrated Containers Engine Table of Contents Overview Containers, Docker, Registries vsphere Integrated Containers Engine Management Portal Registry Roles and Personas 1.1 1.1.1 1.1.2 1.1.2.1 1.1.2.2 1.1.2.3 1.1.2.4 2 Overview of

More information

docker & HEP: containerization of applications for development, distribution and preservation

docker & HEP: containerization of applications for development, distribution and preservation docker & HEP: containerization of applications for development, distribution and preservation Sébastien Binet LAL/IN2P3 2015-04-13 S. Binet (LAL) docker-hep 2015-04-13 1 / 16 Docker: what is it? http://www.docker.io/

More information

At Course Completion Prepares you as per certification requirements for AWS Developer Associate.

At Course Completion Prepares you as per certification requirements for AWS Developer Associate. [AWS-DAW]: AWS Cloud Developer Associate Workshop Length Delivery Method : 4 days : Instructor-led (Classroom) At Course Completion Prepares you as per certification requirements for AWS Developer Associate.

More information

*-Box (star-box) Towards Reliability and Consistency in Dropbox-like File Synchronization Services

*-Box (star-box) Towards Reliability and Consistency in Dropbox-like File Synchronization Services *-Box (star-box) Towards Reliability and Consistency in -like File Synchronization Services Yupu Zhang, Chris Dragga, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University of Wisconsin - Madison 6/27/2013

More information

ovirt and Docker Integration

ovirt and Docker Integration ovirt and Docker Integration October 2014 Federico Simoncelli Principal Software Engineer Red Hat 1 Agenda Deploying an Application (Old-Fashion and Docker) Ecosystem: Kubernetes and Project Atomic Current

More information

Running MarkLogic in Containers (Both Docker and Kubernetes)

Running MarkLogic in Containers (Both Docker and Kubernetes) Running MarkLogic in Containers (Both Docker and Kubernetes) Emma Liu Product Manager, MarkLogic Vitaly Korolev Staff QA Engineer, MarkLogic @vitaly_korolev 4 June 2018 MARKLOGIC CORPORATION Source: http://turnoff.us/image/en/tech-adoption.png

More information

[537] Journaling. Tyler Harter

[537] Journaling. Tyler Harter [537] Journaling Tyler Harter FFS Review Problem 1 What structs must be updated in addition to the data block itself? [worksheet] Problem 1 What structs must be updated in addition to the data block itself?

More information

Towards Efficient, Portable Application-Level Consistency

Towards Efficient, Portable Application-Level Consistency Towards Efficient, Portable Application-Level Consistency Thanumalayan Sankaranarayana Pillai, Vijay Chidambaram, Joo-Young Hwang, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau 1 File System Crash

More information

Lecture 11 Hadoop & Spark

Lecture 11 Hadoop & Spark Lecture 11 Hadoop & Spark Dr. Wilson Rivera ICOM 6025: High Performance Computing Electrical and Computer Engineering Department University of Puerto Rico Outline Distributed File Systems Hadoop Ecosystem

More information

@joerg_schad Nightmares of a Container Orchestration System

@joerg_schad Nightmares of a Container Orchestration System @joerg_schad Nightmares of a Container Orchestration System 2017 Mesosphere, Inc. All Rights Reserved. 1 Jörg Schad Distributed Systems Engineer @joerg_schad Jan Repnak Support Engineer/ Solution Architect

More information

Use Case: Scalable applications

Use Case: Scalable applications Use Case: Scalable applications 1. Introduction A lot of companies are running (web) applications on a single machine, self hosted, in a datacenter close by or on premise. The hardware is often bought

More information

Virtualization. Dr. Yingwu Zhu

Virtualization. Dr. Yingwu Zhu Virtualization Dr. Yingwu Zhu Virtualization Definition Framework or methodology of dividing the resources of a computer into multiple execution environments. Types Platform Virtualization: Simulate a

More information

Introduction to MapReduce

Introduction to MapReduce Basics of Cloud Computing Lecture 4 Introduction to MapReduce Satish Srirama Some material adapted from slides by Jimmy Lin, Christophe Bisciglia, Aaron Kimball, & Sierra Michels-Slettvet, Google Distributed

More information

Container-based virtualization: Docker

Container-based virtualization: Docker Università degli Studi di Roma Tor Vergata Dipartimento di Ingegneria Civile e Ingegneria Informatica Container-based virtualization: Docker Corso di Sistemi Distribuiti e Cloud Computing A.A. 2018/19

More information

SCALING DRUPAL TO THE CLOUD WITH DOCKER AND AWS

SCALING DRUPAL TO THE CLOUD WITH DOCKER AND AWS SCALING DRUPAL TO THE CLOUD WITH DOCKER AND AWS Dr. Djun Kim Camp Pacific OUTLINE Overview Quick Intro to Docker Intro to AWS Designing a scalable application Connecting Drupal to AWS services Intro to

More information

Emulating Goliath Storage Systems with David

Emulating Goliath Storage Systems with David Emulating Goliath Storage Systems with David Nitin Agrawal, NEC Labs Leo Arulraj, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau ADSL Lab, UW Madison 1 The Storage Researchers Dilemma Innovate Create

More information

Singularity: Containers for High-Performance Computing. Grigory Shamov Nov 21, 2017

Singularity: Containers for High-Performance Computing. Grigory Shamov Nov 21, 2017 Singularity: Containers for High-Performance Computing Grigory Shamov Nov 21, 2017 Outline Software and High Performance Computing: Installation/Maintenance of the HPC Software stack Why containers and

More information

Docker und IBM Digital Experience in Docker Container

Docker und IBM Digital Experience in Docker Container Docker und IBM Digital Experience in Docker Container 20. 21. Juni 2017 IBM Labor Böblingen 1 What is docker Introduction VMs vs. containers Terminology v Docker components 2 6/22/2017 What is docker?

More information

Principled Schedulability Analysis for Distributed Storage Systems Using Thread Architecture Models

Principled Schedulability Analysis for Distributed Storage Systems Using Thread Architecture Models Principled Schedulability Analysis for Distributed Storage Systems Using Thread Architecture Models Suli Yang*, Jing Liu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau * work done while at UW-Madison

More information

Harbor Registry. VMware VMware Inc. All rights reserved.

Harbor Registry. VMware VMware Inc. All rights reserved. Harbor Registry VMware 2017 VMware Inc. All rights reserved. VMware Harbor Registry Cloud Foundry Agenda 1 Container Image Basics 2 Project Harbor Introduction 3 Consistency of Images 4 Security 5 Image

More information

Docker for People. A brief and fairly painless introduction to Docker. Friday, November 17 th 11:00-11:45

Docker for People. A brief and fairly painless introduction to Docker. Friday, November 17 th 11:00-11:45 Docker for People A brief and fairly painless introduction to Docker Friday, November 17 th 11:00-11:45 Greg Gómez Sung-Hee Lee The University of New Mexico IT NM TIE 2017 1 Docker for People Agenda: Greg:

More information

Yiying Zhang, Leo Prasath Arulraj, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. University of Wisconsin - Madison

Yiying Zhang, Leo Prasath Arulraj, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. University of Wisconsin - Madison Yiying Zhang, Leo Prasath Arulraj, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau University of Wisconsin - Madison 1 Indirection Reference an object with a different name Flexible, simple, and

More information

File System Performance (and Abstractions) Kevin Webb Swarthmore College April 5, 2018

File System Performance (and Abstractions) Kevin Webb Swarthmore College April 5, 2018 File System Performance (and Abstractions) Kevin Webb Swarthmore College April 5, 2018 Today s Goals Supporting multiple file systems in one name space. Schedulers not just for CPUs, but disks too! Caching

More information

DevOps Course Content

DevOps Course Content DevOps Course Content 1. Introduction: Understanding Development Development SDLC using WaterFall & Agile Understanding Operations DevOps to the rescue What is DevOps DevOps SDLC Continuous Delivery model

More information

Kubernetes Integration with Virtuozzo Storage

Kubernetes Integration with Virtuozzo Storage Kubernetes Integration with Virtuozzo Storage A Technical OCTOBER, 2017 2017 Virtuozzo. All rights reserved. 1 Application Container Storage Application containers appear to be the perfect tool for supporting

More information

Heroku. Rimantas Kybartas

Heroku. Rimantas Kybartas Heroku Rimantas Kybartas Salesforce platform (PaaS) Facts about Heroku Has been in development since June 2007, 2010 acquired by Salesforce Open platform Languages and frameworks: Ruby and Rails Node.js

More information

PHP Composer 9 Benefits of Using a Binary Repository Manager

PHP Composer 9 Benefits of Using a Binary Repository Manager PHP Composer 9 Benefits of Using a Binary Repository Manager White Paper Copyright 2017 JFrog Ltd. March 2017 www.jfrog.com Executive Summary PHP development has become one of the most popular platforms

More information

/ Cloud Computing. Recitation 5 September 26 th, 2017

/ Cloud Computing. Recitation 5 September 26 th, 2017 15-319 / 15-619 Cloud Computing Recitation 5 September 26 th, 2017 1 Overview Administrative issues Office Hours, Piazza guidelines Last week s reflection Project 2.1, OLI Unit 2 modules 5 and 6 This week

More information

Hedvig as backup target for Veeam

Hedvig as backup target for Veeam Hedvig as backup target for Veeam Solution Whitepaper Version 1.0 April 2018 Table of contents Executive overview... 3 Introduction... 3 Solution components... 4 Hedvig... 4 Hedvig Virtual Disk (vdisk)...

More information

Genomics on Cisco Metacloud + SwiftStack

Genomics on Cisco Metacloud + SwiftStack Genomics on Cisco Metacloud + SwiftStack Technology is a large component of driving discovery in both research and providing timely answers for clinical treatments. Advances in genomic sequencing have

More information

Towards a Scalable Docker Registry

Towards a Scalable Docker Registry Towards a Scalable Docker Registry Michael B. Littley Thesis submitted to the Faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree

More information

Introduction to Operating Systems. Note Packet # 1. CSN 115 Operating Systems. Genesee Community College. CSN Lab Overview

Introduction to Operating Systems. Note Packet # 1. CSN 115 Operating Systems. Genesee Community College. CSN Lab Overview Introduction to Operating Systems Note Packet # 1 CSN 115 Operating Systems Genesee Community College CSN Lab Overview Dual booted Windows/Linux Workstations Linux workstations currently run CentOS and

More information

Welcome to Docker Birthday # Docker Birthday events (list available at Docker.Party) RSVPs 600 mentors Big thanks to our global partners:

Welcome to Docker Birthday # Docker Birthday events (list available at Docker.Party) RSVPs 600 mentors Big thanks to our global partners: Docker Birthday #3 Welcome to Docker Birthday #3 2 120 Docker Birthday events (list available at Docker.Party) 7000+ RSVPs 600 mentors Big thanks to our global partners: Travel Planet 24 e-food.gr The

More information

Getting Started With Containers

Getting Started With Containers DEVNET 2042 Getting Started With Containers Matt Johnson Developer Evangelist @mattdashj Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session

More information

MEMBRANE: OPERATING SYSTEM SUPPORT FOR RESTARTABLE FILE SYSTEMS

MEMBRANE: OPERATING SYSTEM SUPPORT FOR RESTARTABLE FILE SYSTEMS Department of Computer Science Institute of System Architecture, Operating Systems Group MEMBRANE: OPERATING SYSTEM SUPPORT FOR RESTARTABLE FILE SYSTEMS SWAMINATHAN SUNDARARAMAN, SRIRAM SUBRAMANIAN, ABHISHEK

More information

Understanding the latent value in all content

Understanding the latent value in all content Understanding the latent value in all content John F. Kennedy (JFK) November 22, 1963 INGEST ENRICH EXPLORE Cognitive skills Data in any format, any Azure store Search Annotations Data Cloud Intelligence

More information

VMware Horizon View. VMware Horizon View with Tintri VMstore. TECHNICAL SOLUTION OVERVIEW, Revision 1.1, January 2013

VMware Horizon View. VMware Horizon View with Tintri VMstore. TECHNICAL SOLUTION OVERVIEW, Revision 1.1, January 2013 VMware Horizon View VMware Horizon View with Tintri VMstore TECHNICAL SOLUTION OVERVIEW, Revision 1.1, January 2013 Table of Contents Introduction... 1 Before VDI and VMware Horizon View... 1 Critical

More information

TITLE: PRE-REQUISITE THEORY. 1. Introduction to Hadoop. 2. Cluster. Implement sort algorithm and run it using HADOOP

TITLE: PRE-REQUISITE THEORY. 1. Introduction to Hadoop. 2. Cluster. Implement sort algorithm and run it using HADOOP TITLE: Implement sort algorithm and run it using HADOOP PRE-REQUISITE Preliminary knowledge of clusters and overview of Hadoop and its basic functionality. THEORY 1. Introduction to Hadoop The Apache Hadoop

More information

Announcements. Persistence: Log-Structured FS (LFS)

Announcements. Persistence: Log-Structured FS (LFS) Announcements P4 graded: In Learn@UW; email 537-help@cs if problems P5: Available - File systems Can work on both parts with project partner Watch videos; discussion section Part a : file system checker

More information

ViryaOS RFC: Secure Containers for Embedded and IoT. A proposal for a new Xen Project sub-project

ViryaOS RFC: Secure Containers for Embedded and IoT. A proposal for a new Xen Project sub-project ViryaOS RFC: Secure Containers for Embedded and IoT A proposal for a new Xen Project sub-project Stefano Stabellini @stabellinist The problem Package applications for the target Contain all dependencies

More information

HPC Cloud Burst Using Docker

HPC Cloud Burst Using Docker HPC Cloud Burst Using Docker Khandave Devendra, Kumbhakarn Swati, Kulkarni Shruti, Janbandhu Shreejeet Pune Institute of Computer Technology, Department of Computer Technology, Pune University Pune,India

More information

NVMFS: A New File System Designed Specifically to Take Advantage of Nonvolatile Memory

NVMFS: A New File System Designed Specifically to Take Advantage of Nonvolatile Memory NVMFS: A New File System Designed Specifically to Take Advantage of Nonvolatile Memory Dhananjoy Das, Sr. Systems Architect SanDisk Corp. 1 Agenda: Applications are KING! Storage landscape (Flash / NVM)

More information

Tackling the Reproducibility Problem in Systems Research with Declarative Experiment Specifications

Tackling the Reproducibility Problem in Systems Research with Declarative Experiment Specifications Tackling the Reproducibility Problem in Systems Research with Declarative Experiment Specifications Ivo Jimenez, Carlos Maltzahn (UCSC) Adam Moody, Kathryn Mohror (LLNL) Jay Lofstead (Sandia) Andrea Arpaci-Dusseau,

More information

COS 318: Operating Systems. NSF, Snapshot, Dedup and Review

COS 318: Operating Systems. NSF, Snapshot, Dedup and Review COS 318: Operating Systems NSF, Snapshot, Dedup and Review Topics! NFS! Case Study: NetApp File System! Deduplication storage system! Course review 2 Network File System! Sun introduced NFS v2 in early

More information

Code: Slides:

Code:   Slides: Workshop Resources Code: https://github.com/beekpr/public-workshops Slides: https://tinyurl.com/yc2uo3wk Make sure minikube and kubectl is setup (labs/1-setup-cluster.md has some instructions) Kubernetes

More information

Docker at Lyft Speeding up development Matthew #dockercon

Docker at Lyft Speeding up development Matthew #dockercon Docker at Lyft Speeding up development Matthew Leventi @mleventi #dockercon Lyft Engineering Lyft Engineering Organization - Rapidly growing headcount - Fluid teams - Everyone does devops Technology -

More information

Advanced Continuous Delivery Strategies for Containerized Applications Using DC/OS

Advanced Continuous Delivery Strategies for Containerized Applications Using DC/OS Advanced Continuous Delivery Strategies for Containerized Applications Using DC/OS ContainerCon @ Open Source Summit North America 2017 Elizabeth K. Joseph @pleia2 1 Elizabeth K. Joseph, Developer Advocate

More information

Important DevOps Technologies (3+2+3days) for Deployment

Important DevOps Technologies (3+2+3days) for Deployment Important DevOps Technologies (3+2+3days) for Deployment DevOps is the blending of tasks performed by a company's application development and systems operations teams. The term DevOps is being used in

More information

RED HAT'S CONTAINER STRATEGY. Lars Herrmann General Manager, RHEL, RHEV and Containers June 24, 2015

RED HAT'S CONTAINER STRATEGY. Lars Herrmann General Manager, RHEL, RHEV and Containers June 24, 2015 RED HAT'S CONTAINER STRATEGY Lars Herrmann General Manager, RHEL, RHEV and Containers June 24, 2015 1 DEVELOPMENT VS I.T. OPERATIONS DEVELOPER IT OPERATIONS 2 DEVELOPERS WANT TO GO FAST DEVELOPER 3 HOW

More information

Building/Running Distributed Systems with Apache Mesos

Building/Running Distributed Systems with Apache Mesos Building/Running Distributed Systems with Apache Mesos Philly ETE April 8, 2015 Benjamin Hindman @benh $ whoami 2007-2012 2009-2010 - 2014 my other computer is a datacenter my other computer is a datacenter

More information

Docker Security. Mika Vatanen

Docker Security. Mika Vatanen Docker Security Mika Vatanen 13.6.2017 About me Mika Vatanen, Solution Architect @ Digia 18 years at the industry, 6 months at Digia Established ii2 a Finnish MySpace, top-5 most used web service in Finland

More information

Industry-leading Application PaaS Platform

Industry-leading Application PaaS Platform Industry-leading Application PaaS Platform Solutions Transactional Apps Digital Marketing LoB App Modernization Services Web Apps Web App for Containers API Apps Mobile Apps IDE Enterprise Integration

More information

Docker for HPC? Yes, Singularity! Josef Hrabal

Docker for HPC? Yes, Singularity! Josef Hrabal Docker for HPC? Yes, Singularity! Josef Hrabal IT4Innovations josef.hrabal@vsb.cz support@it4i.cz Virtual Machine Hardware (CPU, Memory, NIC, HDD) Host OS (Windows, Linux, MacOS) Hypervisor (VirtualBox,

More information