CIS : Computational Reproducibility

Transcription

1 CIS : Computational Reproducibility Virtual Machines Dr. David Koop

2 Figure 2. The MODIS grid, with highlighted tiles (red) of spatial extent for California (green), with citation. Computational Data Citation (MODIS) n a b s s n e t q t s a n i l t t [Buneman et al., 2016] c 2 t

3 Citation Rule and Sample Result (MODIS) { author: m_auth($p,$$v), m_year:($p,$$v), title: m_title($p), version: $v, bounding-box : [$$minlong, $$minlat, $$maxlong, $$maxlat], interval: [$$mint, $$maxt], organization: m_org($p), url: m_url($p), accessed: DATE(), doi = m_doi($p,$$v) } /root/product[prodname=$p]/version[vnum=$$v] /file[lat $$minlat and Lat $$maxlat and Lon $$minlon and Lon $$maxlon and Time $$mint and Time $$maxt] { author: E. Vermote, title: MOD09A1... SIN Grid, version: 6, bounding-box: [-125, 32, -119, 42], interval: [ , ], organization: NASA EOSDIS... South Dakota, URL: lpdaac.usgs.gov, accessed: , doi: /MODIS/MOD09A1.006 } [Buneman et al., 2016] 3

4 Cyberinfrastructure Metadata - Standards for specific domains - Tools need to support this Persistent Unique Identifiers - DOIs - ORCHIDs Data Repositories - Federated Stores (e.g. DataONE) - Registry of Data Stores 4

5 Dryad Repository [Koskela et al., 2013] 5

6 ONEShare: Make it easy to add data Data'&'' Metadata'(EML)' [Koskela et al., 2013] 6

7 DataONE Goals Provide(universal(access(to(data(about(life(on(earth(and(the(environment& 1.&&Building&community& 2.&&Developing&sustainable& data&discovery&and& interoperability&soludons& 3.&Enabling&science&through& tools&and&services& Plan& Analyze& Collect& Integrate& Assure& Discover& Describe' Preserve& [Koskela et al., 2013] 7

8 DataONE Coordinating Nodes Three&major&components&for&a& flexible,&scalable,&sustainable& network& Coordina-ng'Nodes' retain&complete&metadata& catalog&& indexing&for&search& networkiwide&services& ensure&content&availability& (preservadon)&&& replicadon&services& [Koskela et al., 2013] 8

9 DataONE Member Nodes Three&major&components&for&a& flexible,&scalable,&sustainable& network& Coordina-ng'Nodes' retain&complete&metadata& Member'Nodes' catalog&& diverse&insdtudons& indexing&for&search& serve&local&community& networkiwide&services& provide&resources&for& managing&their&data& ensure&content&availability& (preservadon)&&& retain&copies&of&data& replicadon&services& [Koskela et al., 2013] 9

10 DataONE Investigator Toolkit Three&major&components&for&a& flexible,&scalable,&sustainable& network& Coordina-ng'Nodes' retain&complete&metadata& Member'Nodes' catalog&& diverse&insdtudons& Inves-gator'Toolkit' indexing&for&search& serve&local&community& ' networkiwide&services& provide&resources&for& managing&their&data& ensure&content&availability& (preservadon)&&& retain&copies&of&data& replicadon&services& [Koskela et al., 2013] 10

11 DataONE Architecture ORNL&DAAC& FGDC,&ISO,& DIF,&FGDC& USGS&CSAS& KNB& PISCO& SANParks& ESA& ONEShare& UC&Merrik& CLO/AKN& FGDC,&ISO,& FGDC& EML,&ISO& EML& EML,&FGDC& EML& EML& EML& EML& Extract&and&Align&Metadata& Augment&Metadata& Internal& Metadata& Index& Search& API& LTER& EML& 15& [Koskela et al., 2013] 11

12 Member Node Users InformaDon&&&&&&& Center&for&the& Environment& (ICE)&UC&Davis& ICE&Users& agencies& cidzens& faculty& ICE&Collects&Water&Data&& ICE&Collectors& ICE&Users& Inves-gator'Toolkit' DataONE&Users& 16& [Koskela et al., 2013] 12

13 re3data Registry of Research Data Repositories Index repositories, not the data Provide tools for users to find repositories of interest Maintain metadata and badges about repositories 13

14 Badges [re3data.org ] 14

15 Project Find some papers that you may be interested in reproducing Do a survey of the material that is available for each paper: - Code? Is the code under version control? - Data? Is it clear how to process or understand the data? Is there metadata? - Virtual machine or container? Does the hardware/software that deals with these still work? - Provenance? Do we have a record of the steps taken in producing a result? How complete is it? 15

16 Project If you are interested in a topic that aligns with reproducibility, please me/talk to me about your ideas For example, if you are working on a research project that could incorporate reproducibility Formal Specification and Initial Deadline Soon 16

17 Midterm Tuesday, October 25, Dion 101, 3:30-4:45pm Format: Multiple Choice + Free Response Focus on the topics we have covered and the associated papers - Reproducibility Themes - Scientific Writing - Version Control - Data Sharing, Citation, Repositories - Virtual Machines May include material not in the assigned papers but discussed in class (e.g. the Computer Systems Reproducibility work) See sample questions on the course web site 17

18 What is a virtual machine? 18

19 ` E. de Lara based on material by C. Waldspurger N. B. Sahgal and D. Rodgers J. Nakajima

20 Xen Design Principles 1. Support for unmodified application binaries is essential, or users will not transition to Xen. Hence we must virtualize all architectural features required by existing standard ABIs. 2. Supporting full multi-application operating systems is important, as this allows complex server configurations to be virtualized within a single guest OS instance. 3. Paravirtualization is necessary to obtain high performance and strong resource isolation on uncooperative machine architectures such as x Even on cooperative machine architectures, completely hiding the effects of resource virtualization from guest OSes risks both correctness and performance. 20

21 Xen's Paravirtualized x86 Interface Memory Management Segmentation Paging CPU Protection Exceptions System Calls Interrupts Time Device I/O Network, Disk, etc. Cannot install fully-privileged segment descriptors and cannot overlap with the top end of the linear address space. Guest OS has direct read access to hardware page tables, but updates are batched and validated by the hypervisor. A domain may be allocated discontiguous machine pages. Guest OS must run at a lower privilege level than Xen. Guest OS must register a descriptor table for exception handlers with Xen. Aside from page faults, the handlers remain the same. Guest OS may install a fast handler for system calls, allowing direct calls from an application into its guest OS and avoiding indirecting through Xen on every call. Hardware interrupts are replaced with a lightweight event system. Each guest OS has a timer interface and is aware of both real and virtual time. Virtual devices are elegant and simple to access. Data is transferred using asynchronous I/O rings. An event mechanism replaces hardware interrupts for notifications. 21

22 Changed Existing Host OS Required porting, but wasn't that much code: OS subsection # lines Linux XP Architecture-independent Virtual network driver 484 Virtual block-device driver 1070 Xen-specific (non-driver) Total (Portion of total x86 code base 1.36% 0.04%) 22

23 Structure of Host Machine (XenoLinux) Control Plane Software User Software User Software User Software GuestOS (XenoLinux) GuestOS (XenoLinux) GuestOS (XenoBSD) GuestOS (XenoXP) Xeno-Aware Device Drivers Xeno-Aware Device Drivers Xeno-Aware Device Drivers Xeno-Aware Device Drivers Domain0 control interface virtual x86 CPU virtual phy mem virtual network virtual blockdev X E N H/W (SMP x86, phy mem, enet, SCSI/IDE) 23

24 Current Architecture [Xen Project Software Overview] 24

25 Performance Relative score to Linux L X V U L X V U L X V U L X V U L X V U L X V U SPEC INT2000 (score) Linux build time (s) OSDB-IR (tup/s) OSDB-OLTP (tup/s) dbench (score) SPEC WEB99 (score) Figure 3: Relative performance of native Linux (L), XenoLinux (X), VMware workstation 3.2 (V) and User-Mode Linux (U). 25

26 Today The Cloud - Amazon EC2 uses hypervisors for virtual private servers - Other cloud providers also use Xen Linux 3.0 kernels provide support for Xen 26

27 Virtualization Properties Isolation - Fault isolation - Performance isolation Encapsulation - Cleanly capture all VM state - Enables VM snapshots, clones Portability - Independent of physical hardware - Enables migration of live, running VMs Interposition - Transformations on instructions, memory, I/O - Enables transparent resource overcommitment, encryption, compression, replication 27

28 How do these match/not match reproducibility? Isolation Encapsulation Portability Interposition 28

29 Next Class How Cloud Computing (via Virtualization) can Improve Reproducibility Do you agree with the cited improvements? Are the remaining challenges valid? Do you agree that the non-challenges indeed are not challenges? 29