Walkthrough OCCAM Be on the lookout for this fellow: The callouts are ACTIONs for you to do! When you see the check mark, compare your work to the marked element
Objectives In this presentation you ll 1. Learn what is OCCAM Why is it needed? What does it do? 2. Learn how to use OCCAM As an experimentalist Using the web interface Running and configuring experiments 2
OCCAM and the curation of artifacts
Computer Systems Innovation & Evaluation Relies almost exclusively on artifacts Compilers, analyzers, VMs, Software simulation, Hardware emulation, Benchmarks, mini-apps, Traces, data sets, And a cast of a thousand other tools Empirical study keystone of rapid pace for a huge market (e.g., $111B processor market ) IMS Research, 2011 4
Prolific Artifact Production Simulators 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Single-core X X X.. X X X X X X X X X X X X. X X X. X X X X X X X X X Multi-core X X X.. X X X X X X.. X X... X X X X. X X.. X... Homogen. multi-core X X.. X X X X X X.. X X... X. X X X X... X... FINDABLE Heterogen. multi-core... X.... X............... SMT.. X X. X X X. X...... Shared memory X X.. X X X X X.. X X X X. X X X X X X...... Private memory....................... Timing X X X. X X X X X X X X X X. X X. X X X X. X X. X X. Cycle-accurate. X X. X X X X X X X X X X. X. X X X. X X. X X X Functional. X... X X X X X X X X... X. X.. X X. X. X.. ACCESSIBLE Full system X X.. X X X X X.. X X.... X... X...... Caching.. X X. X X X X X X X X X X X. X. X X X X X. X X X. X Cache coherence... X X X X X.. X X X. X. X. X X... X... In-order..... X X X X X X X X. X. X. X X... X X. X X X Out-of-order..... X X X X X X X X X.. X... X X.. X. X.. Superscalar..... X X X X X X. X X X.. X.. X X X. X. X.. INTEROPERATE I have a new idea. I want to find a simulator, datasets, or other artifact. What s out there? Found it! Can I download? Oh, yes, but it requires Windows! Or, it s missing something. Or I downloaded it! Let me try using it. Hmm. What does this mean? What s the format??? Virtualization............ X.......... Virtual memory. X.. X X X. X X X. X X X.. X...... VLIW........ X.......... X...... DRAM controller X..... X X X X. X X. X... X X X... X. X.. Scheduling..... X X X X. X X.... X X X... X. X.. Concurrency..... X X X.. X X.... X X X........ DRAM error sim....................... REUSABLE Downloaded, installed & ran. Let me try changing it Uh, Oh. Where is that bug? No comments, really?? On-chip network.......... X...... X...... System on chip.................. X...... Power consumption.. X X... X X... X X....... X.. X.. Gate-level....... X............... X.. Validated..... X X. X X. X.. X....... X X. 5
FAIR Guiding Principles 1 Findable: It can be located. 2 Accessible: It can be reached. 3 Interoperable: It can be combined. 4 Reusable: It can be reused, extended. The FAIR principles lead to experimental transparency, reproducibility & reusability M.D. Wilkinson et al., The FAIR Guiding Principles for scientific data management and stewardship, Scientific Data, March 2016 6
What we are doing Our goal: Establish and grow a community, experiments (results) and an ecosystem for modeling and simulation that is FAIR. To be FAIR, we need Active Digital Objects Modifiable, Executable, Interactive Greatly simplify effort and accelerate productivity Leverage & collaborate on high quality models Interoperate & reuse models, both by user & machine May also include hardware resources 7
OCCAM: Open Curation for Computer Architecture Modeling Engage community to learn requirements, needs, impediments and solutions Research, prototype & deploy ideas for enabling FAIR in modeling and simulation 1 Build it to discover what does & doesn t work! 2 Bring value to community to attract contribution 3 Demonstrate the promise & feasibility Educate community about the values of and for FAIR modeling and simulation 8
OCCAM: Open Curation for Computer Architecture Modeling Engage community to learn requirements, needs, impediments and solutions Research, prototype & deploy ideas for enabling FAIR in modeling and simulation 1 Build it to discover what does & doesn t work! 2 Bring value to community to attract contribution 3 Demonstrate the promise & feasibility Educate community about the values of and for FAIR modeling and simulation 9
OCCAM: End-to-End and Lean Digital Curation Active Objects: Artifacts, Experiments & Results 1 Innovate 2 Evaluate 3 Review (AE) 4 Publication (Exp.) 5 Derivation (Exp.) Services & resources for full research lifecycle Sharing and collaboration of content Content is modifiable and runnable 10
Developers OCCAM: End-to-End and Lean Digital Curation Designed around users Multiple user Roles Exchange among users Share & collaborate Services for roles Consumers OCCAM Managers Experimentalists 11
OCCAM: Use Cases in the Research Lifecycle 1 Innovation and experimentation User workspace to manage & conduct research Compose to create experiments to test hypothesis Experiments are workflows Workflows generate results Analyze and visualize results Share with collaborators! 12
OCCAM: Use Cases in the Research Lifecycle 2 Peer review (or validation) Share completed experiments with reviewers Reviewers can view, modify & rerun Artifact Evaluation (http://artifact-eval.org) 3 Dissemination / sharing (post review) Provide access via OCCAM instance Groups of collaborators, or public 4 Publication & Derivation Associate experiments with peer-reviewed article Readers can derive new ideas/results 13
Quick Illustration of Two Uses Demo of Innovation and evaluation Evaluate a continuous query (CQ) database on an emerging memory architecture Two instances of CQ database on separate cores Database is resident in memory What is the effect of contention? Demo of derivation from publication Peer-reviewed article published with experiments Experiments are modifiable, executable content Associated with article through digital library 14
Quick Illustration of Two Uses Demo of Innovation and evaluation Evaluate a continuous query (CQ) database on an emerging memory architecture Two instances of CQ database on separate cores Database is resident in memory What is the effect of contention? Demo of derivation from publication Peer-reviewed article published with experiments Experiments are modifiable, executable content Associated with article through digital library 15
Flexible Deployment 1 Private 2 Cloud 3 Federated 4 Marshaled 5 Specialized developer experimentalist consumer manager web portal OCCAM software services artifact experiment Repository Digital Resources dispatch retrieve benchmark tutorial Simulator on GPs Simulator on CPUs FPGA FPGA data data data data Database servers FPGA FPGA Emulator on FPGAs Hardware Resources developer FPGA experimentalist FPGA consumer Remote hardware instance (e.g., emulator) Remote hardware instance (e.g., high-performance cluster) manager web portal OCCAM software services artifact experiment Repository Digital Resources dispatch retrieve benchmark tutorial Simulator on GPGP Simulator on CPUs FPGA FPGA data data data data Database servers FPGA FPGA Emulator on FPGAs Hardw developer experimentalist consumer web portal OCCAM software services dispatch retrieve data data data data Database servers Simulator on GPs FPGA FP IP IP Remote h instance (e.g benchmark artifact tutorial experiment Simulator on CPUs FPGA IP FPGA IP Demand Resources manager Repository Digital Resources FPGA FPGA Emulator on FPGAs FPGA Hardware Resources Remote hardwa (e.g., high-perform 16
Flexible Deployment 1 Private 2 Cloud 3 Federated 4 Marshaled 5 Specialized 6 Group developer experimentalist consumer manager web portal OCCAM software services artifact experiment Repository Digital Resources dispatch retrieve benchmark tutorial Simulator on GPs Simulator on CPUs FPGA FPGA data data data data Database servers FPGA FPGA Emulator on FPGAs Hardware Resources developer experimentalist consumer developer FPGA experimentalist FPGA consumer Remote hardware instance (e.g., emulator) Remote hardware instance (e.g., high-performance cluster) manager web portal OCCAM software services web portal OCCAM dispatch retrieve software services artifact experiment Repository Digital Resources dispatch retrieve benchmark tutorial data data data data Database servers Simulator on GPs Simulator on GPGP Simulator on CPUs FPGA FPGA data data data data Database servers FPGA FPGA Emulator on FPGAs Hardw FPGA FP IP IP Remote h instance (e.g manager benchmark artifact tutorial experiment Repository Digital Resources Simulator on CPUs FPGA FPGA IP FPGA FPGA Emulator on FPGAs FPGA IP Hardware Resources Remote hardwa (e.g., high-perform 17
Using OCCAM The experimentalist s perspective 18
Step 1: Get a simulator request metadata Obtain simulator OCCAM web portal 19
Step 2: Import to Experiment OCCAM web portal build & check 20
Step 2: Import to Experiment provenance OCCAM web portal build & check bin + env sim build 21
Step 3: Define/Run Experiment provenance define & run experiment Experiment Editor Configuration Editor Form, visual, import OCCAM web portal build & check config instantiate dataset exp.1 exp.1 exp.1 exp.1 sim build 22
Step 3: Define/Run Experiment provenance define & run experiment OCCAM web portal build & check config instantiate dataset exp.1 exp.1 exp.1 exp.1 sim build Pulls simulator + environment Schedules & monitors simulations Extracts results upon completion Everything run in its own separate VM Multiple job types (build, simrun, etc.) 23
Step 3: Define/Run Experiment provenance define & run experiment OCCAM web portal build & check config instantiate dataset exp.1 exp.1 exp.1 exp.1 sim build results 24
Step 4: View Results provenance OCCAM web portal build & check config instantiate dataset sim build create graph view results results Or data 25
Containerization o o OCCAM runs simulators with Docker Lightweight virtualization Local & in-repository build & run Docker is used to create running environments A Docker base image Extended with dependences Docker specification generated 26
Containerization Docker runs a daemon with root privileges. This is similar to vulnerabilities and attack surfaces in traditional virtualization/hypervisor use cases. Only if you allow people to directly interact with the daemon. Only OCCAM has access to the Docker daemon Users only interact with the OCCAM software OCCAM crafts calls to Docker does not allow arbitrary Docker commands Eliminating the known attack surfaces. 27
Containerization Docker Vulnerability Runs daemon as root. Mounting of system files read/write Allows root escalation through the writing of root-owned files OCCAM Solution Run webserver and OCCAM daemon as a different non-interactive user with access to Docker. OCCAM crafts the Docker commands and sanitizes them to prevent mounting of arbitrary directories. This is actually mitigated by using user namespace mapping, which is a Docker feature that isn t on by default. Just turn that on. 28
What we are doing... Learn how to use OCCAM as an experimentalist 1. Import a simulator into OCCAM 2. Create and configure an experiment Experiment: Metadata to run a simulation Benchmark program Simulator Plotting tool Due to time constrains a Docker container with SST is already on your system Docker container 29
What we are doing... Learn how to use OCCAM as an experimentalist 1. Import a simulator into OCCAM 2. Create and configure an experiment Experiment: Metadata to run a simulation 3. Launch (run) experiment Schedules the simulation on a simulation engine 4. Visualise experimental results Interactive plots 30
OCCAM user interfaces OCCAM has two means of interaction CLI Command Line Interface Advanced users (developers) Using scripts WI Web Interface (Graphical) For everyone Create, configure, and run experiments We will be using this for now 31
Accessing the web interface Use your laptop s web browser to access your OCCAM instance Follow instructions in the handout (a) Access to https://<ip ADDRESS> Warning: Your browser may complain about the certificate. When this happens, follow the browser instructions to add an exception. 32
Home page Login button Signup button Search bar 33
Create a user (a) Click Signup (b) Type a username & password (c) Click Sign Up 34
Creating a Workset Worksets are like folders They hold the different OCCAM objects Simulators, experiments, etc. (a) Type a name for a new workset e.g.: Exercise 1 (b) Click to create 35
Workset Workset settings Contributors Content 36
Workset Public/Private 37
Workset Add a bookmark (user screen) 38
Workset Clone the workset 39
Workset View workset history 40
Workset Authors Have permissions to modify all contents of a Workset. Collaborators Can be given have different permissions than authors 41
Workset Import objects (e.g. github) Create new objects (e.g. experiments) 42
Importing existing objects Importing OCCAM objects into a Workset Can be imported from a git repository Lets import an object from bitbucket Find the link in your handouts (a) Choose git repository (b) Type the URL (check your handouts) (c) Click import 43
Importing existing objects You ve just imported a simulator Simple MIPS processor Lets use it in an experiment You ll see the imported simulator here 44
Creating a new experiment You ve just imported a simulator Simple MIPS processor using the SST framework Lets use it in an experiment Lets create an experiment that Runs the simulator you imported Plots the results for inspection 45
Experiments Workflows First create a new experiment (a) Select type experiment (b) Give it a name (c) Click add 46
Experiments Workflow Configurations 47
Experiments Workflow Workflows are DAGs that represent the execution of an experiment This one is empty, lets add the plotting tool Experiments are built from the end 48
Experiments Workflow (a) Click the empty node (b) Enter the type: script (c) Enter the name: plotter (d) Click attach 49
Experiment Workflow The plotting tool is in the workflow! The final block on the workflow represents the output of the experiment 50
Experiments Workflow This plotting tool can plot the output of the simulator The next step is to add the inputs to the plot We ll be adding the simulator we imported However, we need to add an intermediate block Intermediate storage for the simulator s output 51
Experiments Workflow (a) Click the plus sign (b) Enter the type: application/json (c) Select the option: {new application/json} (d) Click attach 52
Experiments Workflow (a) Click the plus sign (b) Enter the type: simulator (c) Enter the name: XSim-demo (d) Click attach 53
Experiments Workflow The type of the intermediate block is the same as the output of our simulator and the input of the plotting tool 54
Experiments Workflow Finally lets add some input to the simulator We ve provided a program to you (a) Click the plus sign (b) Enter the type: program/xsim (c) Enter the name: XSim-program (d) Click attach 55
Experiments Workflow We ve just created a workflow. Now we can configure it 56
Experiments Configurations Xsim-demo configurations Configuration options 57
Experiments Configurations (a) Change the Xsim-demo configurations as you see fit (b) Click update to save any changes 58
Experiments Configurations (a) These options are configured by default (b) DO NOT CHANGE THESE These options select the data to be plotted from the simulation output 59
Experiments Configurations (a) You can also try to change the plotter- Figure Options (b) Click update to save any changes 60
Experiments Configurations The update button saves the changes you make in the experiment! Whenever you change a parameter, you must click this button In every configuration tab 61
Experiments Running Running an experiment is as easy as clicking a button! (a) Click on the run tab 62
Experiments Running Running an experiment is as easy as clicking a button! (or two ) (b) Click run 63
Experiments Running If you see some errors like these, don t worry! Docker is just a bit verbose when checking if the containers exist, and when they do not OCCAM creates them! 64
Experiments Running (a) Once your run is complete, you need to refresh the page to see the results 65
Experiments Running (a) Click the Output tab (b) Open the plot 66
Experiments Running (a) Your plot should look like this (b) Check the provenance tab of the plot 67
Experiments Running (a) You can check how this plot was created The software that created it Inspect the experiment that generated it The machine where it ran 68
Summary We presented OCCAM A community-supported digital curator & exchange for simulation, emulation, benchmarking, experimental results OCCAM allows to easily deploy and run software Making reproducing results easy An fostering software reutilization OCCAM is easy to use But how easy is it to create a simulation in OCCAM? We ll create a simulation in the next exercise 69
Acknowledgments http://sst-simulator.org http://occam.cs.pitt.edu We gratefully acknowledge the support and collaboration of our research partners. Portions of this material is based in part upon work supported by the National Science Foundation under Grant Numbers CCF-142331, CNS-1012070, ACI-1535232, CNS- 1305220 and CCF-1148646. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. 70