Walkthrough OCCAM. Be on the lookout for this fellow: The callouts are ACTIONs for you to do!

Transcription

1 Walkthrough OCCAM Be on the lookout for this fellow: The callouts are ACTIONs for you to do! When you see the check mark, compare your work to the marked element

2 Objectives In this presentation you ll 1. Learn what is OCCAM Why is it needed? What does it do? 2. Learn how to use OCCAM As an experimentalist Using the web interface Running and configuring experiments 2

3 OCCAM and the curation of artifacts

4 Computer Systems Innovation & Evaluation Relies almost exclusively on artifacts Compilers, analyzers, VMs, Software simulation, Hardware emulation, Benchmarks, mini-apps, Traces, data sets, And a cast of a thousand other tools Empirical study keystone of rapid pace for a huge market (e.g., $111B processor market ) IMS Research,

5 Prolific Artifact Production Simulators Single-core X X X.. X X X X X X X X X X X X. X X X. X X X X X X X X X Multi-core X X X.. X X X X X X.. X X... X X X X. X X.. X... Homogen. multi-core X X.. X X X X X X.. X X... X. X X X X... X... FINDABLE Heterogen. multi-core... X.... X SMT.. X X. X X X. X Shared memory X X.. X X X X X.. X X X X. X X X X X X Private memory Timing X X X. X X X X X X X X X X. X X. X X X X. X X. X X. Cycle-accurate. X X. X X X X X X X X X X. X. X X X. X X. X X X Functional. X... X X X X X X X X... X. X.. X X. X. X.. ACCESSIBLE Full system X X.. X X X X X.. X X.... X... X Caching.. X X. X X X X X X X X X X X. X. X X X X X. X X X. X Cache coherence... X X X X X.. X X X. X. X. X X... X... In-order..... X X X X X X X X. X. X. X X... X X. X X X Out-of-order..... X X X X X X X X X.. X... X X.. X. X.. Superscalar..... X X X X X X. X X X.. X.. X X X. X. X.. INTEROPERATE I have a new idea. I want to find a simulator, datasets, or other artifact. What s out there? Found it! Can I download? Oh, yes, but it requires Windows! Or, it s missing something. Or I downloaded it! Let me try using it. Hmm. What does this mean? What s the format??? Virtualization X Virtual memory. X.. X X X. X X X. X X X.. X VLIW X X DRAM controller X..... X X X X. X X. X... X X X... X. X.. Scheduling..... X X X X. X X.... X X X... X. X.. Concurrency..... X X X.. X X.... X X X DRAM error sim REUSABLE Downloaded, installed & ran. Let me try changing it Uh, Oh. Where is that bug? No comments, really?? On-chip network X X System on chip X Power consumption.. X X... X X... X X X.. X.. Gate-level X X.. Validated..... X X. X X. X.. X X X. 5

6 FAIR Guiding Principles 1 Findable: It can be located. 2 Accessible: It can be reached. 3 Interoperable: It can be combined. 4 Reusable: It can be reused, extended. The FAIR principles lead to experimental transparency, reproducibility & reusability M.D. Wilkinson et al., The FAIR Guiding Principles for scientific data management and stewardship, Scientific Data, March

7 What we are doing Our goal: Establish and grow a community, experiments (results) and an ecosystem for modeling and simulation that is FAIR. To be FAIR, we need Active Digital Objects Modifiable, Executable, Interactive Greatly simplify effort and accelerate productivity Leverage & collaborate on high quality models Interoperate & reuse models, both by user & machine May also include hardware resources 7

8 OCCAM: Open Curation for Computer Architecture Modeling Engage community to learn requirements, needs, impediments and solutions Research, prototype & deploy ideas for enabling FAIR in modeling and simulation 1 Build it to discover what does & doesn t work! 2 Bring value to community to attract contribution 3 Demonstrate the promise & feasibility Educate community about the values of and for FAIR modeling and simulation 8

9 OCCAM: Open Curation for Computer Architecture Modeling Engage community to learn requirements, needs, impediments and solutions Research, prototype & deploy ideas for enabling FAIR in modeling and simulation 1 Build it to discover what does & doesn t work! 2 Bring value to community to attract contribution 3 Demonstrate the promise & feasibility Educate community about the values of and for FAIR modeling and simulation 9

10 OCCAM: End-to-End and Lean Digital Curation Active Objects: Artifacts, Experiments & Results 1 Innovate 2 Evaluate 3 Review (AE) 4 Publication (Exp.) 5 Derivation (Exp.) Services & resources for full research lifecycle Sharing and collaboration of content Content is modifiable and runnable 10

11 Developers OCCAM: End-to-End and Lean Digital Curation Designed around users Multiple user Roles Exchange among users Share & collaborate Services for roles Consumers OCCAM Managers Experimentalists 11

12 OCCAM: Use Cases in the Research Lifecycle 1 Innovation and experimentation User workspace to manage & conduct research Compose to create experiments to test hypothesis Experiments are workflows Workflows generate results Analyze and visualize results Share with collaborators! 12

13 OCCAM: Use Cases in the Research Lifecycle 2 Peer review (or validation) Share completed experiments with reviewers Reviewers can view, modify & rerun Artifact Evaluation ( 3 Dissemination / sharing (post review) Provide access via OCCAM instance Groups of collaborators, or public 4 Publication & Derivation Associate experiments with peer-reviewed article Readers can derive new ideas/results 13

14 Quick Illustration of Two Uses Demo of Innovation and evaluation Evaluate a continuous query (CQ) database on an emerging memory architecture Two instances of CQ database on separate cores Database is resident in memory What is the effect of contention? Demo of derivation from publication Peer-reviewed article published with experiments Experiments are modifiable, executable content Associated with article through digital library 14

15 Quick Illustration of Two Uses Demo of Innovation and evaluation Evaluate a continuous query (CQ) database on an emerging memory architecture Two instances of CQ database on separate cores Database is resident in memory What is the effect of contention? Demo of derivation from publication Peer-reviewed article published with experiments Experiments are modifiable, executable content Associated with article through digital library 15

16 Flexible Deployment 1 Private 2 Cloud 3 Federated 4 Marshaled 5 Specialized developer experimentalist consumer manager web portal OCCAM software services artifact experiment Repository Digital Resources dispatch retrieve benchmark tutorial Simulator on GPs Simulator on CPUs FPGA FPGA data data data data Database servers FPGA FPGA Emulator on FPGAs Hardware Resources developer FPGA experimentalist FPGA consumer Remote hardware instance (e.g., emulator) Remote hardware instance (e.g., high-performance cluster) manager web portal OCCAM software services artifact experiment Repository Digital Resources dispatch retrieve benchmark tutorial Simulator on GPGP Simulator on CPUs FPGA FPGA data data data data Database servers FPGA FPGA Emulator on FPGAs Hardw developer experimentalist consumer web portal OCCAM software services dispatch retrieve data data data data Database servers Simulator on GPs FPGA FP IP IP Remote h instance (e.g benchmark artifact tutorial experiment Simulator on CPUs FPGA IP FPGA IP Demand Resources manager Repository Digital Resources FPGA FPGA Emulator on FPGAs FPGA Hardware Resources Remote hardwa (e.g., high-perform 16

17 Flexible Deployment 1 Private 2 Cloud 3 Federated 4 Marshaled 5 Specialized 6 Group developer experimentalist consumer manager web portal OCCAM software services artifact experiment Repository Digital Resources dispatch retrieve benchmark tutorial Simulator on GPs Simulator on CPUs FPGA FPGA data data data data Database servers FPGA FPGA Emulator on FPGAs Hardware Resources developer experimentalist consumer developer FPGA experimentalist FPGA consumer Remote hardware instance (e.g., emulator) Remote hardware instance (e.g., high-performance cluster) manager web portal OCCAM software services web portal OCCAM dispatch retrieve software services artifact experiment Repository Digital Resources dispatch retrieve benchmark tutorial data data data data Database servers Simulator on GPs Simulator on GPGP Simulator on CPUs FPGA FPGA data data data data Database servers FPGA FPGA Emulator on FPGAs Hardw FPGA FP IP IP Remote h instance (e.g manager benchmark artifact tutorial experiment Repository Digital Resources Simulator on CPUs FPGA FPGA IP FPGA FPGA Emulator on FPGAs FPGA IP Hardware Resources Remote hardwa (e.g., high-perform 17

18 Using OCCAM The experimentalist s perspective 18

19 Step 1: Get a simulator request metadata Obtain simulator OCCAM web portal 19

20 Step 2: Import to Experiment OCCAM web portal build & check 20

21 Step 2: Import to Experiment provenance OCCAM web portal build & check bin + env sim build 21

22 Step 3: Define/Run Experiment provenance define & run experiment Experiment Editor Configuration Editor Form, visual, import OCCAM web portal build & check config instantiate dataset exp.1 exp.1 exp.1 exp.1 sim build 22

23 Step 3: Define/Run Experiment provenance define & run experiment OCCAM web portal build & check config instantiate dataset exp.1 exp.1 exp.1 exp.1 sim build Pulls simulator + environment Schedules & monitors simulations Extracts results upon completion Everything run in its own separate VM Multiple job types (build, simrun, etc.) 23

24 Step 3: Define/Run Experiment provenance define & run experiment OCCAM web portal build & check config instantiate dataset exp.1 exp.1 exp.1 exp.1 sim build results 24

25 Step 4: View Results provenance OCCAM web portal build & check config instantiate dataset sim build create graph view results results Or data 25

26 Containerization o o OCCAM runs simulators with Docker Lightweight virtualization Local & in-repository build & run Docker is used to create running environments A Docker base image Extended with dependences Docker specification generated 26

27 Containerization Docker runs a daemon with root privileges. This is similar to vulnerabilities and attack surfaces in traditional virtualization/hypervisor use cases. Only if you allow people to directly interact with the daemon. Only OCCAM has access to the Docker daemon Users only interact with the OCCAM software OCCAM crafts calls to Docker does not allow arbitrary Docker commands Eliminating the known attack surfaces. 27

28 Containerization Docker Vulnerability Runs daemon as root. Mounting of system files read/write Allows root escalation through the writing of root-owned files OCCAM Solution Run webserver and OCCAM daemon as a different non-interactive user with access to Docker. OCCAM crafts the Docker commands and sanitizes them to prevent mounting of arbitrary directories. This is actually mitigated by using user namespace mapping, which is a Docker feature that isn t on by default. Just turn that on. 28

29 What we are doing... Learn how to use OCCAM as an experimentalist 1. Import a simulator into OCCAM 2. Create and configure an experiment Experiment: Metadata to run a simulation Benchmark program Simulator Plotting tool Due to time constrains a Docker container with SST is already on your system Docker container 29

30 What we are doing... Learn how to use OCCAM as an experimentalist 1. Import a simulator into OCCAM 2. Create and configure an experiment Experiment: Metadata to run a simulation 3. Launch (run) experiment Schedules the simulation on a simulation engine 4. Visualise experimental results Interactive plots 30

31 OCCAM user interfaces OCCAM has two means of interaction CLI Command Line Interface Advanced users (developers) Using scripts WI Web Interface (Graphical) For everyone Create, configure, and run experiments We will be using this for now 31

32 Accessing the web interface Use your laptop s web browser to access your OCCAM instance Follow instructions in the handout (a) Access to ADDRESS> Warning: Your browser may complain about the certificate. When this happens, follow the browser instructions to add an exception. 32

33 Home page Login button Signup button Search bar 33

34 Create a user (a) Click Signup (b) Type a username & password (c) Click Sign Up 34

35 Creating a Workset Worksets are like folders They hold the different OCCAM objects Simulators, experiments, etc. (a) Type a name for a new workset e.g.: Exercise 1 (b) Click to create 35

36 Workset Workset settings Contributors Content 36

37 Workset Public/Private 37

38 Workset Add a bookmark (user screen) 38

39 Workset Clone the workset 39

40 Workset View workset history 40

41 Workset Authors Have permissions to modify all contents of a Workset. Collaborators Can be given have different permissions than authors 41

42 Workset Import objects (e.g. github) Create new objects (e.g. experiments) 42

43 Importing existing objects Importing OCCAM objects into a Workset Can be imported from a git repository Lets import an object from bitbucket Find the link in your handouts (a) Choose git repository (b) Type the URL (check your handouts) (c) Click import 43

44 Importing existing objects You ve just imported a simulator Simple MIPS processor Lets use it in an experiment You ll see the imported simulator here 44

45 Creating a new experiment You ve just imported a simulator Simple MIPS processor using the SST framework Lets use it in an experiment Lets create an experiment that Runs the simulator you imported Plots the results for inspection 45

46 Experiments Workflows First create a new experiment (a) Select type experiment (b) Give it a name (c) Click add 46

47 Experiments Workflow Configurations 47

48 Experiments Workflow Workflows are DAGs that represent the execution of an experiment This one is empty, lets add the plotting tool Experiments are built from the end 48

49 Experiments Workflow (a) Click the empty node (b) Enter the type: script (c) Enter the name: plotter (d) Click attach 49

50 Experiment Workflow The plotting tool is in the workflow! The final block on the workflow represents the output of the experiment 50

51 Experiments Workflow This plotting tool can plot the output of the simulator The next step is to add the inputs to the plot We ll be adding the simulator we imported However, we need to add an intermediate block Intermediate storage for the simulator s output 51

52 Experiments Workflow (a) Click the plus sign (b) Enter the type: application/json (c) Select the option: {new application/json} (d) Click attach 52

53 Experiments Workflow (a) Click the plus sign (b) Enter the type: simulator (c) Enter the name: XSim-demo (d) Click attach 53

54 Experiments Workflow The type of the intermediate block is the same as the output of our simulator and the input of the plotting tool 54

55 Experiments Workflow Finally lets add some input to the simulator We ve provided a program to you (a) Click the plus sign (b) Enter the type: program/xsim (c) Enter the name: XSim-program (d) Click attach 55

56 Experiments Workflow We ve just created a workflow. Now we can configure it 56

57 Experiments Configurations Xsim-demo configurations Configuration options 57

58 Experiments Configurations (a) Change the Xsim-demo configurations as you see fit (b) Click update to save any changes 58

59 Experiments Configurations (a) These options are configured by default (b) DO NOT CHANGE THESE These options select the data to be plotted from the simulation output 59

60 Experiments Configurations (a) You can also try to change the plotter- Figure Options (b) Click update to save any changes 60

61 Experiments Configurations The update button saves the changes you make in the experiment! Whenever you change a parameter, you must click this button In every configuration tab 61

62 Experiments Running Running an experiment is as easy as clicking a button! (a) Click on the run tab 62

63 Experiments Running Running an experiment is as easy as clicking a button! (or two ) (b) Click run 63

64 Experiments Running If you see some errors like these, don t worry! Docker is just a bit verbose when checking if the containers exist, and when they do not OCCAM creates them! 64

65 Experiments Running (a) Once your run is complete, you need to refresh the page to see the results 65

66 Experiments Running (a) Click the Output tab (b) Open the plot 66

67 Experiments Running (a) Your plot should look like this (b) Check the provenance tab of the plot 67

68 Experiments Running (a) You can check how this plot was created The software that created it Inspect the experiment that generated it The machine where it ran 68

69 Summary We presented OCCAM A community-supported digital curator & exchange for simulation, emulation, benchmarking, experimental results OCCAM allows to easily deploy and run software Making reproducing results easy An fostering software reutilization OCCAM is easy to use But how easy is it to create a simulation in OCCAM? We ll create a simulation in the next exercise 69

70 Acknowledgments We gratefully acknowledge the support and collaboration of our research partners. Portions of this material is based in part upon work supported by the National Science Foundation under Grant Numbers CCF , CNS , ACI , CNS and CCF Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. 70