Proposal of Live Dump
|
|
|
- Kory Randolf Tucker
- 6 years ago
- Views:
Transcription
1 Proposal of Live Dump YOSHIDA Masanori Yokohama Research Laboratory, Hitachi LinuxCon Japan '12 1
2 Agenda 1.What is Live Dump? 2.Implementation 3.Future work 2
3 1.What is Live Dump? 2.Implementation 3.Future work 3
4 Motivation Server consolidation for very important systems Problem:Availability vs Serviceability Availability We have to keep a host OS running even after some of guests crash. Serviceability We have to obtain memory dump of both guests and a host to make sure to identify cause of the crash. 4
5 Goal Assume one guest crashed but others are still running normally. Guest OS Guest OS Guest OS Host OS 5
6 Goal Guest OS Guest OS Guest OS Running Host OS 6
7 Goal Guest OS Guest OS Guest OS Dump Host OS 7
8 Goal Dump consistent memory snapshot of OS, without stopping the OS. Guest OS Guest OS Guest OS Running Host + OS Dump 8
9 Use case Consolidation for very important systems Guest Guest Guest Host OS 9
10 Use case Non-virtualization case Performance degradation analysis Performance Performance Application Application Bare OS Long run Bare OS 10
11 1.What is Live Dump? 2.Implementation 3.Future work 11
12 Core idea Copy on write Memory space Kernel User Kernel code 12
13 Core idea Copy on write Memory space Kernel User Write protection Kernel code 13
14 Core idea Copy on write Memory space Kernel User Write protection Try to update a page 1 Kernel code 14
15 Core idea Copy on write Memory space Kernel User Write protection Page fault Try to update a page 1 Kernel code 15
16 Core idea Copy on write Memory space Kernel User Write protection 2 Page fault Dump page Try to update a page 1 Kernel code 16
17 Core idea Copy on write Memory space Kernel User Write protection Update the page 3 2 Page fault Dump page Try to update a page 1 Kernel code 17
18 Core idea Copy on write Memory space Kernel User Limitation Kernel code Write Page 3 protection fault 1 Only kernel space 2 is dumped consistently. Save page All phys pages are dumped, but those of user space aren't consistent. 18
19 Kernel virtual space 50% 50% (1) Flow of processing Area to store dumped pages Write protection User 19
20 Kernel virtual space 50% 50% Flow of processing Write protection (2) User Handle on fault 20
21 Kernel virtual space 50% 50% Flow of processing Write protection (2) User Handle on fault 21
22 Kernel virtual space 50% 50% Flow of processing Write protection (2) User Handle on fault 22
23 Kernel virtual space 50% 50% Flow of processing Write protection (2) User Handle on fault 23
24 Kernel virtual space 50% 50% Flow of processing Write protection (2) User Handle on fault 24
25 Kernel virtual space 50% 50% Flow of processing Write protection (2) User Handle on fault 25
26 Kernel virtual space 50% 50% Flow of processing Write protection (2) User Handle on fault 26
27 Kernel virtual space 50% 50% Flow of processing Write protection (3) User Handle on fault Sweep pages 27
28 Kernel virtual space 50% 50% Flow of processing Write protection (3) User Handle on fault Sweep pages 28
29 Overview of livedump 2 parts Protection manager Dump manager ioctl crash tool User Kernel Kick dumping Protection manager Protect pages Handle page fault Callback on fault Export Memory dump Dump manager Save memory dump Export memory dump 29
30 Status of development 1150-line prototype with many limitations Limitation of Protection manager Only supports protection of kernel space Only supports x86-64 architecture Only supports 4K pages Need to split large pages into 4K ones in advance. Limitation of Dump manager Need to allocate 50% of RAM to store dump. 30
31 User space Protection manager Dump manager Ioctl(IOC_SPLIT) Split large pages Ioctl(IOC_WRPROTECT) Stop machine Handle sensitive pages Protect pages Resume machine fn_handle_page (At memory update) Handle page faults fn_handle_page ioctl(ioc_sweep) Sweep pages fn_handle_page crash tool fops->read 31
32 User space Protection manager Dump manager Ioctl(IOC_SPLIT) Split large pages Ioctl(IOC_WRPROTECT) Stop machine Handle sensitive pages Protect pages Resume machine fn_handle_page (At memory update) Handle page faults fn_handle_page ioctl(ioc_sweep) Sweep pages fn_handle_page crash tool fops->read 32
33 Split large pages Livedump can only protect 4K pages. Splitting a large (2M or 1G) page during page fault handling is under development. At the moment, all large pages need to be split into 4K pages by set_memory_4k() in advance. This step will be unnecessary in the future. 33
34 User space Protection manager Dump manager Ioctl(IOC_SPLIT) Split large pages Ioctl(IOC_WRPROTECT) Stop machine Handle sensitive pages Protect pages Resume machine fn_handle_page (At memory update) Handle page faults fn_handle_page ioctl(ioc_sweep) Sweep pages fn_handle_page crash tool fops->read 34
35 Write protection Sequence 1.Stop machine 2.Handle sensitive pages 3.Protect pages 4.Resume machine 35
36 Stop machine We need a consistent memory snapshot, and so must protect all pages while processing is suspended. Livedump simply uses stop_machine() for this purpose. SMP in stop machine Leader CPU: Protect pages All CPUs: Wait for leader's job, and then flush TLBs 36
37 Handle sensitive pages Sensitive pages Following pages cannot be protected. Pages that can be updated in PF handling (This leads to Infinite loop of PF) Pages that can be updated in NMI handling (This leads to nested NMI handling => panic) Such pages are: Kernel/Exception/Interrupt stacks Page table structure task_struct.data section of kernel per_cpu areas 37
38 Handle sensitive pages (Cont'd) Livedump dumps sensitive pages in stop machine. Via callback int fn_handle_page(unsigned long pfn); fn_handle_page(pfn) Defined by Dump manager. Saves content of page. 38
39 Protect pages Manipulation of PTEs(Page Table Entries) 1.Copy _PAGE_RW flag to _PAGE_ORG_RW flag #define _PAGE_ORG_RW _PAGE_UNUSED1 Both flags are in each PTE. This is needed because there can be originally read-only pages. 2.Clear _PAGE_RW flag Make a page read-only. 39
40 How long does it take? Processing of write protection 1.Stop machine 2.Handle sensitive pages 3.Protect pages 4.Resume machine Down time (Stop machine) 40
41 How long does it take? (Cont'd) Down time Is measured with Xeon 2.67 Ghz Increases by 6 msec/gib Processing time [msec] Protected pages [GiB] 41
42 User space Protection manager Dump manager Ioctl(IOC_SPLIT) Split large pages Ioctl(IOC_WRPROTECT) Stop machine Handle sensitive pages Protect pages Resume machine fn_handle_page (At memory update) Handle page faults fn_handle_page ioctl(ioc_sweep) Sweep pages fn_handle_page crash tool fops->read 42
43 Handle page faults Livedump's notifier-call-chain in do_page_fault Check cause of fault Test _PAGE_ORG_RW flag Exclusion control test_and_clear_bit(pfn, pgbmp) Dump the page fn_handle_page(pfn) Unprotect the page Copy back _PAGE_ORG_RW flag to _PAGE_RW flag Clear _PAGE_ORG_RW flag 43
44 User space Protection manager Dump manager Ioctl(IOC_SPLIT) Split large pages Ioctl(IOC_WRPROTECT) Stop machine Handle sensitive pages Protect pages Resume machine fn_handle_page (At memory update) Handle page faults fn_handle_page ioctl(ioc_sweep) Sweep pages fn_handle_page crash tool fops->read 44
45 Sweep pages Batch dumping of all pages not dumped yet. via fn_handle_page() in turn 45
46 User space Protection manager Dump manager Ioctl(IOC_SPLIT) Split large pages Ioctl(IOC_WRPROTECT) Stop machine Handle sensitive pages Protect pages Resume machine fn_handle_page (At memory update) Handle page faults fn_handle_page ioctl(ioc_sweep) Sweep pages fn_handle_page crash tool fops->read 46
47 Dump analysis via crash Dumped pages are exported as a character device. fops->read() fops->llseek() You can use crash with the character device. fn_handle_page Dump manager... fn_handle_page Dump saved on memory crash tool fops->read 47
48 1.What is Live Dump? 2.Implementation 3.Future work 48
49 Large page support Support write-protection of 2M pages To reduce TLB consumption Fix splitting phase Only 1G pages are split to 2M x 512 pages. 2M pages remain 2MB. Fix page fault handling 2MB data are copied on page fault. Copy cost = 200usec (on my Xeon machine) 49
50 On-the-fly dumping to disk At the moment On-the-fly version fn_handle_page... fn_handle_page Memory (50% of RAM) fn_handle_page... fn_handle_page Buf Disk 50
51 Conclusion I developed the prototype of Live Dump Memory dump with a running OS Technique based on CoW Performance (down time) 15 ms with 2GB and increases by 6 ms / GB. Limitation It has many many limitations... 51
52 RFC patchset RFC patchset of livedump has been submitted. On May 25 Please give me feedback! 52
53 Trademarks Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. 53
54 Thank you! 54
Application Fault Tolerance Using Continuous Checkpoint/Restart
Application Fault Tolerance Using Continuous Checkpoint/Restart Tomoki Sekiyama Linux Technology Center Yokohama Research Laboratory Hitachi Ltd. Outline 1. Overview of Application Fault Tolerance and
Performance Prediction and Optimization using Linux/cgroups
Linux Con JAPAN 0(Yokohama) June st 0 Performance Prediction and Optimization using Linux/cgroups Yuzuru Maya Hitachi, Ltd., Yokohama Research Laboratory Agenda Background Outline of Linux/Cgroups Performance
Low-Overhead Ring-Buffer of Kernel Tracing in a Virtualization System
Low-Overhead Ring-Buffer of Kernel Tracing in a Virtualization System Yoshihiro Yunomae Linux Technology Center Yokohama Research Lab. Hitachi, Ltd. 1 Introducing 1. Purpose of a low-overhead ring-buffer
CS 333 Introduction to Operating Systems Class 2 OS-Related Hardware & Software The Process Concept
CS 333 Introduction to Operating Systems Class 2 OS-Related Hardware & Software The Process Concept Jonathan Walpole Computer Science Portland State University 1 Lecture 2 overview OS-Related Hardware
Outline. Low-Level Optimizations in the PowerPC/Linux Kernels. PowerPC Architecture. PowerPC Architecture
Low-Level Optimizations in the PowerPC/Linux Kernels Dr. Paul Mackerras Senior Technical Staff Member IBM Linux Technology Center OzLabs Canberra, Australia paulus@samba.org paulus@au1.ibm.com Introduction
Dan Noé University of New Hampshire / VeloBit
Dan Noé University of New Hampshire / VeloBit A review of how the CPU works The operating system kernel and when it runs User and kernel mode Device drivers Virtualization of memory Virtual memory Paging
instruction is 6 bytes, might span 2 pages 2 pages to handle from 2 pages to handle to Two major allocation schemes
Allocation of Frames How should the OS distribute the frames among the various processes? Each process needs minimum number of pages - at least the minimum number of pages required for a single assembly
Xen Summit Spring 2007
Xen Summit Spring 2007 Platform Virtualization with XenEnterprise Rich Persaud 4/20/07 Copyright 2005-2006, XenSource, Inc. All rights reserved. 1 Xen, XenSource and XenEnterprise
Tutorial on Memory Management, Deadlock and Operating System Types
Operating Systems and Systems Integration Tutorial on Memory Management, Deadlock and Operating System Types 1 Background 1.1 Memory management Virtual memory: is a method of managing memory automatically
CS 261 Fall Mike Lam, Professor. Virtual Memory
CS 261 Fall 2016 Mike Lam, Professor Virtual Memory Topics Operating systems Address spaces Virtual memory Address translation Memory allocation Lingering questions What happens when you call malloc()?
CS399 New Beginnings. Jonathan Walpole
CS399 New Beginnings Jonathan Walpole OS-Related Hardware & Software The Process Concept 2 Lecture 2 Overview OS-Related Hardware & Software - complications in real systems - brief introduction to memory
Shared Virtual Memory Virtualization. Liu, Yi L Raj, Ashok Pan, Jacob
Shared Virtual Memory Virtualization Liu, Yi L yi.l.liu@intel.com Raj, Ashok ashok.raj@intel.com Pan, Jacob jacob.jun.pan@intel.com 1 High-Level View of SVM CPU GVA Device GVA GuestOS Managed VMM Managed
ECE 598 Advanced Operating Systems Lecture 14
ECE 598 Advanced Operating Systems Lecture 14 Vince Weaver http://www.eece.maine.edu/~vweaver vincent.weaver@maine.edu 22 March 2016 Announcements 1 Got a Pi3 over break Pi3 Notes Very impressive performance,
Parallels Workstation 4.0 Extreme Read Me
Parallels Workstation 4.0 Extreme Read Me Welcome to Parallels Workstation Extreme build 4.0.6740. This document contains the information you should know to successfully install Parallels Workstation Extreme
Computer. Operating Systems Introduction to Operating System (OS)
Computer Operating Systems Introduction to Operating System (OS) A computer is a system composed of two major components: hardware and software. Hardware is the physical equipment. Software is the collection
Advanced Memory Management
Advanced Memory Management Main Points Applications of memory management What can we do with ability to trap on memory references to individual pages? File systems and persistent storage Goals Abstractions
Operating Systems CS3502 Spring 2018
Operating Systems CS3502 Spring 2018 Presented by Dr. Guoliang Liu Department of Computer Science College of Computing and Software Engineering Kennesaw State University Computer Systems See Appendix G
SRM-Buffer: An OS Buffer Management SRM-Buffer: An OS Buffer Management Technique toprevent Last Level Cache from Thrashing in Multicores
SRM-Buffer: An OS Buffer Management SRM-Buffer: An OS Buffer Management Technique toprevent Last Level Cache from Thrashing in Multicores Xiaoning Ding The Ohio State University dingxn@cse.ohiostate.edu
PARALLELS SERVER 4.0 FOR MAC BARE METAL EDITION README
PARALLELS SERVER 4.0 FOR MAC BARE METAL EDITION README February 18, 2010 1999-2010 Parallels Holdings, Ltd. and its affiliates. All rights reserved. This document provides the first-priority information
Today. Process Memory Layout. CSCI 4061 Introduction to Operating Systems. Where does the memory come from?
Today CSCI 4061 Introduction to Operating Systems OS Management Virtual Paging and Swapping Instructor: Abhishek Chandra 2 Process Layout Where does the memory come from? High Address Args, environment
Swapping. Jin-Soo Kim Computer Systems Laboratory Sungkyunkwan University
Swapping Jin-Soo Kim (jinsookim@skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu Swapping Support processes when not enough physical memory User program should be independent
CSE 120. Translation Lookaside Buffer (TLB) Implemented in Hardware. July 18, Day 5 Memory. Instructor: Neil Rhodes. Software TLB Management
CSE 120 July 18, 2006 Day 5 Memory Instructor: Neil Rhodes Translation Lookaside Buffer (TLB) Implemented in Hardware Cache to map virtual page numbers to page frame Associative memory: HW looks up in
Swapping. Jinkyu Jeong Computer Systems Laboratory Sungkyunkwan University
Swapping Jinkyu Jeong (jinkyu@skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu EEE0: Introduction to Operating Systems, Fall 07, Jinkyu Jeong (jinkyu@skku.edu) Swapping
Analyzing Kernel Behavior by SystemTap
Analyzing Kernel Behavior by SystemTap Kernel Tracer Approach 2009/2/25 Hitachi, Ltd., Software Division Noboru Obata ( ) Hitachi, Ltd. 2009. All rights reserved. Contents 1. Improving RAS Features for
The Shadowy Depths of the KVM MMU. KVM Forum 2007
The Shadowy Depths of the KVM MMU KVM Forum 2007 Agenda A quick recap of x86 paging Shadow paging in general Goals for the KVM MMU The gory details Paging on x86 Function: translate virtual addresses to
Revisiting Virtual Memory for High Performance Computing on Manycore Architectures: A Hybrid Segmentation Kernel Approach
Revisiting Virtual Memory for High Performance Computing on Manycore Architectures: A Hybrid Segmentation Kernel Approach Yuki Soma, Balazs Gerofi, Yutaka Ishikawa 1 Agenda Background on virtual memory
CS 333 Introduction to Operating Systems. Class 2 OS-Related Hardware & Software The Process Concept
CS 333 Introduction to Operating Systems Class 2 OS-Related Hardware & Software The Process Concept Jonathan Walpole Computer Science Portland State University 1 Administrivia CS333 lecture videos are
CSCE Introduction to Computer Systems Spring 2019
CSCE 313-200 Introduction to Computer Systems Spring 2019 Processes Dmitri Loguinov Texas A&M University January 24, 2019 1 Chapter 3: Roadmap 3.1 What is a process? 3.2 Process states 3.3 Process description
CIS Operating Systems Memory Management Address Translation for Paging. Professor Qiang Zeng Spring 2018
CIS 3207 - Operating Systems Memory Management Address Translation for Paging Professor Qiang Zeng Spring 2018 Previous class What is logical address? Who use it? Describes a location in the logical memory
Optimal Algorithm. Replace page that will not be used for longest period of time Used for measuring how well your algorithm performs
Optimal Algorithm Replace page that will not be used for longest period of time Used for measuring how well your algorithm performs page 1 Least Recently Used (LRU) Algorithm Reference string: 1, 2, 3,
Recall from Tuesday. Our solution to fragmentation is to split up a process s address space into smaller chunks. Physical Memory OS.
Paging 11/10/16 Recall from Tuesday Our solution to fragmentation is to split up a process s address space into smaller chunks. Physical Memory OS Process 3 Process 3 OS: Place Process 3 Process 1 Process
Recall: Address Space Map. 13: Memory Management. Let s be reasonable. Processes Address Space. Send it to disk. Freeing up System Memory
Recall: Address Space Map 13: Memory Management Biggest Virtual Address Stack (Space for local variables etc. For each nested procedure call) Sometimes Reserved for OS Stack Pointer Last Modified: 6/21/2004
Multi-level Translation. CS 537 Lecture 9 Paging. Example two-level page table. Multi-level Translation Analysis
Multi-level Translation CS 57 Lecture 9 Paging Michael Swift Problem: what if you have a sparse address space e.g. out of GB, you use MB spread out need one PTE per page in virtual address space bit AS
SE-292 High Performance Computing. Memory Hierarchy. R. Govindarajan
SE-292 High Performance Computing Memory Hierarchy R. Govindarajan govind@serc Reality Check Question 1: Are real caches built to work on virtual addresses or physical addresses? Question 2: What about
Introduction to Operating Systems (Part II)
Introduction to Operating Systems (Part II) Amir H. Payberah amir@sics.se Amirkabir University of Technology (Tehran Polytechnic) Amir H. Payberah (Tehran Polytechnic) Introduction 1393/6/24 1 / 45 Computer
Virtualization with XEN. Trusted Computing CS599 Spring 2007 Arun Viswanathan University of Southern California
Virtualization with XEN Trusted Computing CS599 Spring 2007 Arun Viswanathan University of Southern California A g e n d a Introduction Virtualization approaches Basic XEN Architecture Setting up XEN Bootstrapping
CIS Operating Systems Memory Management Cache and Demand Paging. Professor Qiang Zeng Spring 2018
CIS 3207 - Operating Systems Memory Management Cache and Demand Paging Professor Qiang Zeng Spring 2018 Process switch Upon process switch what is updated in order to assist address translation? Contiguous
Getting Started with VMware Fusion VMware Fusion for Mac OS X Version 1.0
VMware Fusion for Mac OS X Version 1.0 Getting Started with VMware Fusion Revision: 20070806 Item: VMF-ENG-Q307-295 You can find the most up-to-date technical documentation on our Web site at http://www.vmware.com/support/
ECE 571 Advanced Microprocessor-Based Design Lecture 12
ECE 571 Advanced Microprocessor-Based Design Lecture 12 Vince Weaver http://web.eece.maine.edu/~vweaver vincent.weaver@maine.edu 1 March 2018 HW#6 will be posted Project will be coming up Announcements
About Parallels Desktop 13 for Mac
About Parallels Desktop 13 for Mac Parallels Desktop 13 for Mac is the simplest, most powerful solution for running Windows and Windows applications on Mac without rebooting. For more information, see
CMSC 313 COMPUTER ORGANIZATION & ASSEMBLY LANGUAGE PROGRAMMING LECTURE 27, SPRING 2013
CMSC 313 COMPUTER ORGANIZATION & ASSEMBLY LANGUAGE PROGRAMMING LECTURE 27, SPRING 2013 CACHING Why: bridge speed difference between CPU and RAM Modern RAM allows blocks of memory to be read quickly Principle
Amazon EC2 Deep Dive. Michael #awssummit
Berlin Amazon EC2 Deep Dive Michael Hanisch @hanimic #awssummit Let s get started Amazon EC2 instances AMIs & Virtualization Types EBS-backed AMIs AMI instance Physical host server New root volume snapshot
A Global Operating System for HPC Clusters
A Global Operating System Emiliano Betti 1 Marco Cesati 1 Roberto Gioiosa 2 Francesco Piermaria 1 1 System Programming Research Group, University of Rome Tor Vergata 2 BlueGene Software Division, IBM TJ
ECE 598 Advanced Operating Systems Lecture 14
ECE 598 Advanced Operating Systems Lecture 14 Vince Weaver http://web.eece.maine.edu/~vweaver vincent.weaver@maine.edu 22 March 2018 HW#6 was due. Announcements HW#7 will be posted eventually. Project
CIS Operating Systems Memory Management Cache. Professor Qiang Zeng Fall 2017
CIS 5512 - Operating Systems Memory Management Cache Professor Qiang Zeng Fall 2017 Previous class What is logical address? Who use it? Describes a location in the logical memory address space Compiler
RAS Enhancement Activities for Mission-Critical Linux Systems
RAS Enhancement Activities for MissionCritical Linux Systems Hitachi Ltd. Yoshihiro YUNOMAE 01 MissionCritical Systems We apply Linux to missioncritical systems. Banking systems/carrier backend systems/train
Microkernels. Overview. Required reading: Improving IPC by kernel design
Microkernels Required reading: Improving IPC by kernel design Overview This lecture looks at the microkernel organization. In a microkernel, services that a monolithic kernel implements in the kernel are
@2010 Badri Computer Architecture Assembly II. Virtual Memory. Topics (Chapter 9) Motivations for VM Address translation
Virtual Memory Topics (Chapter 9) Motivations for VM Address translation 1 Motivations for Virtual Memory Use Physical DRAM as a Cache for the Disk Address space of a process can exceed physical memory
VERSION 1.0. VMware Guest SDK. Programming Guide
VERSION 1.0 VMware Guest SDK Programming Guide Please note that you can always find the most up-to-date technical documentation on our Web site at http://www.vmware.com/support/. VMware, Inc. 3145 Porter
PAC094 Performance Tips for New Features in Workstation 5. Anne Holler Irfan Ahmad Aravind Pavuluri
PAC094 Performance Tips for New Features in Workstation 5 Anne Holler Irfan Ahmad Aravind Pavuluri Overview of Talk Virtual machine teams 64-bit guests SMP guests e1000 NIC support Fast snapshots Virtual
Systemwalker Software Configuration Manager. Technical Guide. Windows/Linux
Systemwalker Software Configuration Manager Technical Guide Windows/Linux B1X1-0126-04ENZ0(00) January 2013 Preface Purpose of this Document This document explains the functions of Systemwalker Software
SSD and Container Native Storage for High- Performance Databases
SSD and Native Storage for High- Performance Databases Earle F. Philhower, III Sr. Technical Marketing Manager, Western Digital August 2018 Agenda There Will Be Math * 1 Databases s = Null Set? 2 Integral(VMs
Shadow2. Xen Technical Summit, Summer Tim Deegan (XenSource) & Michael Fetterman (U of Cambridge, Intel) Guilty parties:
Shadow2 Xen Technical Summit, Summer 2006 Guilty parties: Tim Deegan (XenSource) & Michael Fetterman (U of Cambridge, Intel) R Shadow2 Full replacement for the old (shadow1) code. As of about two weeks
Making Applications Mobile
Making Applications Mobile using containers Ottawa Linux Symposium, July 2006 Cedric Le Goater Daniel Lezcano Clement Calmels Dave Hansen
Paging. Jin-Soo Kim Computer Systems Laboratory Sungkyunkwan University
Paging Jin-Soo Kim (jinsookim@skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu Paging Allows the physical address space of a process to be noncontiguous Divide virtual
Profiling Grid Data Transfer Protocols and Servers. George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA
Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA Motivation Scientific experiments are generating large amounts of data Education
Memory Management. Disclaimer: some slides are adopted from book authors slides with permission 1
Memory Management Disclaimer: some slides are adopted from book authors slides with permission 1 Recap Paged MMU: Two main Issues Translation speed can be slow TLB Table size is big Multi-level page table
The Beast We Call A3. CS 161: Lecture 10 3/7/17
The Beast We Call A3 CS 161: Lecture 10 3/7/17 But first... Unconfusing Three Confusions Where does the kernel live? Does every kind of processor use a twolevel page table? Does everything have an address?
COSC3330 Computer Architecture Lecture 20. Virtual Memory
COSC3330 Computer Architecture Lecture 20. Virtual Memory Instructor: Weidong Shi (Larry), PhD Computer Science Department University of Houston Virtual Memory Topics Reducing Cache Miss Penalty (#2) Use
MultiLanes: Providing Virtualized Storage for OS-level Virtualization on Many Cores
MultiLanes: Providing Virtualized Storage for OS-level Virtualization on Many Cores Junbin Kang, Benlong Zhang, Tianyu Wo, Chunming Hu, and Jinpeng Huai Beihang University 夏飞 20140904 1 Outline Background
KVM CPU MODEL IN SYSCALL EMULATION MODE ALEXANDRU DUTU, JOHN SLICE JUNE 14, 2015
KVM CPU MODEL IN SYSCALL EMULATION MODE ALEXANDRU DUTU, JOHN SLICE JUNE 14, 2015 AGENDA Background & Motivation Challenges Native Page Tables Emulating the OS Kernel 2 KVM CPU MODEL IN SYSCALL EMULATION
Find the physical addresses of virtual addresses: 0, 8192, 20, 4100, : : : : : 24684
Memory [KB = 1024 Bytes = 2 10 Bytes, MB = 1024 KB = 2 20 Bytes] 1. Virtual address space =? 64KB Physical address space =? 32KB Page size =? 4KB # of pages =? 16 Frame size =? 4KB # of frames =? 8 Converting
I/O Devices. Nima Honarmand (Based on slides by Prof. Andrea Arpaci-Dusseau)
I/O Devices Nima Honarmand (Based on slides by Prof. Andrea Arpaci-Dusseau) Hardware Support for I/O CPU RAM Network Card Graphics Card Memory Bus General I/O Bus (e.g., PCI) Canonical Device OS reads/writes
Maintaining Linux Long Term & Adding Specific Features in Telecom Systems. Keika Kobayashi NEC Communication Systems Sep 29, Japan2010
Maintaining Linux Long Term & Adding Specific Features in Telecom Systems Keika Kobayashi NEC Communication Systems Sep 29, 2010@LinuxCon Japan2010 OUTLINE 1. Background 2. What we did 1. Keep kernel stable.
HPVM & OpenVMS. Sandeep Ramavana OpenVMS Engineering Sep Germany Technical Update Days 2009
HP & OpenS Sandeep Ramavana OpenS Engineering Sep 2009 Germany Technical Update Days 2009 2009 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice
Virtualization. Operating Systems, 2016, Meni Adler, Danny Hendler & Amnon Meisels
Virtualization Operating Systems, 2016, Meni Adler, Danny Hendler & Amnon Meisels 1 What is virtualization? Creating a virtual version of something o Hardware, operating system, application, network, memory,
Main Memory: Address Translation
Main Memory: Address Translation (Chapter 8) CS 4410 Operating Systems Can t We All Just Get Along? Physical Reality: different processes/threads share the same hardware à need to multiplex CPU (temporal)
Infor M3 on IBM POWER7+ and using Solid State Drives
Infor M3 on IBM POWER7+ and using Solid State Drives IBM Systems & Technology Group Robert Driesch cooter@us.ibm.com This document can be found on the web, Version Date: January 31, 2014 Table of Contents
Operating Systems CMPSCI 377 Spring Mark Corner University of Massachusetts Amherst
Operating Systems CMPSCI 377 Spring 2017 Mark Corner University of Massachusetts Amherst Last Class: Intro to OS An operating system is the interface between the user and the architecture. User-level Applications
About Parallels Desktop 11 for Mac
About Parallels Desktop 11 for Mac Parallels Desktop 11 for Mac is a major upgrade to Parallels award-winning software for running Windows on a Mac. About this Update This update for Parallels Desktop
Chapter 2: Operating-System Structures
Chapter 2: Operating-System Structures Chapter 2: Operating-System Structures Operating System Services User Operating System Interface System Calls Types of System Calls System Programs Operating System
Page Tables. Jinkyu Jeong Computer Systems Laboratory Sungkyunkwan University
Page Tables Jinkyu Jeong (jinkyu@skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu EEE3052: Introduction to Operating Systems, Fall 2017, Jinkyu Jeong (jinkyu@skku.edu)
CS24: INTRODUCTION TO COMPUTING SYSTEMS. Spring 2018 Lecture 23
CS24: INTRODUCTION TO COMPUTING SYSTEMS Spring 208 Lecture 23 LAST TIME: VIRTUAL MEMORY Began to focus on how to virtualize memory Instead of directly addressing physical memory, introduce a level of indirection
Memory Management. An expensive way to run multiple processes: Swapping. CPSC 410/611 : Operating Systems. Memory Management: Paging / Segmentation 1
Memory Management Logical vs. physical address space Fragmentation Paging Segmentation An expensive way to run multiple processes: Swapping swap_out OS swap_in start swapping store memory ready_sw ready
Managing memory in variable sized chunks
Managing memory in variable sized chunks Christopher Yeoh Linux.conf.au 2006 Outline Current Linux/K42 memory management Fragmentation problems Proposed system K42 Architecture/Implementation
Application-Transparent Checkpoint/Restart for MPI Programs over InfiniBand
Application-Transparent Checkpoint/Restart for MPI Programs over InfiniBand Qi Gao, Weikuan Yu, Wei Huang, Dhabaleswar K. Panda Network-Based Computing Laboratory Department of Computer Science & Engineering
Overhead Evaluation about Kprobes and Djprobe (Direct Jump Probe)
Overhead Evaluation about Kprobes and Djprobe (Direct Jump Probe) Masami Hiramatsu Hitachi, Ltd., SDL Jul. 13. 25 1. Abstract To implement flight recorder system, the overhead
More on Address Translation. CS170 Fall T. Yang Some of slides from UCB CS162 by Kubiatowicz
More on Address Translation CS170 Fall 2018. T. Yang Some of slides from UCB CS162 by Kubiatowicz Topics Review of last lecture on pagetable based address translation One-level page table Muti-level paging
Chapter 9: Virtual Memory
Chapter 9: Virtual Memory Chapter 9: Virtual Memory 9.1 Background 9.2 Demand Paging 9.3 Copy-on-Write 9.4 Page Replacement 9.5 Allocation of Frames 9.6 Thrashing 9.7 Memory-Mapped Files 9.8 Allocating
Announcement. Exercise #2 will be out today. Due date is next Monday
Announcement Exercise #2 will be out today Due date is next Monday Major OS Developments 2 Evolution of Operating Systems Generations include: Serial Processing Simple Batch Systems Multiprogrammed Batch
Atos TM Virtualization Solutions
Atos TM Virtualization Solutions Alex Pelster & Mischa van Oijen,12 March 2008 Atos, Atos and fish symbol, Atos Origin and fish symbol, Atos Consulting, and the fish symbol itself are registered trademarks
ibench: Quantifying Interference in Datacenter Applications
ibench: Quantifying Interference in Datacenter Applications Christina Delimitrou and Christos Kozyrakis Stanford University IISWC September 23 th 2013 Executive Summary Problem: Increasing utilization
I/O Systems. Jo, Heeseung
I/O Systems Jo, Heeseung Today's Topics Device characteristics Block device vs. Character device Direct I/O vs. Memory-mapped I/O Polling vs. Interrupts Programmed I/O vs. DMA Blocking vs. Non-blocking
Embedded Systems Dr. Santanu Chaudhury Department of Electrical Engineering Indian Institute of Technology, Delhi
Embedded Systems Dr. Santanu Chaudhury Department of Electrical Engineering Indian Institute of Technology, Delhi Lecture - 13 Virtual memory and memory management unit In the last class, we had discussed
CS162 - Operating Systems and Systems Programming. Address Translation => Paging"
CS162 - Operating Systems and Systems Programming Address Translation => Paging" David E. Culler! http://cs162.eecs.berkeley.edu/! Lecture #15! Oct 3, 2014!! Reading: A&D 8.1-2, 8.3.1. 9.7 HW 3 out (due
Lecture 09: VMs and VCS head in the clouds
Lecture 09: VMs and VCS head in the Hands-on Unix system administration DeCal 2012-10-29 1 / 20 Projects groups of four people submit one form per group with OCF usernames, proposed project ideas, and
CIS Operating Systems Memory Management Address Translation. Professor Qiang Zeng Fall 2017
CIS 5512 - Operating Systems Memory Management Address Translation Professor Qiang Zeng Fall 2017 Outline Fixed partitions Dynamic partitions Con$guous alloca$on: Each process occupies a con$guous memory
OS: An Overview. ICS332 Operating Systems
OS: An Overview ICS332 Operating Systems Why are we studying this? After all, you probably will not develop an OS Important to understand what you use: Develop better (apps); What can and cannot be done;
Address Translation. Jinkyu Jeong Computer Systems Laboratory Sungkyunkwan University
Address Translation Jinkyu Jeong (jinkyu@skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu Today s Topics How to reduce the size of page tables? How to reduce the time for
Virtualization, Xen and Denali
Virtualization, Xen and Denali Susmit Shannigrahi November 9, 2011 Susmit Shannigrahi () Virtualization, Xen and Denali November 9, 2011 1 / 70 Introduction Virtualization is the technology to allow two
CIS Operating Systems Memory Management Cache. Professor Qiang Zeng Fall 2015
CIS 5512 - Operating Systems Memory Management Cache Professor Qiang Zeng Fall 2015 Previous class What is logical address? Who use it? Describes a location in the logical address space Compiler and CPU
Large Page Performance ESX Server 3.5 and ESX Server 3i v3.5
Performance Study Large Page Performance ESX Server 3.5 and ESX Server 3i v3.5 VMware ESX Server supports the use of large pages inside virtual machines. The large page support enables server applications
A Design of Hybrid Operating System for a Parallel Computer with Multi-Core and Many-Core Processors
A Design of Hybrid Operating System for a Parallel Computer with Multi-Core and Many-Core Processors Mikiko Sato 1,5 Go Fukazawa 1 Kiyohiko Nagamine 1 Ryuichi Sakamoto 1 Mitaro Namiki 1,5 Kazumi Yoshinaga
COS 318: Operating Systems
COS 318: Operating Systems OS Structures and System Calls Prof. Margaret Martonosi Computer Science Department Princeton University http://www.cs.princeton.edu/courses/archive/fall11/cos318/ Outline Protection
Increase KVM Performance/Density
Increase KVM Performance/Density with Hyper-V Memory Enlightenments Interface Chao Peng (chao.p.peng@intel.com) Contributors: Chao Gao, Yi Sun Expectations in Memory Virtualization Space Time High Performance
Evaluation of Real-time Performance in Embedded Linux. Hiraku Toyooka, Hitachi. LinuxCon Europe Hitachi, Ltd All rights reserved.
Evaluation of Real-time Performance in Embedded Linux LinuxCon Europe 2014 Hiraku Toyooka, Hitachi 1 whoami Hiraku Toyooka Software engineer at Hitachi " Working on operating systems Linux (mainly) for
Motivation. Threads. Multithreaded Server Architecture. Thread of execution. Chapter 4
Motivation Threads Chapter 4 Most modern applications are multithreaded Threads run within application Multiple tasks with the application can be implemented by separate Update display Fetch data Spell
Process Description and Control. Chapter 3
Process Description and Control Chapter 3 Contents Process states Process description Process control Unix process management Process From processor s point of view execute instruction dictated by program
CSE Opera+ng System Principles
CSE 30341 Opera+ng System Principles Lecture 2 Introduc5on Con5nued Recap Last Lecture What is an opera+ng system & kernel? What is an interrupt? CSE 30341 Opera+ng System Principles 2 1 OS - Kernel CSE
Paging. Jinkyu Jeong Computer Systems Laboratory Sungkyunkwan University
Paging Jinkyu Jeong (jinkyu@skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu SSE3044: Operating Systems, Fall 2016, Jinkyu Jeong (jinkyu@skku.edu) Paging Allows the physical