Radix Tree, IDR APIs and their test suite. Rehas Sachdeva & Sandhya Bankar
|
|
- Melvyn King
- 6 years ago
- Views:
Transcription
1 Radix Tree, IDR APIs and their test suite Rehas Sachdeva & Sandhya Bankar
2 Introduction Outreachy intern Dec 2016-March 2017 for Linux kernel, mentored by Rik van Riel and Matthew Wilcox. 4th year undergrad student of Computer Science and Engineering at International Institute of information technology, Hyderabad, India.
3 Overview RADIX TREE What is a Radix tree? Applications of radix tree Kernel radix tree API Enhancing the test suite IDR Allocate and manage file descriptor using IDR. IDR API used in project Testing Result Conclusion
4 What is a Radix tree?
5 Radix tree Stores a key to value mapping. Root to leaf path holds the key and the leaf holds the value. the edges are labelled by a sequence of characters or bits. Space optimized. Fast lookup. Space optimized trie
6 Radix tree applications General applications IP routing hierarchical organization of IP addresses. Search inverted indexes for text documents Kernel specific uses Page Cache Check presence in cache, dirty tag or under writeback etc. As resizeable arrays drivers, filesystems, interrupt controllers.
7 Kernel radix tree API
8 Node structure Each node contains (2^map_shift) pointers in slots array. Slots point to an item in the leaf node, and next, deeper node, in an internal node. Depth of node ~ which chunk of bits of key is used to index the slots. #define RADIX_TREE_MAP_SIZE Node Info: shift, offset, count, parent pointer, root pointer, tags etc. (1UL << RADIX_TREE_MAP_SHIFT)... struct radix_tree_node { unsigned char shift; /* Bits remaining in each slot */ unsigned char offset; /* Slot offset in parent */ unsigned char count; /* Total entry count */ unsigned char exceptional; /* Exceptional entry count */ Array of slots... void rcu *slots[radix_tree_map_size]; unsigned long tags[radix_tree_max_tags][radix_tree_tag_longs]; };
9 Initializing a radix tree #define RADIX_TREE(name, mask) \ struct radix_tree_root name = RADIX_TREE_INIT(mask) Example: RADIX_TREE(tree, GFP_KERNEL); initializes a radix tree with the given name. gfp_mask to tell the code how memory allocations are to be performed. GFP_ATOMIC for atomic insertions, GFP_KERNEL for kernel-internal allocations and so on.
10 Inserting an entry A tree of height N can contain any index between 0 and (2^(map_shift*N))-1. If the new index to be inserted is larger than the current max index, insert new nodes above the current top node to create a deeper tree. Failure cases: should a memory allocation fail (-ENOMEM) or an entry already exists at the index (-EEXIST).
11 Inserting an entry Consider the following tree as example. Only 1 bit is used to index the slots at each node.
12 Inserting an entry H is inserted, only first 2 bits need to be considered to uniquely lookup for it.
13 Inserting an entry I is inserted. Nodes are created as all 5 bits need to be considered.
14 Inserting an entry root: index: order: item: radix tree root index key key covers the 2^order indices around index item to insert static inline int radix_tree_insert(struct radix_tree_root *root, unsigned long index, void *entry); For inserting an entry. Wrapper around radix_tree_insert for 0 order entry. int radix_tree_insert(struct radix_tree_root *, unsigned long index, unsigned order, void *item); For inserting an entry of arbitrary order.
15 Deleting an entry If deleting an element results in a top node with only one child at offset 0, replace the top node with its only child, creating a shallower tree. Consider the following tree as example.
16 Deleting an entry If deleting an element results in a top node with only one child at offset 0, replace the top node with its only child, creating a shallower tree. Consider the following tree as example.
17 Deleting an entry If deleting an element results in a top node with only one child at offset 0, replace the top node with its only child, creating a shallower tree. Consider the following tree as example.
18 Deleting an entry root: index: radix tree root index key void *radix_tree_delete(struct radix_tree_root *root, unsigned long index); item: void *radix_tree_delete_item(struct radix_tree_root *root, unsigned long index, void *item); Delete if the entry at index is expected item. iter: slot: void radix_tree_iter_delete(struct radix_tree_root *root, struct radix_tree_iter *iter, void rcu **slot); Delete the entry at this iterator position expected item iterator state pointer to slot
19 Lookup root: index: radix tree root index key void *radix_tree_lookup(const struct radix_tree_root *root, unsigned long index); looks for key in the tree and returns the associated item (or NULL on failure). results: first_index: max_items: unsigned int radix_tree_gang_lookup(const struct radix_tree_root *root, void **results, unsigned long first_index, unsigned int max_items); perform multiple lookups. void rcu **radix_tree_lookup_slot(const struct radix_tree_root *root, unsigned long index); lookup a slot at index. where the results of the lookup are placed start the lookup from this key place up to this many items at *results
20 Iteration slot: root: iter: start: Iterate over non-empty slots #define radix_tree_for_each_slot(slot, root, iter, start) for (slot = radix_tree_iter_init(iter, start) ; slot (slot = radix_tree_next_chunk(root, iter, 0)) ; slot = radix_tree_next_slot(slot, iter, 0)) iterate over non-empty slots the void** variable for pointer to slot the struct radix_tree_root pointer the struct radix_tree_iter pointer iteration starting index \ \ \ Iterate over contiguous slots #define radix_tree_for_each_contig(slot, root, iter, start) for (slot = radix_tree_iter_init(iter, start) ; slot (slot = radix_tree_next_chunk(root, iter, RADIX_TREE_ITER_CONTIG)); slot = radix_tree_next_slot(slot, iter, RADIX_TREE_ITER_CONTIG)) \ \ \
21 Tags Tags on entries are stored as bit masks. Tags are replicated all the way up to the root. Setting a tag sets it on all parents. Clearing a tag may clear it on a parent if all other entries are also clear. root: index: tag: void *radix_tree_tag_set(struct radix_tree_root *root, unsigned long index, unsigned int tag); void *radix_tree_tag_clear(struct radix_tree_root *root, unsigned long index, unsigned int tag) int radix_tree_tag_get(const struct radix_tree_root *root, unsigned long index, unsigned int tag) check whether the entry at index carries a particular tag. radix tree root index key tag index
22 Multiorder support Multiorder support means allowing entries with keys that cover 2^order indices around the key. Pass the order while inserting. root: index: order: int radix_tree_split(struct radix_tree_root *root, unsigned long index, unsigned order); Split an entry into smaller order entries. index: order: item: int radix_tree_join(struct radix_tree_root *root, unsigned long index, unsigned order, void *item); replace multiple entries with one multi order entry. radix tree root index key order of new entries an index inside the new entry order of the new entry new entry
23 Radix Tree Test Suite
24 Test Suite Merged into Linux 4.6. Location: tools/testing/radix-tree. Regression tests, functional tests and performance tests. Short run or long run. Levels of verbose output.
25 Regression tests Merged into Linux 4.6. Location: tools/testing/radix-tree. Regression tests, functional tests and performance tests. Short run or long run. Levels of verbose output.
26 Functional tests Merged into Linux 4.6. Location: tools/testing/radix-tree. Regression tests, functional tests and performance tests. Short run or long run. Levels of verbose output.
27 Performance tests Merged into Linux 4.6. Location: tools/testing/radix-tree. Regression tests, functional tests and performance tests. Short run or long run. Levels of verbose output.
28 Enhancements as part of Outreachy Project Adding different levels of verbosity to output of test suite. #define printv(verbosity_level, fmt,...) \ if(test_verbose >= verbosity_level) \ printf(fmt, ## VA_ARGS ) Idea extendible to many areas parts in kernel, for debugging, testing etc. Config option in makefile to test for various values of map shift. grep -qw $(SHIFT) generated/map-shift.h; then echo "#define RADIX_TREE_MAP_SHIFT $(SHIFT)" > generated/map-shift.h; fi Config option to build tests for 32 bit or 64 bit machine. \ \ \
29 Enhancements as part of Outreachy Project Automate generation of.gcov files to check their test coverage. Adding new functional tests. idr_get_next() ida_simple_get() ida_simple_remove() radix_tree_clear_tags() Adding new performance tests. For radix tree insertion, deletion, tagging, join and split.
30 Enhancements as part of Outreachy Project Functional test example void radix_tree_clear_tags_test(void) {... item_insert(&tree, 0); item_tag_set(&tree, 0, 0); radix_tree_lookup(&tree, 0, &node, &slot); radix_tree_clear_tags(&tree, node, slot); assert(item_tag_get(&tree, 0, 0) == 0); for (index = 0; index < 1000; index++) { item_insert(&tree, index); item_tag_set(&tree, index, 0); } radix_tree_for_each_slot(slot, &tree, &iter, 0) { radix_tree_clear_tags(&tree, iter.node, slot); assert(item_tag_get(&tree, iter.index, 0) == 0); }...
31 Enhancements as part of Outreachy Project Performance test example static long long benchmark_split(unsigned long index, int old_order, int new_order) { struct timespec start, finish; long long nsec;... item_insert_order(&tree, index, old_order); clock_gettime(clock_monotonic, &start); radix_tree_split(&tree, index, new_order); clock_gettime(clock_monotonic, &finish); nsec = (finish.tv_sec - start.tv_sec) * NSEC_PER_SEC + (finish.tv_nsec - start.tv_nsec);... }
32 Conclusion
33 Conclusion - Improving the general test suite infrastructure by adding various configuration options. - Automating test coverage using gcov. - More functional tests and performance benchmarks.
34 Implement IDR in alloc_fd() Sandhya Bankar
35 About me
36 Linux Kernel Intern through Outreachy in The Linux Foundation with the support of mentors Rik van Riel and Matthew Wilcox. Master of Engineering in Computer networks Bachelor of Engineering in Electronics and communication.
37 Goal of the Project Linux kernel has lots of special allocators However, there now is an IDR library that can do allocation of numbers for us Simplify the kernel by replacing custom allocators with common allocation code
38 IDR - IDR is a type of radix tree that maps integer IDs with specific pointer values. - Originally written for POSIX timer system call implementations. It generates the ID that can handle a specific timer object It is now widely used in various device drivers. - IDR takes a given pointer and creates the corresponding integer ID. With that ID, you can quickly find the original pointer.
39 Allocate and manage file descriptor using IDR
40 About project - Implement IDR in file descriptor allocation code path Replace custom allocator with IDR Remove struct fdtable Convert select() to implement idr_get_tag_batch() Replace close_on_exec bitmap with an IDR tag Use idr_tag_set() and idr_tag_get() for close_on_exec operation. - Rewrite close_files() - Use idr_tag_get in fd_is_open() - Remove full_fds_bits, open_fds bitmaps
41 Cont - Replace array of file pointer with IDR Remove next_fd Memory Saving Performance improvement
42 File Descriptor - File descriptor is used to access a file or other I/O resources (e. g pipe and socket) - A file descriptor is a non-negative integer, generally represented in the C programming language as the type int (negative values being reserved to indicate "no value" or an error condition).
43 Cont... - Linux has three standard POSIX file descriptors, corresponding to the three standard streams - stdin - stdout - stderr
44 Operations on file descriptors - open() - open a file creat() - create a new file / rewrite an existing one pipe() - creates a pipe read() - read from a file descriptor write() - write to a file descriptor close() - close a file descriptor lseek() - reposition read/write file offset select() - synchronous I/O multiplexing socket() - create an endpoint for communication accept() - accept a connection on a socket dup(), dup2() - duplicate an open file descriptor
45 Before IDR implementation open()
46 IDR API used in Project
47 static inline void idr_init(struct idr *idr) - Initialize the idr handle
48 static inline void idr_preload(gfp_t gfp_mask) - Preload for idr_alloc() - Preallocate memory to use for the next call to idr_alloc(). This function returns with preemption disabled. It will be enabled by idr_preload_end(). allocation mask to use for preloading
49 static inline void idr_preload_end(void) - end preload section started with idr_preload() - Enable preemption
50 int idr_alloc(struct idr *idr, void *ptr, int start, int end, gfp_t gfp) - Allocates an unused ID in the range [start, end]. Returns ENOSPC if there are no unused IDs in that range. idr pointer to be associated with the new the minimum id the maximum id memory allocation flags
51 static inline bool idr_check_preload(const struct idr *src) - Check the preload is still sufficient IDR to be copied from - Between the successful allocation of memory and acquiring the lock that the IDR may have expanded. If this function returns false, more memory needs to be preallocated. - Return: true if enough memory remains allocated, false to retry the preallocation.
52 #define idr_for_each_entry(idr, entry, id) - iterate over an idr's elements of a given idr the type * to use as id entry's key do not need to be initialized before the loop, and after normal is left with the value NULL. This is convenient for a "not found" value.
53 static inline void *idr_find(const struct idr *idr, int id) - return pointer for given idr lookup key Return the pointer given the id it has been registered with. A %NULL return indicates is not valid or you passed %NULL in idr_get_new().
54 void idr_destroy(struct idr *idr) - release all internal memory from an IDR idr handle - After this function is called, the IDR is empty, and may be reused or the data structure containing it may be freed. - A typical clean-up sequence for objects stored in an idr tree will use idr_for_each() to free all objects, if necessary, then idr_destroy() to free the memory used to keep track of those objects.
55 static inline void *idr_remove(struct idr *idr, int id) - Remove specific ID - IDR handle - ID to be remove
56 void *idr_replace(struct idr *idr, void *ptr, int id) - replace pointer for given idr New pointer to associate with the Lookup key Replace the pointer registered with an ID and return the old value. - Returns: 0 on success. %-ENOENT indicates was not found. %-EINVAL indicates were not valid.
57 static inline void *idr_tag_set(struct idr *idr, int id, unsigned int tag) - Set a tag on an IDR ID of entry to Tag index to set If there is an entry in this IDR, set a tag on it and return the address of the entry. is outside the range of the IDR, return NULL.
58 static inline bool idr_tag_get(const struct idr *idr, int id, unsigned int tag) - Return whether a particular entry has a tag IDR ID of entry to Tag index to check Returns true/false depending is set on this ID.
59 static inline void *idr_tag_clear(struct idr *idr, int id, unsigned int tag) - Clear a tag on an IDR ID of entry to Tag index to clear If there is an entry in this IDR, clear its tag and return the address of the entry. is outside the range of the IDR, return NULL.
60 After implementing IDR open()
61 Testing
62 Testing - Performance benchmark - Test cases to check below system call - open()/close system call behaviour - dup(), dup2() syscall behaviour - select() syscall behaviour - pipe() syscall behaviour - Open file descriptor limit - Test case which sets close_on_exec tag
63 Result
64 Result Before implementing IDR struct / bitmap Size in bytes struct file_struct 704 struct fdtable 64 struct file pointers 2048 bitmap
65 After implementing IDR struct /radix_tree size in bytes struct files_struct 32 radix_tree node (3 required)
66 - Total memory saving is 1152 bytes (~1K) - It also reduces the size of the tiny config build on i386 by 672 bytes of code and 192 bytes of data.
67 Conclusion
68 Conclusion - Implementation of IDR in alloc_fd() and related code path saved the memory and slightly improved the performance. - With current changes ~1KB kernel memory is saved - fd allocation code (kernel code) size reduced and it is much readable than earlier - Wherever in kernel if we need to map number with any type of pointer then IDR can be best option. - Custom allocator can be replaced with IDR
69 References
70 References NA2016%20-%20Radix%20Tree.pdf - Paul McKenney on RCU: Linux Kernel Development - Robert Love - Understanding Linux Kernel - Daniel Bovet and Marco Cesati
71 Thank You!
72 Questions? We are Linux Kernel Newbies
73
74
Implementing PID allocation Talk Title Here with the IDR API. Author Name, Company Gargi Sharma, Outreachy Intern
Implementing PID allocation Talk Title Here with the IDR API Author Name, Company Gargi Sharma, Outreachy Intern Process IDs Every process has a unique identifier that represents it, called the process
More informationCSE506: Operating Systems CSE 506: Operating Systems
CSE 506: Operating Systems Block Cache Address Space Abstraction Given a file, which physical pages store its data? Each file inode has an address space (0 file size) So do block devices that cache data
More informationCSE 506: Opera.ng Systems. The Page Cache. Don Porter
The Page Cache Don Porter 1 Logical Diagram Binary Formats RCU Memory Management File System Memory Allocators Threads System Calls Today s Lecture Networking (kernel level Sync mem. management) Device
More informationLecture 19: File System Implementation. Mythili Vutukuru IIT Bombay
Lecture 19: File System Implementation Mythili Vutukuru IIT Bombay File System An organization of files and directories on disk OS has one or more file systems Two main aspects of file systems Data structures
More informationMy malloc: mylloc and mhysa. Johan Montelius HT2016
1 Introduction My malloc: mylloc and mhysa Johan Montelius HT2016 So this is an experiment where we will implement our own malloc. We will not implement the world s fastest allocator, but it will work
More informationDesign Overview of the FreeBSD Kernel CIS 657
Design Overview of the FreeBSD Kernel CIS 657 Organization of the Kernel Machine-independent 86% of the kernel (80% in 4.4BSD) C code Machine-dependent 14% of kernel Only 0.6% of kernel in assembler (2%
More informationDesign Overview of the FreeBSD Kernel. Organization of the Kernel. What Code is Machine Independent?
Design Overview of the FreeBSD Kernel CIS 657 Organization of the Kernel Machine-independent 86% of the kernel (80% in 4.4BSD) C C code Machine-dependent 14% of kernel Only 0.6% of kernel in assembler
More informationCMPS 105 Systems Programming. Prof. Darrell Long E2.371
+ CMPS 105 Systems Programming Prof. Darrell Long E2.371 darrell@ucsc.edu + Chapter 3: File I/O 2 + File I/O 3 n What attributes do files need? n Data storage n Byte stream n Named n Non-volatile n Shared
More informationRCU. ò Walk through two system calls in some detail. ò Open and read. ò Too much code to cover all FS system calls. ò 3 Cases for a dentry:
Logical Diagram VFS, Continued Don Porter CSE 506 Binary Formats RCU Memory Management File System Memory Allocators System Calls Device Drivers Networking Threads User Today s Lecture Kernel Sync CPU
More informationReplacing the Radix Tree MATTHEW WILCOX OPEN SOURCE SUMMIT NORTH AMERICA
Replacing the Radix Tree MATTHEW WILCOX OPEN SOURCE SUMMIT NORTH AMERICA 2017-09-13 The two hardest problems in computer science Cache invalidation Naming things Off-by-one errors Phil Karlton & Leon Bambrick
More informationVFS, Continued. Don Porter CSE 506
VFS, Continued Don Porter CSE 506 Logical Diagram Binary Formats Memory Allocators System Calls Threads User Today s Lecture Kernel RCU File System Networking Sync Memory Management Device Drivers CPU
More informationThe Page Cache 3/16/16. Logical Diagram. Background. Recap of previous lectures. The address space abstracvon. Today s Problem.
The Page Cache Don Porter Binary Formats RCU Memory Management File System Logical Diagram Memory Allocators Threads System Calls Today s Lecture Networking (kernel level Sync mem. management) Device CPU
More informationFile Systems. CS170 Fall 2018
File Systems CS170 Fall 2018 Table of Content File interface review File-System Structure File-System Implementation Directory Implementation Allocation Methods of Disk Space Free-Space Management Contiguous
More informationFile Management 1/34
1/34 Learning Objectives system organization and recursive traversal buffering and memory mapping for performance Low-level data structures for implementing filesystems Disk space management for sample
More informationProcesses COMPSCI 386
Processes COMPSCI 386 Elements of a Process A process is a program in execution. Distinct processes may be created from the same program, but they are separate execution sequences. call stack heap STACK
More informationCSC209: Software tools. Unix files and directories permissions utilities/commands Shell programming quoting wild cards files
CSC209 Review CSC209: Software tools Unix files and directories permissions utilities/commands Shell programming quoting wild cards files ... and systems programming C basic syntax functions arrays structs
More informationCSC209: Software tools. Unix files and directories permissions utilities/commands Shell programming quoting wild cards files. Compiler vs.
CSC209 Review CSC209: Software tools Unix files and directories permissions utilities/commands Shell programming quoting wild cards files... and systems programming C basic syntax functions arrays structs
More informationUNIX Kernel. UNIX History
UNIX History UNIX Kernel 1965-1969 Bell Labs participates in the Multics project. 1969 Ken Thomson develops the first UNIX version in assembly for an DEC PDP-7 1973 Dennis Ritchie helps to rewrite UNIX
More informationBasic OS Progamming Abstrac7ons
Basic OS Progamming Abstrac7ons Don Porter Recap We ve introduced the idea of a process as a container for a running program And we ve discussed the hardware- level mechanisms to transi7on between the
More informationBasic OS Progamming Abstrac2ons
Basic OS Progamming Abstrac2ons Don Porter Recap We ve introduced the idea of a process as a container for a running program And we ve discussed the hardware- level mechanisms to transi2on between the
More informationVirtual File System. Don Porter CSE 306
Virtual File System Don Porter CSE 306 History Early OSes provided a single file system In general, system was pretty tailored to target hardware In the early 80s, people became interested in supporting
More informationNAME attr extended attributes on XFS filesystem objects. SYNOPSIS attr [ LRq ] s attrname [ V attrvalue ] pathname
ATTR(1) XFS Compatibility API ATTR(1) attr extended attributes on XFS filesystem objects SYNOPSIS attr [ LRq ] s attrname [ V attrvalue ] pathname attr [ LRq ] g attrname pathname attr [ LRq ] r attrname
More informationCSE 333 SECTION 3. POSIX I/O Functions
CSE 333 SECTION 3 POSIX I/O Functions Administrivia Questions (?) HW1 Due Tonight Exercise 7 due Monday (out later today) POSIX Portable Operating System Interface Family of standards specified by the
More informationRCU. ò Dozens of supported file systems. ò Independent layer from backing storage. ò And, of course, networked file system support
Logical Diagram Virtual File System Don Porter CSE 506 Binary Formats RCU Memory Management File System Memory Allocators System Calls Device Drivers Networking Threads User Today s Lecture Kernel Sync
More informationdmrlib Documentation Release Wijnand Modderman-Lenstra
dmrlib Documentation Release 0.99.3 Wijnand Modderman-Lenstra September 03, 2016 Contents 1 Overview 1 2 Documentation 3 2.1 bits: bit and byte manipulation...................................... 3 2.2
More informationProcess Management! Goals of this Lecture!
Process Management! 1 Goals of this Lecture! Help you learn about:" Creating new processes" Programmatically redirecting stdin, stdout, and stderr" Unix system-level functions for I/O" The Unix stream
More informationCSC209 Review. Yeah! We made it!
CSC209 Review Yeah! We made it! 1 CSC209: Software tools Unix files and directories permissions utilities/commands Shell programming quoting wild cards files 2 ... and C programming... C basic syntax functions
More informationFILE SYSTEMS, PART 2. CS124 Operating Systems Fall , Lecture 24
FILE SYSTEMS, PART 2 CS124 Operating Systems Fall 2017-2018, Lecture 24 2 Last Time: File Systems Introduced the concept of file systems Explored several ways of managing the contents of files Contiguous
More informationSection 7: Wait/Exit, Address Translation
William Liu October 15, 2014 Contents 1 Wait and Exit 2 1.1 Thinking about what you need to do.............................. 2 1.2 Code................................................ 2 2 Vocabulary 4
More informationProcess Management 1
Process Management 1 Goals of this Lecture Help you learn about: Creating new processes Programmatically redirecting stdin, stdout, and stderr (Appendix) communication between processes via pipes Why?
More informationLinux Signals and Daemons
Linux and Daemons Alessandro Barenghi Dipartimento di Elettronica, Informazione e Bioingegneria Politecnico di Milano alessandro.barenghi - at - polimi.it April 17, 2015 Recap By now, you should be familiar
More informationOS COMPONENTS OVERVIEW OF UNIX FILE I/O. CS124 Operating Systems Fall , Lecture 2
OS COMPONENTS OVERVIEW OF UNIX FILE I/O CS124 Operating Systems Fall 2017-2018, Lecture 2 2 Operating System Components (1) Common components of operating systems: Users: Want to solve problems by using
More informationVirtual File System. Don Porter CSE 506
Virtual File System Don Porter CSE 506 History ò Early OSes provided a single file system ò In general, system was pretty tailored to target hardware ò In the early 80s, people became interested in supporting
More informationOutline. Relationship between file descriptors and open files
Outline 3 File I/O 3-1 3.1 File I/O overview 3-3 3.2 open(), read(), write(), and close() 3-7 3.3 The file offset and lseek() 3-21 3.4 Atomicity 3-30 3.5 Relationship between file descriptors and open
More informationPointers about pointers. Announcements. Pointer type. Example
Announcements Pointers about pointers Midterm next week Material covered up until June 18 (last week, signals) Allowed to have 1 cheat sheet No tutorial Come to class at 6 Test is 6:10 7:00 Assignment
More informationPipes and FIFOs. Woo-Yeong Jeong Computer Systems Laboratory Sungkyunkwan University
Pipes and FIFOs Woo-Yeong Jeong (wooyeong@csl.skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu Open Files in Kernel How the Unix kernel represents open files? Two descriptors
More informationAsynchronous Events on Linux
Asynchronous Events on Linux Frederic.Rossi@Ericsson.CA Open System Lab Systems Research June 25, 2002 Ericsson Research Canada Introduction Linux performs well as a general purpose OS but doesn t satisfy
More informationProcess Management! Goals of this Lecture!
Process Management! 1 Goals of this Lecture! Help you learn about:" Creating new processes" Programmatically redirecting stdin, stdout, and stderr" (Appendix) communication between processes via pipes"
More informationA2 Design Considerations. CS161, Spring
A2 Design Considerations CS161, Spring 2014 http://goo.gl/izxych Agenda 1. processes 2. file descriptors 3. fork 4. waitpid & exit 5. exec 6. scheduler 7. suggestions for testing 8. lessons learned from
More informationFall 2017 :: CSE 306. File Systems Basics. Nima Honarmand
File Systems Basics Nima Honarmand File and inode File: user-level abstraction of storage (and other) devices Sequence of bytes inode: internal OS data structure representing a file inode stands for index
More informationNetworking Services Team, Red Hat Alexander Duyck February 17 th, Picking Low Hanging Fruit from the FIB Tree
Picking Low Hanging Fruit from the FIB Tree Networking Services Team, Red Hat Alexander Duyck February 17 th, 2015 1 Agenda What was wrong with the FIB trie? Identifying the low hanging fruit Results of
More informationFile Descriptors and Piping
File Descriptors and Piping CSC209: Software Tools and Systems Programming Furkan Alaca & Paul Vrbik University of Toronto Mississauga https://mcs.utm.utoronto.ca/~209/ Week 8 Today s topics File Descriptors
More informationToolbox Kernel Korner
Toolbox Kernel Korner In this article, Robert offers a refresher on kernel memory allocation and how it has changed for the 2. by Robert Love Unfortunately for kernel developers, allocating memory in the
More informationOperating Systems (234123) Spring 2017 (Homework Wet 1) Homework 1 Wet
Homework 1 Wet Due Date: 30/4/2017 23:00 Teaching assistant in charge: Yehonatan Buchnik Important: the Q&A for the exercise will take place at a public forum Piazza only. Critical updates about the HW
More informationLecture 8: Other IPC Mechanisms. CSC 469H1F Fall 2006 Angela Demke Brown
Lecture 8: Other IPC Mechanisms CSC 469H1F Fall 2006 Angela Demke Brown Topics Messages through sockets / pipes Receiving notification of activity Generalizing the event notification mechanism Kqueue Semaphores
More informationTopics. Lecture 8: Other IPC Mechanisms. Socket IPC. Unix Communication
Topics Lecture 8: Other IPC Mechanisms CSC 469H1F Fall 2006 Angela Demke Brown Messages through sockets / pipes Receiving notification of activity Generalizing the event notification mechanism Kqueue Semaphores
More informationCSE 410: Systems Programming
CSE 410: Systems Programming Input and Output Ethan Blanton Department of Computer Science and Engineering University at Buffalo I/O Kernel Services We have seen some text I/O using the C Standard Library.
More informationFile Systems: Consistency Issues
File Systems: Consistency Issues File systems maintain many data structures Free list/bit vector Directories File headers and inode structures res Data blocks File Systems: Consistency Issues All data
More informationCSE506: Operating Systems CSE 506: Operating Systems
CSE 506: Operating Systems Virtual File System History Early OSes provided a single file system In general, system was tailored to target hardware In the early 80s, desire for more than one file system
More informationCS 361 Computer Systems Fall 2017 Homework Assignment 4 - Inter-Process Communications & I/O
CS 361 Computer Systems Fall 2017 Homework Assignment 4 - Inter-Process Communications & I/O Overall Assignment For this assignment, you are to write three programs that will work together using inter-process
More informationOutline. OS Interface to Devices. System Input/Output. CSCI 4061 Introduction to Operating Systems. System I/O and Files. Instructor: Abhishek Chandra
Outline CSCI 6 Introduction to Operating Systems System I/O and Files File I/O operations File Descriptors and redirection Pipes and FIFOs Instructor: Abhishek Chandra 2 System Input/Output Hardware devices:
More informationCSE 333 SECTION 3. POSIX I/O Functions
CSE 333 SECTION 3 POSIX I/O Functions Administrivia Questions (?) HW1 Due Tonight HW2 Due Thursday, July 19 th Midterm on Monday, July 23 th 10:50-11:50 in TBD (And regular exercises in between) POSIX
More informationECE 550D Fundamentals of Computer Systems and Engineering. Fall 2017
ECE 550D Fundamentals of Computer Systems and Engineering Fall 2017 The Operating System (OS) Prof. John Board Duke University Slides are derived from work by Profs. Tyler Bletsch and Andrew Hilton (Duke)
More informationLecture 23: System-Level I/O
CSCI-UA.0201-001/2 Computer Systems Organization Lecture 23: System-Level I/O Mohamed Zahran (aka Z) mzahran@cs.nyu.edu http://www.mzahran.com Some slides adapted (and slightly modified) from: Clark Barrett
More informationLogical disks. Bach 2.2.1
Logical disks Bach 2.2.1 Physical disk is divided into partitions or logical disks Logical disk linear sequence of fixed size, randomly accessible, blocks disk device driver maps underlying physical storage
More informationETFS Design and Implementation Notes#
ETFS Design and Implementation Notes# One of the first questions that comes to mind when learning a new file system, is "What is the on disk file system structure?" ETFS does not have one. Well, not a
More informationLike select() and poll(), epoll can monitor multiple FDs epoll returns readiness information in similar manner to poll() Two main advantages:
Outline 22 Alternative I/O Models 22-1 22.1 Overview 22-3 22.2 Nonblocking I/O 22-5 22.3 Signal-driven I/O 22-11 22.4 I/O multiplexing: poll() 22-14 22.5 Problems with poll() and select() 22-31 22.6 The
More informationWindows architecture. user. mode. Env. subsystems. Executive. Device drivers Kernel. kernel. mode HAL. Hardware. Process B. Process C.
Structure Unix architecture users Functions of the System tools (shell, editors, compilers, ) standard library System call Standard library (printf, fork, ) OS kernel: processes, memory management, file
More informationOutline. Overview. Linux-specific, since kernel 2.6.0
Outline 25 Alternative I/O Models 25-1 25.1 Overview 25-3 25.2 Signal-driven I/O 25-9 25.3 I/O multiplexing: poll() 25-12 25.4 Problems with poll() and select() 25-29 25.5 The epoll API 25-32 25.6 epoll
More informationCS 3214 Midterm. Here is the distribution of midterm scores for both sections (combined). Midterm Scores 20. # Problem Points Score
CS 3214 Here is the distribution of midterm scores for both sections (combined). Scores 20 18 16 14 12 10 8 6 4 2 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 0 The overall average was 58 points.
More informationAny of the descriptors in the set {1, 4} have an exception condition pending
Page 1 of 6 6.3 select Function This function allows the process to instruct the kernel to wait for any one of multiple events to occur and to wake up the process only when one or more of these events
More informationInter-Process Communication
CS 326: Operating Systems Inter-Process Communication Lecture 10 Today s Schedule Shared Memory Pipes 2/28/18 CS 326: Operating Systems 2 Today s Schedule Shared Memory Pipes 2/28/18 CS 326: Operating
More informationwe are here Page 1 Recall: How do we Hide I/O Latency? I/O & Storage Layers Recall: C Low level I/O
CS162 Operating Systems and Systems Programming Lecture 18 Systems October 30 th, 2017 Prof. Anthony D. Joseph http://cs162.eecs.berkeley.edu Recall: How do we Hide I/O Latency? Blocking Interface: Wait
More informationMemory management. Johan Montelius KTH
Memory management Johan Montelius KTH 2017 1 / 22 C program # include int global = 42; int main ( int argc, char * argv []) { if( argc < 2) return -1; int n = atoi ( argv [1]); int on_stack
More informationMotivation. Operating Systems. File Systems. Outline. Files: The User s Point of View. File System Concepts. Solution? Files!
Motivation Operating Systems Process store, retrieve information Process capacity restricted to vmem size When process terminates, memory lost Multiple processes share information Systems (Ch 0.-0.4, Ch.-.5)
More informationCS 5460/6460 Operating Systems
CS 5460/6460 Operating Systems Fall 2009 Instructor: Matthew Flatt Lecturer: Kevin Tew TAs: Bigyan Mukherjee, Amrish Kapoor 1 Join the Mailing List! Reminders Make sure you can log into the CADE machines
More informationFile Management. Information Structure 11/5/2013. Why Programmers Need Files
File Mgr Device Mgr Memory Mgr Process Mgr File Mgr Device Mgr Memory Mgr Process Mgr 11/5/2013 Slide 13-1 Slide 13-2 File Management 13 Fig 13-2: The External View of the File Manager Slide 13-3 Why Programmers
More informationTDDB68 Concurrent Programming and Operating Systems. Lecture: File systems
TDDB68 Concurrent Programming and Operating Systems Lecture: File systems Mikael Asplund, Senior Lecturer Real-time Systems Laboratory Department of Computer and Information Science Copyright Notice: Thanks
More informationCS 5523 Operating Systems: Midterm II - reivew Instructor: Dr. Tongping Liu Department Computer Science The University of Texas at San Antonio
CS 5523 Operating Systems: Midterm II - reivew Instructor: Dr. Tongping Liu Department Computer Science The University of Texas at San Antonio Fall 2017 1 Outline Inter-Process Communication (20) Threads
More informationCS370 Operating Systems
CS370 Operating Systems Colorado State University Yashwant K Malaiya Fall 2017 Lecture 24 File Systems Slides based on Text by Silberschatz, Galvin, Gagne Various sources 1 1 Questions from last time How
More informationLecture 3. Introduction to Unix Systems Programming: Unix File I/O System Calls
Lecture 3 Introduction to Unix Systems Programming: Unix File I/O System Calls 1 Unix File I/O 2 Unix System Calls System calls are low level functions the operating system makes available to applications
More informationFile Systems. What do we need to know?
File Systems Chapter 4 1 What do we need to know? How are files viewed on different OS s? What is a file system from the programmer s viewpoint? You mostly know this, but we ll review the main points.
More informationCS-354 Interim Document Mr. Christopher Whyley 13 Dec :00 AM
13/12/2013 Science Intranet - Home College of Science Coursework Submission Module Coursework Lecturer Deadline CS-354 Interim Document Mr. Christopher Whyley 13 Dec 2013 11:00 AM Student Number 644475
More informationKernel Synchronization I. Changwoo Min
1 Kernel Synchronization I Changwoo Min 2 Summary of last lectures Tools: building, exploring, and debugging Linux kernel Core kernel infrastructure syscall, module, kernel data structures Process management
More informationSystems Programming. 08. Standard I/O Library. Alexander Holupirek
Systems Programming 08. Standard I/O Library Alexander Holupirek Database and Information Systems Group Department of Computer & Information Science University of Konstanz Summer Term 2008 Last lecture:
More informationProcesses. Processes (cont d)
Processes UNIX process creation image-file arg1 arg2 Shell command line example ls -l Equivalent to /bin/ls -l Why? How do you find out where the image file is? Background processes ls -l & Execute a process
More informationLab 09 - Virtual Memory
Lab 09 - Virtual Memory Due: November 19, 2017 at 4:00pm 1 mmapcopy 1 1.1 Introduction 1 1.1.1 A door predicament 1 1.1.2 Concepts and Functions 2 1.2 Assignment 3 1.2.1 mmap copy 3 1.2.2 Tips 3 1.2.3
More informationCS 4284 Systems Capstone
CS 4284 Systems Capstone Disks & File Systems Godmar Back Filesystems Files vs Disks File Abstraction Byte oriented Names Access protection Consistency guarantees Disk Abstraction Block oriented Block
More informationCSci 4061 Introduction to Operating Systems. Input/Output: High-level
CSci 4061 Introduction to Operating Systems Input/Output: High-level I/O Topics First, cover high-level I/O Next, talk about low-level device I/O I/O not part of the C language! High-level I/O Hide device
More informationOperating System Labs. Yuanbin Wu
Operating System Labs Yuanbin Wu cs@ecnu Operating System Labs Introduction to Unix (*nix) Course Overview Operating System Labs Introduction to Unix (*nix) Course Overview Unix / *nix What A family of
More informationConcurrent Pagecache
Concurrent Pagecache Peter Zijlstra Red Hat pzijlstr@redhat.com Abstract In this paper we present a concurrent pagecache for Linux, which is a continuation of the existing lockless pagecache work [5].
More informationProject 2-1 User Programs
Project 2-1 User Programs Prof. Jin-Soo Kim ( jinsookim@skku.edu) T.A. Sejun Kwon (sejun000@csl.skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu Supporting User Programs
More informationMemory Management. To do. q Basic memory management q Swapping q Kernel memory allocation q Next Time: Virtual memory
Memory Management To do q Basic memory management q Swapping q Kernel memory allocation q Next Time: Virtual memory Memory management Ideal memory for a programmer large, fast, nonvolatile and cheap not
More informationVirtual Memory Management in Linux (Part II)
Virtual Memory Management in Linux (Part II) Minsoo Ryu Department of Computer Science and Engineering 2 1 Page Table and Page Fault Handling Page X 2 Page Cache Page X 3 Page Frame Reclamation (Swapping
More informationECE 2035 Programming HW/SW Systems Spring problems, 5 pages Exam Three 8 April Your Name (please print clearly)
Your Name (please print clearly) This exam will be conducted according to the Georgia Tech Honor Code. I pledge to neither give nor receive unauthorized assistance on this exam and to abide by all provisions
More informationThe course that gives CMU its Zip! I/O Nov 15, 2001
15-213 The course that gives CMU its Zip! I/O Nov 15, 2001 Topics Files Unix I/O Standard I/O A typical hardware system CPU chip register file ALU system bus memory bus bus interface I/O bridge main memory
More informationCOS 318: Operating Systems. Journaling, NFS and WAFL
COS 318: Operating Systems Journaling, NFS and WAFL Jaswinder Pal Singh Computer Science Department Princeton University (http://www.cs.princeton.edu/courses/cos318/) Topics Journaling and LFS Network
More informationUnix API Books. Linux Kernel Books. Assignments and Exams. Grading for CSC 256. Operating Systems 8/31/2018. CSC 256/456 Fall
Prerequisites CSC 2/456: Operating s CSC 252 or equivalent C/C++ programming experience on Unix systems Instructor: Sandhya Dwarkadas TAs: Zhuojia Shen, Mohsen Mohammadi 8/31/2018 CSC 2/456 1 2 Meaning
More informationOperating Systems. Week 9 Recitation: Exam 2 Preview Review of Exam 2, Spring Paul Krzyzanowski. Rutgers University.
Operating Systems Week 9 Recitation: Exam 2 Preview Review of Exam 2, Spring 2014 Paul Krzyzanowski Rutgers University Spring 2015 March 27, 2015 2015 Paul Krzyzanowski 1 Exam 2 2012 Question 2a One of
More informationCS 333 Introduction to Operating Systems. Class 3 Threads & Concurrency. Jonathan Walpole Computer Science Portland State University
CS 333 Introduction to Operating Systems Class 3 Threads & Concurrency Jonathan Walpole Computer Science Portland State University 1 Process creation in UNIX All processes have a unique process id getpid(),
More informationHow do we define pointers? Memory allocation. Syntax. Notes. Pointers to variables. Pointers to structures. Pointers to functions. Notes.
, 1 / 33, Summer 2010 Department of Computer Science and Engineering York University Toronto June 15, 2010 Table of contents, 2 / 33 1 2 3 Exam, 4 / 33 You did well Standard input processing Testing Debugging
More informationProcesses. Dr. Yingwu Zhu
Processes Dr. Yingwu Zhu Process Growing Memory Stack expands automatically Data area (heap) can grow via a system call that requests more memory - malloc() in c/c++ Entering the kernel (mode) Hardware
More informationbytes per disk block (a block is usually called sector in the disk drive literature), sectors in each track, read/write heads, and cylinders (tracks).
Understanding FAT 12 You need to address many details to solve this problem. The exercise is broken down into parts to reduce the overall complexity of the problem: Part A: Construct the command to list
More informationContents. IPC (Inter-Process Communication) Representation of open files in kernel I/O redirection Anonymous Pipe Named Pipe (FIFO)
Pipes and FIFOs Prof. Jin-Soo Kim( jinsookim@skku.edu) TA JinHong Kim( jinhong.kim@csl.skku.edu) Computer Systems Laboratory Sungkyunkwan University http://csl.skku.edu Contents IPC (Inter-Process Communication)
More informationOperating Systems. IV. Memory Management
Operating Systems IV. Memory Management Ludovic Apvrille ludovic.apvrille@telecom-paristech.fr Eurecom, office 470 http://soc.eurecom.fr/os/ @OS Eurecom Outline Basics of Memory Management Hardware Architecture
More informationCS 326: Operating Systems. Process Execution. Lecture 5
CS 326: Operating Systems Process Execution Lecture 5 Today s Schedule Process Creation Threads Limited Direct Execution Basic Scheduling 2/5/18 CS 326: Operating Systems 2 Today s Schedule Process Creation
More informationCSCI 8530 Advanced Operating Systems. Part 7 Low-level Memory Management
CSCI 8530 Advanced Operating Systems Part 7 Low-level Memory Management Updated 9/22/2016 Location of Low-level Memory Management in the Hierarchy Apparent Impossibility of a Hierarchical OS Design Process
More informationPebbles Kernel Specification September 26, 2004
15-410, Operating System Design & Implementation Pebbles Kernel Specification September 26, 2004 Contents 1 Introduction 2 1.1 Overview...................................... 2 2 User Execution Environment
More information1. Overview This project will help you understand address spaces and virtual memory management.
Project 2--Memory Worth: 12 points Assigned: Due: 1. Overview This project will help you understand address spaces and virtual memory management. In this project, you will implement an external pager,
More informationwe are here I/O & Storage Layers Recall: C Low level I/O Recall: C Low Level Operations CS162 Operating Systems and Systems Programming Lecture 18
I/O & Storage Layers CS162 Operating Systems and Systems Programming Lecture 18 Systems April 2 nd, 2018 Profs. Anthony D. Joseph & Jonathan Ragan-Kelley http://cs162.eecs.berkeley.edu Application / Service
More information