OPERATING SYSTEMS B.TECH CSE III YEAR I SEMESTER (JNTUA-R13) Mrs. N.HEMALATHA ASST.PROFESSOR

Transcription

1 LECTURE NOTES ON OPERATING SYSTEMS B.TECH CSE III YEAR I SEMESTER (JNTUA-R13) Mrs. N.HEMALATHA ASST.PROFESSOR DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING CHADALAWADA RAMANAMMA ENGINEERING COLLEGE CHADALAWADA NAGAR, RENIGUNTA ROAD, TIRUPATI (A.P) CREC, Dept. f CSE

2 JAWAHARLAL NEHRU TECHNOLOGICAL UNIVERSITY ANANTAPUR B.Tech. III - I Sem. Curse Objective: (13A05501) OPERATING SYSTEMS Th Tu C T make the students understand the basic perating system cncepts such as prcesses, threads, scheduling, synchrnizatin, deadlcks, memry management, file and I/O subsystems and prtectin. T get acquaintance with the class f abstractins affrd by general purpse perating systems that aid the develpment f user applicatins. Learning Outcme: Able t use perating systems effectively. Write System and applicatin prgrams t explit perating system functinality. Add functinality t the exiting perating systems Design new perating systems. UNIT I Operating Systems Overview: Operating system functins, Operating system structure, perating systems Operatins, prtectin and security, Kernel data Structures, Cmputing Envirnments, Open- Surce Operating Systems Operating System Structure: Operating System Services, User and Operating-System Interface, systems calls, Types f System Calls, system prgrams, perating system structure, perating system debugging, System Bt. Prcesses: Prcess cncept, prcess Scheduling, Operatins n prcesses, Inter prcess Cmmunicatin, Examples f IPC systems. UNIT II Threads: verview, Multicre Prgramming, Multithreading Mdels, Thread Libraries, Implicit threading, Threading Issues. Prcess Synchrnizatin: The critical-sectin prblem, Petersn s Slutin, Synchrnizatin Hardware, Mutex Lcks, Semaphres, Classic prblems f synchrnizatin, Mnitrs, Synchrnizatin examples, Alternative appraches. CPU Scheduling: Scheduling-Criteria, Scheduling Algrithms, Thread Scheduling, Multiple- Prcessr Scheduling, Real-Time CPU Scheduling, Algrithm Evaluatin. UNIT III Memry Management: Swapping, cntiguus memry allcatin, segmentatin, paging, structure f the page table. Virtual memry: demand paging, page-replacement, Allcatin f frames, Thrashing, Memry- Mapped Files, Allcating Kernel Memry Deadlcks: System Mdel, deadlck characterizatin, Methds f handling Deadlcks, Deadlck preventin, Detectin and Avidance, Recvery frm deadlck. UNIT IV Mass-strage structure: Overview f Mass-strage structure, Disk structure, Disk attachment, Disk scheduling, Swap-space management, RAID structure, Stable-strage implementatin. File system Interface: The cncept f a file, Access Methds, Directry and Disk structure, File system munting, File sharing, Prtectin. File system Implementatin: File-system structure, File-system Implementatin, Directry Implementatin, Allcatin Methds, Free-Space management.

3 UNIT V I/O systems: I/O Hardware, Applicatin I/O interface, Kernel I/O subsystem, Transfrming I/O requests t Hardware peratins. Prtectin: Gals f Prtectin, Principles f Prtectin, Dmain f prtectin, Access Matrix, Implementatin f Access Matrix, Access cntrl, Revcatin f Access Rights, Capability- Based systems, Language Based Prtectin Security: The Security prblem, Prgram threats, System and Netwrk threats, Cryptgraphy as a security tl, User authenticatin, Implementing security defenses, Firewalling t prtect systems and netwrks, Cmputer security classificatins. Text Bks: 1. Operating System Cncepts, Abraham Silberchatz, Peter B. Galvin, Greg Gagne, Ninth Editin, 2012, Wiley. 2. Operating Systems: Internals and Design Principles, Stallings, Sixth Editin, 2009, Pearsn Educatin. Reference Bks: 1. Mdern Operating Systems, Andrew S Tanenbaum, Secnd Editin, PHI. 2. Operating Systems, S.Haldar, A.A.Aravind, Pearsn Educatin. 3. Principles f Operating Systems, B.L.Stuart, Cengage learning, India Editin. 4. Operating Systems, A.S.Gdble, Secnd Editin, TMH. 5. An Intrductin t Operating Systems, P.C.P. Bhatt, PHI. 6. Operating Systems, G.Nutt, N.Chaki and S.Negy, Third Editin, Pearsn Educatin. 7. Operating Systems, R.Elmasri, A,G.Carrick and D.Levine, Mc Graw Hill.

4 Operating System Definitin: OS is a resurce allcatr UNIT-1 Operating System Overview Manages all resurces Decides between cnflicting requests fr efficient and fair resurce use OS is a cntrl prgram Cntrls executin f prgrams t prevent errrs and imprper use f the cmputer Operating System Structure: Multiprgramming needed fr efficiency Single user cannt keep CPU and I/O devices busy at all times Multiprgramming rganizes jbs (cde and data) s CPU always has ne t execute A subset f ttal jbs in system is kept in memry One jb selected and run via jb scheduling When it has t wait (fr I/O fr example), OS switches t anther jb Timesharing (multitasking) is lgical extensin in which CPU switches jbs s frequently that users can interact with each jb while it is running, creating interactive cmputing Respnse time shuld be < 1 secnd Each user has at least ne prgram executing in memry [prcess If several jbs ready t run at the same time [CPU scheduling If prcesses dn t fit in memry, swapping mves them in and ut t run Virtual memry allws executin f prcesses nt cmpletely in memry Memry Layut fr Multi prgrammed System CREC, Dept. f CSE Page 1

5 Operating-System Operatins Interrupt driven by hardware Sftware errr r request creates exceptin r trap Divisin by zer, request fr perating system service Other prcess prblems include infinite lp, prcesses mdifying each ther r the perating system Dual-mde peratin allws OS t prtect itself and ther system cmpnents User mde and kernel mde Mde bit prvided by hardware 4 Prvides ability t distinguish when system is running user cde r kernel cde 4 Sme instructins designated as privileged, nly executable in kernel mde 4 System call changes mde t kernel, return frm call resets it t user Transitin frm User t Kernel Mde Timer t prevent infinite lp / prcess hgging resurces Set interrupt after specific perid Operating system decrements cunter When cunter zer generate an interrupt Set up befre scheduling prcess t regain cntrl r terminate prgram that exceeds alltted CREC, Dept. f CSE Page 2

6 Prtectin and Security: Prtectin any mechanism fr cntrlling access f prcesses r users t resurces defined by the OS Security defense f the system against internal and external attacks Huge range, including denial-f-service, wrms, viruses, identity theft, theft f service Systems generally first distinguish amng users, t determine wh can d what User identities (user IDs, security IDs) include name and assciated number, ne per user User ID then assciated with all files, prcesses f that user t determine access cntrl Grup identifier (grup ID) allws set f users t be defined and cntrls managed, then als assciated with each prcess, file Privilege escalatin allws user t change t effective ID with mre rights Cmputing Envirnments: Client-Server Cmputing Dumb terminals supplanted by smart PCs Many systems nw servers, respnding t requests generated by clients 4 Cmpute-server prvides an interface t client t request services (i.e., database) 4 File-server prvides interface fr clients t stre and retrieve files CREC, Dept. f CSE Page 3

7 Peer t Peer: P2P des nt distinguish clients and servers Instead all ndes are cnsidered peers May each act as client, server r bth Nde must jin P2P netwrk 4 Registers its service with central lkup service n netwrk, r 4 Bradcast request fr service and respnd t requests fr service via discvery prtcl Examples include Napster and Gnutella Web-Based Cmputing Web has becme ubiquitus PCs mst prevalent devices Mre devices becming netwrked t allw web access New categry f devices t manage web traffic amng similar servers: lad balancers Use f perating systems like Windws 95, client-side, have evlved int Linux and Windws XP, which can be clients and servers Open-Surce Operating Systems: Operating systems made available in surce-cde frmat rather than just binary clsedsurce Cunter t the cpy prtectinand Digital Rights Management (DRM)mvement Started by Free Sftware Fundatin (FSF), which has cpyleft GNU Public License (GPL) CREC, Dept. f CSE Page 4

8 Examples include GNU/Linux and BSD UNIX(including cre f Mac OS X), and many mre Operating System Services: Operating systems prvide an envirnment fr executin f prgrams and services t prgrams and users One set f perating-system services prvides functins that are helpful t the user: User interface - Almst all perating systems have a user interface (UI). 4 Varies between Cmmand-Line (CLI), Graphics User Interface (GUI), Batch Prgram executin - The system must be able t lad a prgram int memry and t run that prgram, end executin, either nrmally r abnrmally (indicating errr) I/O peratins - A running prgram may require I/O, which may invlve a file r an I/O device File-system manipulatin - The file system is f particular interest. Prgrams need t read and write files and directries, create and delete them, search them, list file Infrmatin, permissin management. Cmmunicatins Prcesses may exchange infrmatin, n the same cmputer r between cmputers ver a netwrk 4 Cmmunicatins may be via shared memry r thrugh message passing (packets mved by the OS) Errr detectin OS needs t be cnstantly aware f pssible errrs 4 May ccur in the CPU and memry hardware, in I/O devices, in user prgram 4 Fr each type f errr, OS shuld take the apprpriate actin t ensure crrect and cnsistent cmputing 4 Debugging facilities can greatly enhance the user s and prgrammer s abilities t efficiently use the system Anther set f OS functins exists fr ensuring the efficient peratin f the system itself via resurce sharing Resurce allcatin - When multiple users r multiple jbs running cncurrently, resurces must be allcated t each f them 4 Many types f resurces - Sme (such as CPU cycles, main memry, and file strage) may have special allcatin cde, thers (such as I/O devices) may have general request and release cde CREC, Dept. f CSE Page 5

9 Accunting - T keep track f which users use hw much and what kinds f cmputer resurces Prtectin and security - The wners f infrmatin stred in a multiuser r netwrked cmputer system may want t cntrl use f that infrmatin, cncurrent prcesses shuld nt interfere with each ther 4 Prtectin invlves ensuring that all access t system resurces is cntrlled 4 Security f the system frm utsiders requires user authenticatin, extends t defending external I/O devices frm invalid access attempts 4 If a system is t be prtected and secure, precautins must be instituted thrughut it. A chain is nly as strng as its weakest link. System Calls: Prgramming interface t the services prvided by the OS Typically written in a high-level language (C r C++) Mstly accessed by prgrams via a high-level Applicatin Prgram Interface (API)rather than direct system call use Three mst cmmn APIs are Win32 API fr Windws, POSIX API fr POSIX-based systems (including virtually all versins f UNIX, Linux, and Mac OS X), and Java API fr the Java virtual machine (JVM) Why use APIs rather than system calls? CREC, Dept. f CSE Page 6

10 (Nte that the system-call names used thrughut this text are generic) Types f System Calls: Prcess cntrl end, abrt lad, execute create prcess, terminate prcess get prcess attributes, set prcess attributes wait fr time wait event, signal event allcate and free memry File management create file, delete file pen, clse file read, write, repsitin get and set file attributes Device management request device, release device CREC, Dept. f CSE Page 7

11 read, write, repsitin get device attributes, set device attributes lgically attach r detach devices Infrmatin maintenance get time r date, set time r date get system data, set system data get and set prcess, file, r device attributes Cmmunicatins create, delete cmmunicatin cnnectin send, receive messages transfer status infrmatin attach and detach remte devices System Prgrams: System prgrams prvide a cnvenient envirnment fr prgram develpment and executin. They can be divided int: File manipulatin Status infrmatin File mdificatin Prgramming language supprt Prgram lading and executin Cmmunicatins Applicatin prgrams Mst users view f the peratin system is defined by system prgrams, nt the actual system calls Prvide a cnvenient envirnment fr prgram develpment and executin Sme f them are simply user interfaces t system calls; thers are cnsiderably mre cmplex File management - Create, delete, cpy, rename, print, dump, list, and generally manipulate files and directries Status infrmatin CREC, Dept. f CSE Page 8

12 Sme ask the system fr inf - date, time, amunt f available memry, disk space, number f users Others prvide detailed perfrmance, lgging, and debugging infrmatin Typically, these prgrams frmat and print the utput t the terminal r ther utput devices Sme systems implement a registry - used t stre and retrieve cnfiguratin infrmatin File mdificatin Text editrs t create and mdify files Special cmmands t search cntents f files r perfrm transfrmatins f the text Prgramming-language supprt - Cmpilers, assemblers, debuggers and interpreters smetimes prvided Prgram lading and executin- Abslute laders, relcatable laders, linkage editrs, and verlay-laders, debugging systems fr higher-level and machine language Cmmunicatins - Prvide the mechanism fr creating virtual cnnectins amng prcesses, users, and cmputer systems Allw users t send messages t ne anther s screens, brwse web pages, send electrnic-mail messages, lg in remtely, transfer files frm ne machine t anther Operating-System Debugging: Debuggingis finding and fixing errrs, r bugs OSes generate lg filescntaining errr infrmatin Failure f an applicatin can generate cre dumpfile capturing memry f the prcess Operating system failure can generate crash dumpfile cntaining kernel memry Beynd crashes, perfrmance tuning can ptimize system perfrmance Kernighan s Law: Debugging is twice as hard as writing the cde in the first place. Therefre, if yu write the cde as cleverly as pssible, yu are, by definitin, nt smart enugh t debug it. DTrace tl in Slaris, FreeBSD, Mac OS X allws live instrumentatin n prductin systems Prbes fire when cde is executed, capturing state data and sending it t cnsumers f thse prbes Operating System Generatin: CREC, Dept. f CSE Page 9

13 Operating systems are designed t run n any f a class f machines; the system must be cnfigured fr each specific cmputer site SYSGEN prgram btains infrmatin cncerning the specific cnfiguratin f the hardware system Bting starting a cmputer by lading the kernel Btstrap prgram cde stred in ROM that is able t lcate the kernel, lad it int memry, and start its executin System Bt Operating system must be made available t hardware s hardware can start it Small piece f cde btstrap lader, lcates the kernel, lads it int memry, and starts it Smetimes tw-step prcess where bt blck at fixed lcatin lads btstrap lader When pwer initialized n system, executin starts at a fixed memry lcatin 4 Firmware used t hld initial bt cde Prcess Cncept: An perating system executes a variety f prgrams: Batch system jbs Time-shared systems user prgrams r tasks Textbk uses the terms jb and prcess almst interchangeably Prcess a prgram in executin; prcess executin must prgress in sequential fashin A prcess includes: prgram cunter stack data sectin The Prcess: Multiple parts The prgram cde, als called text sectin Current activity including prgram cunter, prcessr registers Stack cntaining temprary data CREC, Dept. f CSE Page 10

14 4 Functin parameters, return addresses, lcal variables Data sectin cntaining glbal variables Heap cntaining memry dynamically allcated during run time Prgram is passive entity, prcess is active Prgram becmes prcess when executable file laded int memry Executin f prgram started via GUI muse clicks, cmmand line entry f its name, etc One prgram can be several prcesses Cnsider multiple users executing the same prgram Prcess State: As a prcess executes, it changes state new: The prcess is being created running: Instructins are being executed waiting: The prcess is waiting fr sme event t ccur ready: The prcess is waiting t be assigned t a prcessr terminated: The prcess has finished executin CREC, Dept. f CSE Page 11

15 Prcess Cntrl Blck (PCB): Infrmatin assciated with each prcess Prcess state Prgram cunter CPU registers CPU scheduling infrmatin Memry-management infrmatin Accunting infrmatin I/O status infrmatin Prcess Scheduling: Maximize CPU use, quickly switch prcesses nt CPU fr time sharing Prcess scheduler selects amng available prcesses fr next executin n CPU Maintains scheduling queues f prcesses CREC, Dept. f CSE Page 12

16 Jb queue set f all prcesses in the system Ready queue set f all prcesses residing in main memry, ready and waiting t execute Device queues set f prcesses waiting fr an I/O device Prcesses migrate amng the varius queues Schedulers: Lng-term scheduler(r jb scheduler) selects which prcesses shuld be brught int the ready queue Shrt-term scheduler(r CPU scheduler) selects which prcess shuld be executed next and allcates CPU Smetimes the nly scheduler in a system Shrt-term scheduler is invked very frequently (millisecnds) (must be fast) Lng-term scheduler is invked very infrequently (secnds, minutes) (may be slw) The lng-term scheduler cntrls the degree f multiprgramming Prcesses can be described as either: I/O-bund prcess spends mre time ding I/O than cmputatins, many shrt CPU bursts CPU-bund prcess spends mre time ding cmputatins; few very lng CPU bursts CREC, Dept. f CSE Page 13

17 CREC, Dept. f CSE Page 14

18 UNIT-2 THREADS, PROCESS SYNCHRONISATON, CPU SCHEDULING Threads run within applicatin Multiple tasks with the applicatin can be implemented by separate threads Update display Fetch data Spell checking Answer a netwrk request Prcess creatin is heavy-weight while thread creatin is light-weight Can simplify cde, increase efficiency Kernels are generally multithreaded Multithreaded Server Architecture: CREC, Dept. f CSE Page 15

19 Cncurrent Executin n a Single-cre System Parallel Executin n a Multicre System User Threads: Thread management dne by user-level threads library Three primary thread libraries: POSIX Pthreads Win32 threads Java threads Kernel Threads: Supprted by the Kernel Examples Windws XP/2000 Slaris Linux Tru64 UNIX Mac OS X Multithreading Mdels: Many-t-One One-t-One Many-t-Many Many-t-One Many user-level threads mapped t single kernel thread CREC, Dept. f CSE Page 16

20 Examples: Slaris Green Threads GNU Prtable Threads One-t-One: Each user-level thread maps t kernel thread Examples Windws NT/XP/2000 Linux Slaris 9 and later Many-t-Many Mdel: Allws many user level threads t be mapped t many kernel threads Allws the perating system t create a sufficient number f kernel threads Slaris prir t versin 9 CREC, Dept. f CSE Page 17

21 Windws NT/2000 with the ThreadFiber package Thread Libraries: Thread library prvides prgrammer with API fr creating and managing threads Tw primary ways f implementing Library entirely in user space Kernel-level library supprted by the OS Pthreads May be prvided either as user-level r kernel-level A POSIX standard (IEEE c) API fr thread creatin and synchrnizatin API specifies behavir f the thread library, implementatin is up t develpment f the library Cmmn in UNIX perating systems (Slaris, Linux, Mac OS X) CREC, Dept. f CSE Page 18

22 Java Threads: Java threads are managed by the JVM Typically implemented using the threads mdel prvided by underlying OS Java threads may be created by: Extending Thread class Implementing the Runnable interface Threading Issues: Semantics f frk() and exec() system calls CREC, Dept. f CSE Page 19

23 Thread cancellatin f target thread Asynchrnus r deferred Signal handling Synchrnus and asynchrnus Thread pls Thread-specific data n Create Facility needed fr data private t thread Scheduler activatins Thread Cancellatin: Terminating a thread befre it has finished Tw general appraches: Asynchrnus cancellatin terminates the target thread immediately. Deferred cancellatin allws the target thread t peridically check if it shuld be cancelled. Thread Pls: Create a number f threads in a pl where they await wrk Advantages: Usually slightly faster t service a request with an existing thread than create a new thread Allws the number f threads in the applicatin(s) t be bund t the size f the pl Scheduler Activatins: Bth M:M and Tw-level mdels require cmmunicatin t maintain the apprpriate number f kernel threads allcated t the applicatin Scheduler activatins prvide upcalls - a cmmunicatin mechanism frm the kernel t the thread library This cmmunicatin allws an applicatin t maintain the crrect number kernel threads Lightweight Prcesses CREC, Dept. f CSE Page 20

24 Critical Sectin Prblem: Cnsider system f n prcesses {p 0, p 1, p n-1 } Each prcess has critical sectin segment f cde Prcess may be changing cmmn variables, updating table, writing file, etc When ne prcess in critical sectin, n ther may be in its critical sectin Critical sectin prblem is t design prtcl t slve this Each prcess must ask permissin t enter critical sectin in entry sectin, may fllw critical sectin with exit sectin, then remainder sectin Especially challenging with preemptive kernels General structure f prcess p i is Slutin t Critical-Sectin Prblem: 1. Mutual Exclusin - If prcess P i is executing in its critical sectin, then n ther prcesses can be executing in their critical sectins CREC, Dept. f CSE Page 21

25 2. Prgress - If n prcess is executing in its critical sectin and there exist sme prcesses that wish t enter their critical sectin, then the selectin f the prcesses that will enter the critical sectin next cannt be pstpned indefinitely 3. Bunded Waiting - A bund must exist n the number f times that ther prcesses are allwed t enter their critical sectins after a prcess has made a request t enter its critical sectin and befre that request is granted Petersn s Slutin: Assume that each prcess executes at a nnzer speed N assumptin cncerning relative speed f the n prcesses Tw prcess slutin Assume that the LOAD and STORE instructins are atmic; that is, cannt be interrupted The tw prcesses share tw variables: intturn; Blean flag[2] The variable turn indicates whse turn it is t enter the critical sectin The flag array is used t indicate if a prcess is ready t enter the critical sectin. flag[i] = true implies that prcess P i is ready! Algrithm fr Prcess P i d { flag[i] = TRUE; turn = j; while (flag[j] && turn == j); critical sectin flag[i] = FALSE; remainder sectin } while (TRUE); Prvable that 1. Mutual exclusin is preserved 2. Prgress requirement is satisfied 3. Bunded-waiting requirement is met Synchrnizatin Hardware: Many systems prvide hardware supprt fr critical sectin cde CREC, Dept. f CSE Page 22

26 Uniprcessrs culd disable interrupts Currently running cde wuld execute withut preemptin Generally t inefficient n multiprcessr systems 4 Operating systems using this nt bradly scalable 4 Mdern machines prvide special atmic hardware instructins 4 Atmic = nn-interruptable Either test memry wrd and set value Or swap cntents f tw memry wrds d { acquire lck critical sectin release lck remainder sectin } while (TRUE); Semaphre: Synchrnizatin tl that des nt require busy waiting Semaphre S integer variable Tw standard peratins mdify S: wait() and signal() Originally called P() andv() Less cmplicated Can nly be accessed via tw indivisible (atmic) peratins wait (S) { while S <= 0 ; // n-p S--; } signal (S) { S++; } Semaphre as General Synchrnizatin Tl Cunting semaphre integer value can range ver an unrestricted dmain CREC, Dept. f CSE Page 23

27 Binary semaphre integer value can range nly between 0 and 1; can be simpler t implement l Als knwn as mutex lcks Can implement a cunting semaphre S as a binary semaphre Prvides mutual exclusin Semaphre mutex; // initialized t 1 d { wait (mutex); // Critical Sectin signal (mutex); // remainder sectin } while (TRUE); Semaphre Implementatin Must guarantee that n tw prcesses can execute wait () and signal () n the same semaphre at the same time Thus, implementatin becmes the critical sectin prblem where the wait and signal cde are placed in the crtical sectin Culd nw have busy waiting in critical sectin implementatin 4 But implementatin cde is shrt 4 Little busy waiting if critical sectin rarely ccupied 4 Nte that applicatins may spend lts f time in critical sectins and therefre this is nt a gd slutin Deadlck and Starvatin Deadlck tw r mre prcesses are waiting indefinitely fr an event that can be caused by nly ne f the waiting prcesses Let S and Q be tw semaphres initialized t 1 P 0 P 1 wait (S); wait (Q); wait (Q); wait (S);.. CREC, Dept. f CSE Page 24

28 .. signal (S); signal (Q); signal (Q); signal (S); Starvatin indefinite blcking A prcess may never be remved frm the semaphre queue in which it is suspended Pririty Inversin Scheduling prblem when lwer-pririty prcess hlds a lck needed by higher-pririty prcess Slved via pririty-inheritance prtcl Classical Prblems f Synchrnizatin: Classical prblems used t test newly-prpsed synchrnizatin schemes Bunded-Buffer Prblem Readers and Writers Prblem Dining-Philsphers Prblem Bunded-Buffer Prblem N buffers, each can hld ne item Semaphre mutex initialized t the value 1 Semaphre full initialized t the value 0 Semaphre empty initialized t the value N The structure f the prducer prcess d { // prduce an item in nextp wait (empty); wait (mutex); // add the item t the buffer signal (mutex); signal (full); } while (TRUE); CREC, Dept. f CSE Page 25

29 The structure f the cnsumer prcess d { wait (full); wait (mutex); // remve an item frm buffer t nextc signal (mutex); signal (empty); // cnsume the item in nextc } while (TRUE); Readers-Writers Prblem: A data set is shared amng a number f cncurrent prcesses Readers nly read the data set; they d nt perfrm any updates Writers can bth read and write Prblem allw multiple readers t read at the same time Only ne single writer can access the shared data at the same time Several variatins f hw readers and writers are treated all invlve pririties Shared Data Data set Semaphre mutex initialized t 1 Semaphre wrt initialized t 1 Integer readcunt initialized t 0 The structure f a writer prcess d { wait (wrt) ; // writing is perfrmed signal (wrt) ; } while (TRUE); The structure f a reader prcess d { CREC, Dept. f CSE Page 26

30 wait (mutex) ; readcunt ++ ; if (readcunt == 1) wait (wrt) ; signal (mutex) // reading is perfrmed wait (mutex) ; readcunt - - ; if (readcunt == 0) signal (wrt) ; signal (mutex) ; } while (TRUE); Dining-Philsphers Prblem Philsphers spend their lives thinking and eating Dn t interact with their neighbrs, ccasinally try t pick up 2 chpsticks (ne at a time) t eat frm bwl Need bth t eat, then release bth when dne In the case f 5 philsphers Shared data 4 Bwl f rice (data set) Semaphre chpstick [5] initialized t 1 The structure f Philspher i: d { CREC, Dept. f CSE Page 27

31 wait ( chpstick[i] ); wait ( chpstick[ (i + 1) % 5] ); // eat signal ( chpstick[i] ); signal (chpstick[ (i + 1) % 5] ); // think } while (TRUE); Mnitrs A high-level abstractin that prvides a cnvenient and effective mechanism fr prcess synchrnizatin Abstract data type, internal variables nly accessible by cde within the prcedure Only ne prcess may be active within the mnitr at a time But nt pwerful enugh t mdel sme synchrnizatin schemes mnitr mnitr-name { // shared variable declaratins prcedure P1 ( ) {. } prcedurepn ( ) { } Initializatin cde ( ) { } } } Schematic view f a Mnitr Mnitr with Cnditin Variables CREC, Dept. f CSE Page 28

32 Scheduling Criteria: CPU utilizatin keep the CPU as busy as pssible Thrughput # f prcesses that cmplete their executin per time unit Turnarund time amunt f time t execute a particular prcess Waiting time amunt f time a prcess has been waiting in the ready queue Respnse time amunt f time it takes frm when a request was submitted until the first respnse is prduced, nt utput (fr time-sharing envirnment) Scheduling Algrithm Optimizatin Criteria Max CPU utilizatin Max thrughput Min turnarund time Min waiting time Min respnse time First-Cme, First-Served (FCFS) Scheduling Prcess Burst Time P 1 24 P 2 3 P 3 3 Suppse that the prcesses arrive in the rder: P 1,P 2, P 3 The Gantt Chart fr the schedule is: P 1 P 2 P CREC, Dept. f CSE Page 29

33 Waiting time fr P 1 = 0; P 2 = 24; P 3 = 27 Average waiting time: ( )/3 = 17 Suppse that the prcesses arrive in the rder: P 2,P 3, P 1 The Gantt chart fr the schedule is: P 2 P 3 P Waiting time fr P 1 = 6;P 2 = 0 ; P 3 = 3 Average waiting time: ( )/3 = 3 Much better than previus case Cnvy effect - shrt prcess behind lng prcess Cnsider ne CPU-bund and many I/O-bund prcesses Shrtest-Jb-First (SJF) Scheduling Assciate with each prcess the length f its next CPU burst Use these lengths t schedule the prcess with the shrtest time SJF is ptimal gives minimum average waiting time fr a given set f prcesses The difficulty is knwing the length f the next CPU request Culd ask the user PrcessArrival Time Burst Time P P P P SJF scheduling chart CREC, Dept. f CSE Page 30

34 Average waiting time = ( ) / 4 = 7 Pririty Scheduling A pririty number (integer) is assciated with each prcess The CPU is allcated t the prcess with the highest pririty (smallest integer highest pririty) Preemptive Nnpreemptive SJF is pririty scheduling where pririty is the inverse f predicted next CPU burst time Prblem Starvatin lw pririty prcesses may never execute Slutin Aging as time prgresses increase the pririty f the prcess Prcess Burst Time Pririty P P P P P Pririty scheduling Gantt Chart P 1 P 2 P 4 P 5 P 1 P 3 P 3 P 4 P Average waiting time = 8.2 msec 192 CREC, Dept. f CSE Page 31

35 UNIT-3 Virtual Memry, Main Memry, Deadlcks Prgram must be brught (frm disk) int memry and placed within a prcess fr it t be run Main memry and registers are nly strage CPU can access directly Memry unit nly sees a stream f addresses + read requests, r address + data and write requests Register access in ne CPU clck (r less) Main memry can take many cycles Cache sits between main memry and CPU registers Prtectin f memry required t ensure crrect peratin Base and Limit Registers A pair f base and limit registers define the lgical address space Hardware Address Prtectin with Base and Limit Registers CREC, Dept. f CSE Page 32

36 Lgical vs. Physical Address Space The cncept f a lgical address space that is bund t a separate physical address space is central t prper memry management Lgical address generated by the CPU; als referred t as virtual address Physical address address seen by the memry unit Lgical and physical addresses are the same in cmpile-time and lad-time address-binding schemes; lgical (virtual) and physical addresses differ in executin-time address-binding scheme Lgical address space is the set f all lgical addresses generated by a prgram Physical address space is the set f all physical addresses generated by a prgram Memry-Management Unit (MMU) Hardware device that at run time maps virtual t physical address Many methds pssible, cvered in the rest f this chapter T start, cnsider simple scheme where the value in the relcatin register is added t every address generated by a user prcess at the time it is sent t memry Base register nw called relcatin register MS-DOS n Intel 80x86 used 4 relcatin registers The user prgram deals with lgical addresses; it never sees the real physical addresses Executin-time binding ccurs when reference is made t lcatin in memry Lgical address bund t physical addresses Dynamic relcatin using relcatin register CREC, Dept. f CSE Page 33

37 Dynamic Lading Rutine is nt laded until it is called Better memry-space utilizatin; unused rutine is never laded All rutines kept n disk in relcatable lad frmat Useful when large amunts f cde are needed t handle infrequently ccurring cases N special supprt frm the perating system is required Implemented thrugh prgram design OS can help by prviding libraries t implement dynamic lading Dynamic Linking Static linking system libraries and prgram cde cmbined by the lader int the binary prgram image Dynamic linking linking pstpned until executin time Small piece f cde, stub, used t lcate the apprpriate memry-resident library rutine Stub replaces itself with the address f the rutine, and executes the rutine Operating system checks if rutine is in prcesses memry address If nt in address space, add t address space Dynamic linking is particularly useful fr libraries System als knwn as shared libraries Cnsider applicability t patching system libraries Versining may be needed Swapping A prcess can be swapped temprarily ut f memry t a backing stre, and then brught back int memry fr cntinued executin Ttal physical memry space f prcesses can exceed physical memry Backing stre fast disk large enugh t accmmdate cpies f all memry images fr all users; must prvide direct access t these memry images Rll ut, rll in swapping variant used fr pririty-based scheduling algrithms; lwerpririty prcess is swapped ut s higher-pririty prcess can be laded and executed Majr part f swap time is transfer time; ttal transfer time is directly prprtinal t the amunt f memry swapped CREC, Dept. f CSE Page 34

38 System maintains a ready queue f ready-t-run prcesses which have memry images n disk Des the swapped ut prcess need t swap back in t same physical addresses? Depends n address binding methd Plus cnsider pending I/O t / frm prcess memry space Mdified versins f swapping are fund n many systems (i.e., UNIX, Linux, and Windws) Swapping nrmally disabled Started if mre than threshld amunt f memry allcated Disabled again nce memry demand reduced belw threshld Cntiguus Allcatin Main memry usually int tw partitins: Resident perating system, usually held in lw memry with interrupt vectr User prcesses then held in high memry Each prcess cntained in single cntiguus sectin f memry Relcatin registers used t prtect user prcesses frm each ther, and frm changing perating-system cde and data Base register cntains value f smallest physical address Limit register cntains range f lgical addresses each lgical address must be less than the limit register MMU maps lgical address dynamically CREC, Dept. f CSE Page 35

39 Can then allw actins such as kernel cde being transient and kernel changing size Hardware Supprt fr Relcatin and Limit Registers Multiple-partitin allcatin Degree f multiprgramming limited by number f partitins Hle blck f available memry; hles f varius size are scattered thrughut memry When a prcess arrives, it is allcated memry frm a hle large enugh t accmmdate it Prcess exiting frees its partitin, adjacent free partitins cmbined Operating system maintains infrmatin abut: a) allcated partitins b) free partitins (hle) Dynamic Strage-Allcatin Prblem First-fit: Allcate the first hle that is big enugh Best-fit: Allcate the smallest hle that is big enugh; must search entire list, unless rdered by size Prduces the smallest leftver hle Wrst-fit: Allcate the largest hle; must als search entire list Prduces the largest leftver hle Fragmentatin External Fragmentatin ttal memry space exists t satisfy a request, but it is nt cntiguus Internal Fragmentatin allcated memry may be slightly larger than requested memry; this size difference is memry internal t a partitin, but nt being used CREC, Dept. f CSE Page 36

40 First fit analysis reveals that given N blcks allcated, 0.5 N blcks lst t fragmentatin 1/3 may be unusable ->50-percent rule Paging Physical address space f a prcess can be nncntiguus; prcess is allcated physical memry whenever the latter is available Divide physical memry int fixed-sized blcks called frames Size is pwer f 2, between 512 bytes and 16 Mbytes Divide lgical memry int blcks f same size called pages Keep track f all free frames T run a prgram f size N pages, need t find N free frames and lad prgram Set up a page table t translate lgical t physical addresses Backing stre likewise split int pages Still have Internal fragmentatin Address generated by CPU is divided int: Page number (p) used as an index int a page table which cntains base address f each page in physical memry Page ffset (d) cmbined with base address t define the physical memry address that is sent t the memry unit Paging Hardware CREC, Dept. f CSE Page 37

41 Paging Mdel f Lgical and Physical Memry Free Frames Paging Hardware With TLB CREC, Dept. f CSE Page 38

42 Valid (v) r Invalid (i) Bit In A Page Table CREC, Dept. f CSE Page 39

43 Shared Pages Example Structure f the Page Table Memry structures fr paging can get huge using straight-frward methds Cnsider a 32-bit lgical address space as n mdern cmputers Page size f 4 KB (2 12 ) Page table wuld have 1 millin entries (2 32 / 2 12) If each entry is 4 bytes -> 4 MB f physical address space / memry fr page table alne 4 That amunt f memry used t cst a lt 4 Dn t want t allcate that cntiguusly in main memry Hierarchical Paging Hashed Page Tables Inverted Page Tables Hierarchical Page Tables CREC, Dept. f CSE Page 40

44 Break up the lgical address space int multiple page tables A simple technique is a tw-level page table We then page the page table Hashed Page Tables Cmmn in address spaces > 32 bits The virtual page number is hashed int a page table This page table cntains a chain f elements hashing t the same lcatin Each element cntains (1) the virtual page number (2) the value f the mapped page frame (3) a pinter t the next element Virtual page numbers are cmpared in this chain searching fr a match If a match is fund, the crrespnding physical frame is extracted CREC, Dept. f CSE Page 41

45 Inverted Page Table Rather than each prcess having a page table and keeping track f all pssible lgical pages, track all physical pages One entry fr each real page f memry Entry cnsists f the virtual address f the page stred in that real memry lcatin, with infrmatin abut the prcess that wns that page Decreases memry needed t stre each page table, but increases time needed t search the table when a page reference ccurs Use hash table t limit the search t ne r at mst a few page-table entries TLB can accelerate access But hw t implement shared memry? One mapping f a virtual address t the shared physical address CREC, Dept. f CSE Page 42

46 Segmentatin Memry-management scheme that supprts user view f memry A prgram is a cllectin f segments A segment is a lgical unit such as: main prgram, prcedure,functin, methd, bject, lcal variables, glbal variables, cmmn blck, stack, symbl table, arrays Segmentatin Architecture Lgical address cnsists f a tw tuple: <segment-number, ffset>, Segment table maps tw-dimensinal physical addresses; each table entry has: base cntains the starting physical address where the segments reside in memry limit specifies the length f the segment Segment-table base register (STBR) pints t the segment table s lcatin in memry Segment-table length register (STLR) indicates number f segments used by a prgram; segment number s is legal if s<stlr CREC, Dept. f CSE Page 43

47 Virtual memry separatin f user lgical memry frm physical memry Only part f the prgram needs t be in memry fr executin Lgical address space can therefre be much larger than physical address space Allws address spaces t be shared by several prcesses Allws fr mre efficient prcess creatin Mre prgrams running cncurrently Less I/O needed t lad r swap prcesses Virtual memry can be implemented via: Demand paging Demand segmentatin CREC, Dept. f CSE Page 44

48 Demand Paging Culd bring entire prcess int memry at lad time Or bring a page int memry nly when it is needed Less I/O needed, n unnecessary I/O Less memry needed Faster respnse Mre users Page is needed reference t it invalid reference abrt nt-in-memry bring t memry Lazy swapper never swaps a page int memry unless page will be needed Swapper that deals with pages is a pager CREC, Dept. f CSE Page 45

49 Valid-Invalid Bit With each page table entry a valid invalid bit is assciated (v in-memry memry resident, i nt-in-memry) Initially valid invalid bit is set tin all entries Page Fault If there is a reference t a page, first reference t that page will trap t perating system: CREC, Dept. f CSE Page 46

50 page fault 1. Operating system lks at anther table t decide: Invalid reference abrt Just nt in memry 2. Get empty frame 3. Swap page int frame via scheduled disk peratin 4. Reset tables t indicate page nw in memry Set validatin bit = v 5. Restart the instructin that caused the page fault 6. Extreme case start prcess with n pages in memry OS sets instructin pinter t first instructin f prcess, nn-memry-resident -> page fault And fr every ther prcess pages n first access Pure demand paging 7. Actually, a given instructin culd access multiple pages -> multiple page faults Pain decreased because f lcality f reference 8. Hardware supprt needed fr demand paging Page table with valid / invalid bit Secndary memry (swap device with swap space) Instructin restart CREC, Dept. f CSE Page 47

51 Page Replacement Prevent ver-allcatin f memry by mdifying page-fault service rutine t include page replacement Use mdify (dirty) bit t reduce verhead f page transfers nly mdified pages are written t disk Page replacement cmpletes separatin between lgical memry and physical memry large virtual memry can be prvided n a smaller physical memry Need Fr Page Replacement CREC, Dept. f CSE Page 48

52 Page and Frame Replacement Algrithms Frame-allcatin algrithm determines Hw many frames t give each prcess Which frames t replace Page-replacement algrithm Want lwest page-fault rate n bth first access and re-access Evaluate algrithm by running it n a particular string f memry references (reference string) and cmputing the number f page faults n that string String is just page numbers, nt full addresses Repeated access t the same page des nt cause a page fault In all ur examples, the reference string is 7,0,1,2,0,3,0,4,2,3,0,3,0,3,2,1,2,0,1,7,0,1 First-In-First-Out (FIFO) Algrithm: Reference string: 7,0,1,2,0,3,0,4,2,3,0,3,0,3,2,1,2,0,1,7,0,1 3 frames (3 pages can be in memry at a time per prcess) CREC, Dept. f CSE Page 49

53 15 page faults Optimal Algrithm: Replace page that will nt be used fr lngest perid f time Least Recently Used (LRU) Algrithm: N f page faults: 9 Use past knwledge rather than future Replace page that has nt been used in the mst amunt f time Assciate time f last use with each page Page faults:12 LRU Apprximatin Algrithms LRU needs special hardware and still slw Reference bit CREC, Dept. f CSE Page 50

54 With each page assciate a bit, initially = 0 When page is referenced bit set t 1 Replace any with reference bit = 0 (if ne exists) 4 We d nt knw the rder, hwever Secnd-chance algrithm Generally FIFO, plus hardware-prvided reference bit Clck replacement If page t be replaced has 4 Reference bit = 0 -> replace it 4 reference bit = 1 then: set reference bit 0, leave page in memry replace next page, subject t same rules Cunting Algrithms CREC, Dept. f CSE Page 51

55 Keep a cunter f the number f references that have been made t each page l Nt cmmn LFU Algrithm: replaces page with smallest cunt MFU Algrithm: based n the argument that the page with the smallest cunt was prbably just brught in and has yet t be used Applicatins and Page Replacement All f these algrithms have OS guessing abut future page access Sme applicatins have better knwledge i.e. databases Memry intensive applicatins can cause duble buffering l l OS keeps cpy f page in memry as I/O buffer Applicatin keeps page in memry fr its wn wrk Operating system can given direct access t the disk, getting ut f the way f the applicatins l Raw disk mde Bypasses buffering, lcking, etc Allcatin f Frames Each prcess needs minimum number f frames Example: IBM pages t handle SS MOVE instructin: instructin is 6 bytes, might span 2 pages 2 pages t handle frm 2 pages t handle t Maximum f curse is ttal frames in the system Tw majr allcatin schemes fixed allcatin pririty allcatin Many variatins Fixed Allcatin Equal allcatin Fr example, if there are 100 frames (after allcating frames fr the OS) and 5 prcesses, give each prcess 20 frames Keep sme as free frame buffer pl CREC, Dept. f CSE Page 52

56 Prprtinal allcatin Allcate accrding t the size f prcess Dynamic as degree f multiprgramming, prcess sizes change Pririty Allcatin s i S m a i size f prcessp  s i ttalnumber f frames allcatin fr p i i m s s a si m a S ª ª Use a prprtinal allcatin scheme using pririties rather than size If prcess P i generates a page fault, select fr replacement ne f its frames select fr replacement a frame frm a prcess with lwer pririty number Glbal vs. Lcal Allcatin Glbal replacement prcess selects a replacement frame frm the set f all frames; ne prcess can take a frame frm anther But then prcess executin time can vary greatly But greater thrughput s mre cmmn Lcal replacement each prcess selects frm nly its wn set f allcated frames Mre cnsistent per-prcess perfrmance But pssibly underutilized memry Thrashing If a prcess des nt have enugh pages, the page-fault rate is very high Page fault t get page Replace existing frame But quickly need replaced frame back This leads t: 4 Lw CPU utilizatin 4 Operating system thinking that it needs t increase the degree f multiprgramming 4 Anther prcess added t the system CREC, Dept. f CSE Page 53

57 Thrashing a prcess is busy swapping pages in and ut Allcating Kernel Memry Treated differently frm user memry Often allcated frm a free-memry pl Kernel requests memry fr structures f varying sizes Sme kernel memry needs t be cntiguus 4 I.e. fr device I/O Buddy System Allcates memry frm fixed-size segment cnsisting f physically-cntiguus pages Memry allcated using pwer-f-2 allcatr Satisfies requests in units sized as pwer f 2 Request runded up t next highest pwer f 2 When smaller allcatin needed than is available, current chunk split int tw buddies f next-lwer pwer f 2 4 Cntinue until apprpriate sized chunk available Fr example, assume 256KB chunk available, kernel requests 21KB Split int A Land A r f 128KB each 4 One further divided int B L and B R f 64KB One further int C L and C R f 32KB each ne used t satisfy request Advantage quickly calesce unused chunks int larger chunk CREC, Dept. f CSE Page 54

58 Disadvantage - fragmentatin Slab Allcatr Alternate strategy Slab is ne r mre physically cntiguus pages Cache cnsists f ne r mre slabs Single cache fr each unique kernel data structure Each cache filled with bjects instantiatins f the data structure When cache created, filled with bjects marked as free When structures stred, bjects marked as used If slab is full f used bjects, next bject allcated frm empty slab If n empty slabs, new slab allcated Benefits include n fragmentatin, fast memry request satisfactin CREC, Dept. f CSE Page 55

59 The Deadlck Prblem A set f blcked prcesses each hlding a resurce and waiting t acquire a resurce held by anther prcess in the set Example System has 2 disk drives P 1 and P 2 each hld ne disk drive and each needs anther ne Example semaphres A and B, initialized t 1 P 0 P 1 wait (A); wait(b) wait (B); wait(a) System Mdel Resurce types R 1, R 2,..., R m CPU cycles, memry space, I/O devices Each resurce type R i has W i instances. Each prcess utilizes a resurce as fllws: request CREC, Dept. f CSE Page 56

60 use release Deadlck Characterizatin Deadlck can arise if fur cnditins hld simultaneusly. Mutual exclusin: nly ne prcess at a time can use a resurce Hld and wait: a prcess hlding at least ne resurce is waiting t acquire additinal resurces held by ther prcesses N preemptin: a resurce can be released nly vluntarily by the prcess hlding it, after that prcess has cmpleted its task Circular wait: there exists a set {P 0, P 1,, P n } f waiting prcesses such that P 0 is waiting fr a resurce that is held by P 1, P 1 is waiting fr a resurce that is held by P 2,,P n 1 is waiting fr a resurce that is held by P n, and P n is waiting fr a resurce that is held by P 0. Resurce-Allcatin Graph A set f vertices V and a set f edges E. V is partitined int tw types: P = {P 1, P 2,, P n }, the set cnsisting f all the prcesses in the system R = {R 1, R 2,, R m }, the set cnsisting f all resurce types in the system request edge directed edge P i ÆR j assignment edge directed edge R j ÆP i Resurce Allcatin Graph With A Deadlck CREC, Dept. f CSE Page 57

61 Graph With A Cycle But N Deadlck If graph cntains n cycles n deadlck If graph cntains a cycle if nly ne instance per resurce type, then deadlck if several instances per resurce type, pssibility f deadlck Methds fr Handling Deadlcks Ensure that the system will never enter a deadlck state Allw the system t enter a deadlck state and then recver Ignre the prblem and pretend that deadlcks never ccur in the system; used by mst perating systems, including UNIX Deadlck Preventin Restrain the ways request can be made CREC, Dept. f CSE Page 58

62 Mutual Exclusin nt required fr sharable resurces; must hld fr nnsharable resurces Hld and Wait must guarantee that whenever a prcess requests a resurce, it des nt hld any ther resurces Require prcess t request and be allcated all its resurces befre it begins executin, r allw prcess t request resurces nly when the prcess has nne Lw resurce utilizatin; starvatin pssible N Preemptin If a prcess that is hlding sme resurces requests anther resurce that cannt be immediately allcated t it, then all resurces currently being held are released Preempted resurces are added t the list f resurces fr which the prcess is waiting Prcess will be restarted nly when it can regain its ld resurces, as well as the new nes that it is requesting Circular Wait impse a ttal rdering f all resurce types, and require that each prcess requests resurces in an increasing rder f enumeratin Deadlck Avidance Requires that the system has sme additinal a priri infrmatin available Simplest and mst useful mdel requires that each prcess declare the maximum number f resurces f each type that it may need The deadlck-avidance algrithm dynamically examines the resurce-allcatin state t ensure that there can never be a circular-wait cnditin Resurce-allcatin state is defined by the number f available and allcated resurces, and the maximum demands f the prcesses Safe State When a prcess requests an available resurce, system must decide if immediate allcatin leaves the system in a safe state System is in safe state if there exists a sequence <P 1, P 2,, P n > f ALL the prcesses in the systems such that fr each P i, the resurces that P i can still request can be satisfied by currently available resurces + resurces held by all the P j, with j <I That is: If P i resurce needs are nt immediately available, then P i can wait until all P j have finished When P j is finished, P i can btain needed resurces, execute, return allcated resurces, and terminate CREC, Dept. f CSE Page 59

63 When P i terminates, P i +1 can btain its needed resurces, and s n If a system is in safe state n deadlcks If a system is in unsafe state pssibility f deadlck Avidance ensure that a system will never enter an unsafe state Avidance algrithms Single instance f a resurce type Use a resurce-allcatin graph Multiple instances f a resurce type Use the banker s algrithm Resurce-Allcatin Graph Scheme Claim edgep i ÆR j indicated that prcess P j may request resurce R j ; represented by a dashed line Claim edge cnverts t request edge when a prcess requests a resurce Request edge cnverted t an assignment edge when the resurce is allcated t the prcess When a resurce is released by a prcess, assignment edge recnverts t a claim edge Resurces must be claimed a priri in the system CREC, Dept. f CSE Page 60

64 Unsafe State In Resurce-Allcatin Graph Banker s Algrithm Multiple instances Each prcess must a priri claim maximum use When a prcess requests a resurce it may have t wait When a prcess gets all its resurces it must return them in a finite amunt f time Let n = number f prcesses, and m = number f resurces types. Available: Vectr f length m. If available [j] = k, there are k instances f resurce type R j available Max: n x m matrix. If Max [i,j] = k, then prcess P i may request at mst k instances f resurce type R j Allcatin: n x m matrix. If Allcatin[i,j] = k then P i is currently allcated k instances f R j Need: n x m matrix. If Need[i,j] = k, then P i may need k mre instances f R j t cmplete its task Need [i,j] = Max[i,j] Allcatin [i,j] CREC, Dept. f CSE Page 61

65 Safety Algrithm 1. Let Wrk and Finish be vectrs f length m and n, respectively. Initialize: Wrk = Available Finish [i] = false fri = 0, 1,,n Find an isuch that bth: (a) Finish [i] = false (b) Need i Wrk If n such iexists, g t step 4 3. Wrk = Wrk + Allcatin i Finish[i] = true g t step 2 4.IfFinish [i] == true fr all i, then the system is in a safe state Resurce-Request Algrithm fr Prcess P i Request = request vectr fr prcess P i. If Request i [j] = k then prcess P i wants k instances f resurce type R j 1. If Request i Need i g t step 2. Otherwise, raise errr cnditin, since prcess has exceeded its maximum claim 2. If Request i Available, g t step 3. Otherwise P i must wait, since resurces are nt available 3. Pretend t allcate requested resurces t P i by mdifying the state as fllws: Available = Available Request; Allcatin i = Allcatin i + Request i ; Need i =Need i Request i ; If safe the resurces are allcated t Pi If unsafe Pi must wait, and the ld resurce-allcatin state is restred Example f Banker s Algrithm 5 prcesses P 0 thrugh P 4 ; 3 resurce types: A (10 instances), B (5instances), and C (7 instances) Snapsht at time T 0 : CREC, Dept. f CSE Page 62

66 Allcatin Max Available A B C A B C A B C P P P P P The cntent f the matrix Need is defined t be Max Allcatin Need A B C P P P P P The system is in a safe state since the sequence <P 1, P 3, P 4, P 2, P 0 > satisfies safety criteria P 1 Request (1,0,2) Check that Request Available (that is, (1,0,2) (3,3,2) true Allcatin Need Available A B C A B C A B C P P P P P Executing safety algrithm shws that sequence <P 1, P 3, P 4, P 0, P 2 > satisfies safety requirement CREC, Dept. f CSE Page 63

67 Can request fr (3,3,0) by P 4 be granted? Can request fr (0,2,0) by P 0 be granted? Deadlck Detectin Allw system t enter deadlck state Detectin algrithm Recvery scheme Single Instance f Each Resurce Type Maintain wait-fr graph l l Ndes are prcesses P i ÆP j if P i is waiting frp j Peridically invke an algrithm that searches fr a cycle in the graph. If there is a cycle, there exists a deadlck An algrithm t detect a cycle in a graph requires an rder f n 2 peratins, where n is the number f vertices in the graph Resurce-Allcatin Graph and Wait-fr Graph Resurce-Allcatin Graph Crrespnding wait-fr graph Several Instances f a Resurce Type Available: A vectr f length m indicates the number f available resurces f each type. Allcatin: An n x m matrix defines the number f resurces f each type currently allcated t each prcess. Request: An n x m matrix indicates the current request f each prcess. If Request [i][j] = k, then prcess P i is requesting k mre instances f resurce type.r j. Detectin Algrithm Let Wrk and Finish be vectrs f length m and n, respectively Initialize: CREC, Dept. f CSE Page 64

68 (a) Wrk = Available (b) Fr i = 1,2,, n, if Allcatin i π 0, then Finish[i] = false; therwise, Finish[i] = true 2. Find an index isuch that bth: (a) Finish[i] == false (b) Request i Wrk If n such i exists, g t step 4 3. Wrk = Wrk + Allcatin i Finish[i] = true g t step 2 4. If Finish[i] == false, fr sme i, 1 i n, then the system is in deadlck state. Mrever, if Finish[i] == false, then P i is deadlcked Recvery frm Deadlck: Prcess Terminatin Abrt all deadlcked prcesses Abrt ne prcess at a time until the deadlck cycle is eliminated In which rder shuld we chse t abrt? Pririty f the prcess Hw lng prcess has cmputed, and hw much lnger t cmpletin Resurces the prcess has used Resurces prcess needs t cmplete Hw many prcesses will need t be terminated Is prcess interactive r batch? Resurce Preemptin Selecting a victim minimize cst Rllback return t sme safe state, restart prcess fr that state Starvatin same prcess may always be picked as victim, include number f rllback in cst factr CREC, Dept. f CSE Page 65

69 UNIT-4 Mass-Strage Systems, File-System Interface and Implementatin Overview f Mass Strage Structure Magnetic disks prvide bulk f secndary strage f mdern cmputers Drives rtate at 60 t 250 times per secnd Transfer rate is rate at which data flw between drive and cmputer Psitining time (randm-access time) is time t mve disk arm t desired cylinder (seek time) and time fr desired sectr t rtate under the disk head (rtatinal latency) Head crash results frm disk head making cntact with the disk surface 4 That s bad Disks can be remvable Drive attached t cmputer via I/O bus Busses vary, including EIDE, ATA, SATA, USB, Fibre Channel, SCSI, SAS, Firewire Hst cntrller in cmputer uses bus t talk t disk cntrller built int drive r strage array Magnetic Disks Platters range frm.85 t 14 (histrically) Cmmnly 3.5, 2.5, and 1.8 Range frm 30GB t 3TB per drive Perfrmance Transfer Rate theretical 6 Gb/sec Effective Transfer Rate real 1Gb/sec Seek time frm 3ms t 12ms 9ms cmmn fr desktp drives Average seek time measured r calculated based n 1/3 f tracks Latency based n spindle speed 4 1/(RPM * 60) Average latency = ½ latency CREC, Dept. f CSE Page 66

70 Magnetic Tape Was early secndary-strage medium Evlved frm pen spls t cartridges Relatively permanent and hlds large quantities f data Access time slw Randm access ~1000 times slwer than disk Mainly used fr backup, strage f infrequently-used data, transfer medium between systems Kept in spl and wund r rewund past read-write head Once data under head, transfer rates cmparable t disk 140MB/sec and greater 200GB t 1.5TB typical strage Cmmn technlgies are LTO-{3,4,5} and T10000 Disk Structure Disk drives are addressed as large 1-dimensinal arrays f lgical blcks, where the lgical blck is the smallest unit f transfer The 1-dimensinal array f lgical blcks is mapped int the sectrs f the disk sequentially Sectr 0 is the first sectr f the first track n the utermst cylinder Mapping prceeds in rder thrugh that track, then the rest f the tracks in that cylinder, and then thrugh the rest f the cylinders frm utermst t innermst Lgical t physical address shuld be easy CREC, Dept. f CSE Page 67

71 4 Except fr bad sectrs 4 Nn-cnstant # f sectrs per track via cnstant angular velcity Disk Attachment Hst-attached strage accessed thrugh I/O prts talking t I/O busses SCSI itself is a bus, up t 16 devices n ne cable, SCSI initiatr requests peratin and SCSI targets perfrm tasks Each target can have up t 8 lgical units (disks attached t device cntrller) FC is high-speed serial architecture Can be switched fabric with 24-bit address space the basis f stragearea netwrks (SANs) in which many hsts attach t many strage units I/O directed t bus ID, device ID, lgical unit (LUN) Strage Area Netwrk Cmmn in large strage envirnments Multiple hsts attached t multiple strage arrays flexible SAN is ne r mre strage arrays Cnnected t ne r mre Fibre Channel switches Hsts als attach t the switches Strage made available via LUN Masking frm specific arrays t specific servers Easy t add r remve strage, add new hst and allcate it strage Over lw-latency Fibre Channel fabric Why have separate strage netwrks and cmmunicatins netwrks? Cnsider iscsi, FCOE CREC, Dept. f CSE Page 68

72 Netwrk-Attached Strage Netwrk-attached strage (NAS) is strage made available ver a netwrk rather than ver a lcal cnnectin (such as a bus) Remtely attaching t file systems NFS and CIFS are cmmn prtcls Implemented via remte prcedure calls (RPCs) between hst and strage ver typically TCP r UDP n IP netwrk iscsi prtcl uses IP netwrk t carry the SCSI prtcl Remtely attaching t devices (blcks) Disk Scheduling The perating system is respnsible fr using hardware efficiently fr the disk drives, this means having a fast access time and disk bandwidth Minimize seek time Seek time ª seek distance Disk bandwidth is the ttal number f bytes transferred, divided by the ttal time between the first request fr service and the cmpletin f the last transfer There are many surces f disk I/O request OS System prcesses Users prcesses I/O request includes input r utput mde, disk address, memry address, number f sectrs t transfer OS maintains queue f requests, per disk r device Idle disk can immediately wrk n I/O request, busy disk means wrk must queue CREC, Dept. f CSE Page 69

73 Optimizatin algrithms nly make sense when a queue exists Nte that drive cntrllers have small buffers and can manage a queue f I/O requests (f varying depth ) Several algrithms exist t schedule the servicing f disk I/O requests The analysis is true fr ne r many platters We illustrate scheduling algrithms with a request queue (0-199) 98, 183, 37, 122, 14, 124, 65, 67 Head pinter 53 SSTF Shrtest Seek Time First selects the request with the minimum seek time frm the current head psitin SSTF scheduling is a frm f SJF scheduling; may cause starvatin f sme requests Illustratin shws ttal head mvement f 236 cylinders CREC, Dept. f CSE Page 70

74 SCAN The disk arm starts at ne end f the disk, and mves tward the ther end, servicing requests until it gets t the ther end f the disk, where the head mvement is reversed and servicing cntinues. SCAN algrithm Smetimes called the elevatr algrithm Illustratin shws ttal head mvement f 208 cylinders But nte that if requests are unifrmly dense, largest density at ther end f disk and thse wait the lngest C-SCAN CREC, Dept. f CSE Page 71

75 Prvides a mre unifrm wait time than SCAN The head mves frm ne end f the disk t the ther, servicing requests as it ges When it reaches the ther end, hwever, it immediately returns t the beginning f the disk, withut servicing any requests n the return trip Treats the cylinders as a circular list that wraps arund frm the last cylinder t the first ne Ttal number f cylinders? C-LOOK LOOK a versin f SCAN, C-LOOK a versin f C-SCAN Arm nly ges as far as the last request in each directin, then reverses directin immediately, withut first ging all the way t the end f the disk CREC, Dept. f CSE Page 72

76 Disk Management Lw-level frmatting, r physical frmatting Dividing a disk int sectrs that the disk cntrller can read and write Each sectr can hld header infrmatin, plus data, plus errr crrectin cde (ECC) Usually 512 bytes f data but can be selectable T use a disk t hld files, the perating system still needs t recrd its wn data structures n the disk Partitin the disk int ne r mre grups f cylinders, each treated as a lgical disk Lgical frmatting r making a file system T increase efficiency mst file systems grup blcks int clusters 4 Disk I/O dne in blcks 4 File I/O dne in clusters 4 Bt blck initializes system The btstrap is stred in ROM Btstrap lader prgram stred in bt blcks f bt partitin Methds such as sectr sparing used t handle bad blcks Swap-Space Management Swap-space Virtual memry uses disk space as an extensin f main memry Less cmmn nw due t memry capacity increases Swap-space can be carved ut f the nrmal file system, r, mre cmmnly, it can be in a separate disk partitin (raw) Swap-space management 4.3BSD allcates swap space when prcess starts; hlds text segment (the prgram) and data segment Kernel uses swap maps t track swap-space use Slaris 2 allcates swap space nly when a dirty page is frced ut f physical memry, nt when the virtual memry page is first created 4 File data written t swap space until write t file system requested 4 Other dirty pages g t swap space due t n ther hme 4 Text segment pages thrwn ut and reread frm the file system as needed CREC, Dept. f CSE Page 73

77 RAID Structure RAID multiple disk drives prvides reliability via redundancy Increases the mean time t failure Frequently cmbined with NVRAM t imprve write perfrmance RAID is arranged int six different levels Several imprvements in disk-use techniques invlve the use f multiple disks wrking cperatively Disk striping uses a grup f disks as ne strage unit RAID schemes imprve perfrmance and imprve the reliability f the strage system by string redundant data n n n Mirrring r shadwing (RAID 1) keeps duplicate f each disk Striped mirrrs (RAID 1+0) r mirrred stripes (RAID 0+1) prvides high perfrmance and high reliability Blck interleaved parity (RAID 4, 5, 6) uses much less redundancy RAID within a strage array can still fail if the array fails, s autmatic replicatin f the data between arrays is cmmn Frequently, a small number f ht-spare disks are left unallcated, autmatically replacing a failed disk and having data rebuilt nt them CREC, Dept. f CSE Page 74

78 File-System Interface File Cncept Cntiguus lgical address space Types: Data 4 numeric 4 character 4 binary Prgram File Structure Nne - sequence f wrds, bytes Simple recrd structure Lines Fixed length Variable length Cmplex Structures Frmatted dcument Relcatable lad file CREC, Dept. f CSE Page 75

79 Can simulate last tw with first methd by inserting apprpriate cntrl characters Wh decides: Operating system Prgram File Attributes Name nly infrmatin kept in human-readable frm Identifier unique tag (number) identifies file within file system Type needed fr systems that supprt different types Lcatin pinter t file lcatin n device Size current file size Prtectin cntrls wh can d reading, writing, executing Time, date, and user identificatin data fr prtectin, security, and usage mnitring Infrmatin abut files are kept in the directry structure, which is maintained n the disk File Operatins File is an abstract data type Create Write Read Repsitin within file Delete Truncate Open(F i ) search the directry structure n disk fr entry F i, and mve the cntent f entry t memry Clse (F i ) mve the cntent f entry F i in memry t directry structure n disk File Types Name, Extensin CREC, Dept. f CSE Page 76

80 Access Methds Sequential Access read next write next reset n read after last write (rewrite) Direct Access readn writen psitin t n read next write next rewriten n = relative blck number Sequential-access File CREC, Dept. f CSE Page 77

81 Simulatin f Sequential Access n Direct-access File Example f Index and Relative Files CREC, Dept. f CSE Page 78

82 Directry Structure A cllectin f ndes cntaining infrmatin abut all files Disk can be subdivided int partitins Disks r partitins can be RAID prtected against failure Disk r partitin can be used raw withut a file system, r frmatted with a file system Partitins als knwn as minidisks, slices Entity cntaining file system knwn as a vlume Each vlume cntaining file system als tracks that file system s inf in device directry r vlume table f cntents As well as general-purpse file systems there are many special-purpse file systems, frequently all within the same perating system r cmputer CREC, Dept. f CSE Page 79

83 Operatins Perfrmed n Directry Search fr a file Create a file Delete a file List a directry Rename a file Traverse the file system Single-Level Directry A single directry fr all users Tw-Level Directry Separate directry fr each user CREC, Dept. f CSE Page 80

84 Path name Can have the same file name fr different user Efficient searching N gruping capability Tree-Structured Directries Efficient searching Gruping Capability Current directry (wrking directry) cd /spell/mail/prg type list CREC, Dept. f CSE Page 81

85 Acyclic-Graph Directries Tw different names (aliasing) If dict deletes list dangling pinter Slutins: Backpinters, s we can delete all pinters Variable size recrds a prblem Backpinters using a daisy chain rganizatin Entry-hld-cunt slutin New directry entry type Link anther name (pinter) t an existing file Reslve the link fllw pinter t lcate the file General Graph Directry CREC, Dept. f CSE Page 82

86 File System Munting A file system must be munted befre it can be accessed A unmunted file system (i.e., Fig (b)) is munted at a munt pint (a) Existing (b) Unmunted Partitin File Sharing Sharing f files n multi-user systems is desirable Sharing may be dne thrugh a prtectin scheme On distributed systems, files may be shared acrss a netwrk Netwrk File System (NFS) is a cmmn distributed file-sharing methd File Sharing Multiple Users User IDs identify users, allwing permissins and prtectins t be per-user Grup IDs allw users t be in grups, permitting grup access rights CREC, Dept. f CSE Page 83

87 Remte File Systems Uses netwrking t allw file system access between systems Manually via prgrams like FTP Autmatically, seamlessly using distributed file systems Semi autmatically via the wrld wide web Client-server mdel allws clients t munt remte file systems frm servers Server can serve multiple clients Client and user-n-client identificatin is insecure r cmplicated NFS is standard UNIX client-server file sharing prtcl CIFS is standard Windws prtcl Standard perating system file calls are translated int remte calls Distributed Infrmatin Systems (distributed naming services) such as LDAP, DNS, NIS, Active Directry implement unified access t infrmatin needed fr remte cmputing Failure Mdes Remte file systems add new failure mdes, due t netwrk failure, server failure Recvery frm failure can invlve state infrmatin abut status f each remte request Stateless prtcls such as NFS include all infrmatin in each request, allwing easy recvery but less security Cnsistency Semantics Cnsistency semantics specify hw multiple users are t access a shared file simultaneusly Similar t Ch 7 prcess synchrnizatin algrithms 4 Tend t be less cmplex due t disk I/O and netwrk latency (fr remte file systems Andrew File System (AFS) implemented cmplex remte file sharing semantics Unix file system (UFS) implements: 4 Writes t an pen file visible immediately t ther users f the same pen file 4 Sharing file pinter t allw multiple users t read and write cncurrently AFS has sessin semantics 4 Writes nly visible t sessins starting after the file is clsed File System Implementatin\ CREC, Dept. f CSE Page 84

88 File-System Structure File structure Lgical strage unit Cllectin f related infrmatin File system resides n secndary strage (disks) Prvided user interface t strage, mapping lgical t physical Prvides efficient and cnvenient access t disk by allwing data t be stred, lcated retrieved easily Disk prvides in-place rewrite and randm access I/O transfers perfrmed in blcks f sectrs (usually 512 bytes) File cntrl blck strage structure cnsisting f infrmatin abut a file Device driver cntrls the physical device File system rganized int layers Layered File System File-System Implementatin We have system calls at the API level, but hw d we implement their functins? On-disk and in-memry structures Bt cntrl blck cntains inf needed by system t bt OS frm that vlume Needed if vlume cntains OS, usually first blck f vlume Vlume cntrl blck (superblck, master file table) cntains vlume details Ttal # f blcks, # f free blcks, blck size, free blck pinters r array Directry structure rganizes the files CREC, Dept. f CSE Page 85

89 Names and inde numbers, master file table Per-file File Cntrl Blck (FCB) cntains many details abut the file Inde number, permissins, size, dates NFTS stres int in master file table using relatinal DB structures A Typical File Cntrl Blck In-Memry File System Structures Munt table string file system munts, munt pints, file system types The fllwing figure illustrates the necessary file system structures prvided by the perating systems Figure 12-3(a) refers t pening a file Figure 12-3(b) refers t reading a file Plus buffers hld data blcks frm secndary strage Open returns a file handle fr subsequent use Data frm read eventually cpied t specified user prcess memry address CREC, Dept. f CSE Page 86

90 Virtual File Systems Virtual File Systems (VFS) n Unix prvide an bject-riented way f implementing file systems VFS allws the same system call interface (the API) t be used fr different types f file systems Separates file-system generic peratins frm implementatin details Implementatin can be ne f many file systems types, r netwrk file system 4 Implements vndes which hld indes r netwrk file details Then dispatches peratin t apprpriate file system implementatin rutines The API is t the VFS interface, rather than any specific type f file system CREC, Dept. f CSE Page 87

91 Directry Implementatin Linear list f file names with pinter t the data blcks Simple t prgram Time-cnsuming t execute 4 Linear search time 4 Culd keep rdered alphabetically via linked list r use B+ tree Hash Table linear list with hash data structure Decreases directry search time Cllisins situatins where tw file names hash t the same lcatin Only gd if entries are fixed size, r use chained-verflw methd Allcatin Methds Cntiguus An allcatin methd refers t hw disk blcks are allcated fr files: Cntiguus allcatin each file ccupies set f cntiguus blcks Best perfrmance in mst cases Simple nly starting lcatin (blck #) and length (number f blcks) are required Prblems include finding space fr file, knwing file size, external fragmentatin, need fr cmpactin ff-line (dwntime) r n-line CREC, Dept. f CSE Page 88

92 Linked Linked allcatin each file a linked list f blcks File ends at nil pinter N external fragmentatin Each blck cntains pinter t next blck N cmpactin, external fragmentatin Free space management system called when new blck needed Imprve efficiency by clustering blcks int grups but increases internal fragmentatin Reliability can be a prblem Lcating a blck can take many I/Os and disk seeks FAT (File Allcatin Table) variatin Beginning f vlume has table, indexed by blck number Much like a linked list, but faster n disk and cacheable New blck allcatin simple CREC, Dept. f CSE Page 89

93 File-Allcatin Table Indexed Indexed allcatin Each file has its wn index blck(s) f pinters t its data blcks Free-Space Management File system maintains free-space list t track available blcks/clusters Linked list (free list) Cannt get cntiguus space easily N waste f space N need t traverse the entire list (if # free blcks recrded) Linked Free Space List n Disk CREC, Dept. f CSE Page 90

94 Gruping Mdify linked list t stre address f next n-1 free blcks in first free blck, plus a pinter t next blck that cntains free-blck-pinters (like this ne). Cunting Because space is frequently cntiguusly used and freed, with cntiguus-allcatin allcatin, extents, r clustering. Keep address f first free blck and cunt f fllwing free blcks. Free space list then has entries cntaining addresses and cunts. CREC, Dept. f CSE Page 91

95 I/O Hardware UNIT-5 I/O Systems, Prtectin, Security Incredible variety f I/O devices Strage Transmissin Human-interface Cmmn cncepts signals frm I/O devices interface with cmputer Prt cnnectin pint fr device Bus - daisy chain r shared direct access Cntrller (hst adapter) electrnics that perate prt, bus, device A Typical PC Bus Structure 4 Smetimes integrated 4 Smetimes separate circuit bard (hst adapter) 4 Cntains prcessr, micrcde, private memry, bus cntrller, etc Sme talk t per-device cntrller with bus cntrller, micrcde, memry, etc I/O instructins cntrl devices CREC, Dept. f CSE Page 92

96 Devices usually have registers where device driver places cmmands, addresses, and data t write, r read data frm registers after cmmand executin l l Data-in register, data-ut register, status register, cntrl register Typically 1-4 bytes, r FIFO buffer Devices have addresses, used by l l Direct I/O instructins Memry-mapped I/O 4 Device data and cmmand registers mapped t prcessr address space 4 Especially fr large address spaces (graphics) Plling Fr each byte f I/O 1. Read busy bit frm status register until 0 2. Hst sets read r write bit and if write cpies data int data-ut register 3. Hst sets cmmand-ready bit 4. Cntrller sets busy bit, executes transfer 5. Cntrller clears busy bit, errr bit, cmmand-ready bit when transfer dne 6. Step 1 is busy-wait cycle t wait fr I/O frm device 7. Reasnable if device is fast 8. But inefficient if device slw 9. CPU switches t ther tasks? 4 But if miss a cycle data verwritten / lst Interrupts Plling can happen in 3 instructin cycles Read status, lgical-and t extract status bit, branch if nt zer Hw t be mre efficient if nn-zer infrequently? CPU Interrupt-request line triggered by I/O device Checked by prcessr after each instructin Interrupt handler receives interrupts Maskable t ignre r delay sme interrupts Interrupt vectr t dispatch interrupt t crrect handler CREC, Dept. f CSE Page 93

97 Cntext switch at start and end Based n pririty Sme nn maskable Interrupt chaining if mre than ne device at same interrupt number Interrupt-Driven I/O Cycle Interrupt mechanism als used fr exceptins Terminate prcess, crash system due t hardware errr Page fault executes when memry access errr System call executes via trap t trigger kernel t execute request Multi-CPU systems can prcess interrupts cncurrently If perating system designed t handle it Used fr time-sensitive prcessing, frequent, must be fast CREC, Dept. f CSE Page 94

98 Direct Memry Access Used t avid prgrammed I/O (ne byte at a time) fr large data mvement Requires DMA cntrller Bypasses CPU t transfer data directly between I/O device and memry OS writes DMA cmmand blck int memry Surce and destinatin addresses Read r write mde Cunt f bytes Writes lcatin f cmmand blck t DMA cntrller Bus mastering f DMA cntrller grabs bus frm CPU When dne, interrupts t signal cmpletin Six Step Prcess t Perfrm DMA Transfer Applicatin I/O Interface: I/O system calls encapsulate device behavirs in generic classes Device-driver layer hides differences amng I/O cntrllers frm kernel New devices talking already-implemented prtcls need n extra wrk Each OS has its wn I/O subsystem structures and device driver framewrks CREC, Dept. f CSE Page 95

99 Devices vary in many dimensins Character-stream r blck Sequential r randm-access Synchrnus r asynchrnus (r bth) Sharable r dedicated Speed f peratin read-write, read nly, r write nly A Kernel I/O Structure Characteristics f I/O Devices CREC, Dept. f CSE Page 96

100 Subtleties f devices handled by device drivers Bradly I/O devices can be gruped by the OS int Blck I/O Character I/O (Stream) Memry-mapped file access Netwrk sckets Fr direct manipulatin f I/O device specific characteristics, usually an escape / back dr Unix ictl() call t send arbitrary bits t a device cntrl register and data t device data register Blck and Character Devices Blck devices include disk drives Cmmands include read, write, seek Raw I/O, direct I/O, r file-system access Memry-mapped file access pssible 4 File mapped t virtual memry and clusters brught via demand paging DMA Character devices include keybards, mice, serial prts Cmmands include get(), put() Libraries layered n tp allw line editing Netwrk Devices Varying enugh frm blck and character t have wn interface Unix and Windws NT/9x/2000 include scket interface Separates netwrk prtcl frm netwrk peratin Includes select() functinality Appraches vary widely (pipes, FIFOs, streams, queues, mailbxes) Clcks and Timers Prvide current time, elapsed time, timer Nrmal reslutin abut 1/60 secnd CREC, Dept. f CSE Page 97

101 Sme systems prvide higher-reslutin timers Prgrammable interval timer used fr timings, peridic interrupts ictl() (n UNIX) cvers dd aspects f I/O such as clcks and timers Blcking and Nnblcking I/O Blcking - prcess suspended until I/O cmpleted Easy t use and understand Insufficient fr sme needs Nnblcking - I/O call returns as much as available User interface, data cpy (buffered I/O) Implemented via multi-threading Returns quickly with cunt f bytes read r written select() t find if data ready then read() r write() t transfer Asynchrnus - prcess runs while I/O executes Difficult t use I/O subsystem signals prcess when I/O cmpleted Tw I/O Methds Kernel I/O Subsystem Scheduling Sme I/O request rdering via per-device queue CREC, Dept. f CSE Page 98

102 Sme OSs try fairness Sme implement Quality Of Service (i.e. IPQOS) Buffering - stre data in memry while transferring between devices T cpe with device speed mismatch T cpe with device transfer size mismatch T maintain cpy semantics Duble buffering tw cpies f the data 4 Kernel and user 4 Varying sizes 4 Full / being prcessed and nt-full / being used 4 Cpy-n-write can be used fr efficiency in sme cases Device-status Table Caching - faster device hlding cpy f data Always just a cpy Key t perfrmance Smetimes cmbined with buffering Spling - hld utput fr a device If device can serve nly ne request at a time i.e., Printing CREC, Dept. f CSE Page 99

103 Device reservatin - prvides exclusive access t a device System calls fr allcatin and de-allcatin Watch ut fr deadlck Errr Handling OS can recver frm disk read, device unavailable, transient write failures Retry a read r write, fr example Sme systems mre advanced Slaris FMA, AIX 4 Track errr frequencies, stp using device with increasing frequency f retry-able errrs Mst return an errr number r cde when I/O request fails System errr lgs hld prblem reprts I/O Prtectin User prcess may accidentally r purpsefully attempt t disrupt nrmal peratin via illegal I/O instructins All I/O instructins defined t be privileged I/O must be perfrmed via system calls Use f a System Call t Perfrm I/O Memry-mapped and I/O prt memry lcatins must be prtected CREC, Dept. f CSE Page 100

104 Kernel Data Structures Kernel keeps state inf fr I/O cmpnents, including pen file tables, netwrk cnnectins, character device state Many, many cmplex data structures t track buffers, memry allcatin, dirty blcks Sme use bject-riented methds and message passing t implement I/O Windws uses message passing 4 Message with I/O infrmatin passed frm user mde int kernel 4 Message mdified as it flws thrugh t device driver and back t prcess 4 Prs / cns? UNIX I/O Kernel Structure Transfrming I/O Requests t Hardware Operatins Cnsider reading a file frm disk fr a prcess: Determine device hlding file Translate name t device representatin Physically read data frm disk int buffer CREC, Dept. f CSE Page 101

105 Make data available t requesting prcess Return cntrl t prcess Life Cycle f An I/O Request CREC, Dept. f CSE Page 102

106 Prtectin Gals f Prtectin: In ne prtectin mdel, cmputer cnsists f a cllectin f bjects, hardware r sftware Each bject has a unique name and can be accessed thrugh a well-defined set f peratins Prtectin prblem - ensure that each bject is accessed crrectly and nly by thse prcesses that are allwed t d s Principles f Prtectin Guiding principle principle f least privilege Prgrams, users and systems shuld be given just enugh privileges t perfrm their tasks Limits damage if entity has a bug, gets abused Can be static (during life f system, during life f prcess) Or dynamic (changed by prcess as needed) dmain switching, privilege escalatin Need t knw a similar cncept regarding access t data Must cnsider grain aspect Rugh-grained privilege management easier, simpler, but least privilege nw dne in large chunks Fr example, traditinal Unix prcesses either have abilities f the assciated user, r f rt Fine-grained management mre cmplex, mre verhead, but mre prtective File ACL lists, RBAC Dmain can be user, prcess, prcedure Dmain Structure Access-right = <bject-name, rights-set> where rights-set is a subset f all valid peratins that can be perfrmed n the bject Dmain = set f access-rights CREC, Dept. f CSE Page 103

107 Dmain Implementatin (UNIX) Dmain = user-id Dmain switch accmplished via file system 4 Each file has assciated with it a dmain bit (setuid bit) 4 When file is executed and setuid = n, then user-id is set t wner f the file being executed 4 When executin cmpletes user-id is reset 4 Dmain switch accmplished via passwrds su cmmand temprarily switches t anther user s dmain when ther dmain s passwrd prvided Dmain switching via cmmands sud cmmand prefix executes specified cmmand in anther dmain (if riginal dmain has privilege r passwrd given) Dmain Implementatin (MULTICS) Let D i and D j be any tw dmain rings If j < I D i Õ D j CREC, Dept. f CSE Page 104

108 Access Matrix View prtectin as a matrix (access matrix) Rws represent dmains Clumns represent bjects Access(i, j) is the set f peratins that a prcess executing in Dmain i can invke n Object j Use f Access Matrix If a prcess in Dmain D i tries t d p n bject O j, then p must be in the access matrix User wh creates bject can define access clumn fr that bject Can be expanded t dynamic prtectin Operatins t add, delete access rights Special access rights: 4 wner f O i 4 cpy p frm O i t O j (dented by * ) 4 cntrl D i can mdify D j access rights 4 transfer switch frm dmain D i t D j Cpy and Owner applicable t an bject Cntrl applicable t dmain bject Access matrix design separates mechanism frm plicy Mechanism 4 Operating system prvides access-matrix + rules CREC, Dept. f CSE Page 105

109 4 If ensures that the matrix is nly manipulated by authrized agents and that rules are strictly enfrced Plicy 4 User dictates plicy 4 Wh can access what bject and in what mde 4 But desn t slve the general cnfinement prblem Access Matrix f Figure A with Dmains as Objects Access Matrix with Cpy Rights Access Matrix With Owner Rights CREC, Dept. f CSE Page 106

110 Implementatin f Access Matrix Generally, a sparse matrix Optin 1 Glbal table Stre rdered triples < dmain, bject, rights-set > in table A requested peratin M n bject O j within dmain D i -> search table fr < D i, O j, R k > 4 with M R k But table culd be large -> wn t fit in main memry Difficult t grup bjects (cnsider an bject that all dmains can read) Optin 2 Access lists fr bjects Each clumn implemented as an access list fr ne bject CREC, Dept. f CSE Page 107

111 Resulting per-bject list cnsists f rdered pairs < dmain, rights-set > defining all dmains with nn-empty set f access rights fr the bject Easily extended t cntain default set -> If M default set, als allw access Each clumn = Access-cntrl list fr ne bject Defines wh can perfrm what peratin Dmain 1 = Read, Write Dmain 2 = Read Dmain 3 = Read Each Rw = Capability List (like a key) Fr each dmain, what peratins allwed n what bjects Object F1 Read Object F4 Read, Write, Execute Object F5 Read, Write, Delete, Cpy Optin 3 Capability list fr dmains Instead f bject-based, list is dmain based Capability list fr dmain is list f bjects tgether with peratins allws n them Object represented by its name r address, called a capability Execute peratin M n bject O j, prcess requests peratin and specifies capability as parameter 4 Pssessin f capability means access is allwed Capability list assciated with dmain but never directly accessible by dmain 4 Rather, prtected bject, maintained by OS and accessed indirectly 4 Like a secure pinter 4 Idea can be extended up t applicatins Optin 4 Lck-key Cmprmise between access lists and capability lists Each bject has list f unique bit patterns, called lcks CREC, Dept. f CSE Page 108

112 Each dmain as list f unique bit patterns called keys Prcess in a dmain can nly access bject if dmain has key that matches ne f the lcks Access Cntrl Prtectin can be applied t nn-file resurces Slaris 10 prvides rle-based access cntrl (RBAC) t implement least privilege Privilege is right t execute system call r use an ptin within a system call Can be assigned t prcesses Users assigned rles granting access t privileges and prgrams 4 Enable rle via passwrd t gain its privileges Similar t access matrix Revcatin f Access Rights Varius ptins t remve the access right f a dmain t an bject Immediate vs. delayed Selective vs. general CREC, Dept. f CSE Page 109

113 Partial vs. ttal Temprary vs. permanent Access List Delete access rights frm access list Simple search access list and remve entry Immediate, general r selective, ttal r partial, permanent r temprary Capability List Scheme required t lcate capability in the system befre capability can be revked Reacquisitin peridic delete, with require and denial if revked Back-pinters set f pinters frm each bject t all capabilities f that bject (Multics) Indirectin capability pints t glbal table entry which pints t bject delete entry frm glbal table, nt selective (CAL) Keys unique bits assciated with capability, generated when capability created Capability-Based Systems 4 Master key assciated with bject, key matches master key fr access 4 Revcatin create new master key 4 Plicy decisin f wh can create and mdify keys bject wner r thers? Hydra Fixed set f access rights knwn t and interpreted by the system 4 i.e. read, write, r execute each memry segment 4 User can declare ther auxiliary rights and register thse with prtectin system 4 Accessing prcess must hld capability and knw name f peratin 4 Rights amplificatin allwed by trustwrthy prcedures fr a specific type Interpretatin f user-defined rights perfrmed slely by user's prgram; system prvides access prtectin fr use f these rights Operatins n bjects defined prcedurally prcedures are bjects accessed indirectly by capabilities Slves the prblem f mutually suspicius subsystems Includes library f prewritten security rutines CREC, Dept. f CSE Page 110

114 Cambridge CAP System Simpler but pwerful Data capability - prvides standard read, write, execute f individual strage segments assciated with bject implemented in micrcde Sftware capability -interpretatin left t the subsystem, thrugh its prtected prcedures Language-Based Prtectin 4 Only has access t its wn subsystem 4 Prgrammers must learn principles and techniques f prtectin Specificatin f prtectin in a prgramming language allws the high-level descriptin f plicies fr the allcatin and use f resurces Language implementatin can prvide sftware fr prtectin enfrcement when autmatic hardware-supprted checking is unavailable Interpret prtectin specificatins t generate calls n whatever prtectin system is prvided by the hardware and the perating system The Security Prblem: Security System secure if resurces used and accessed as intended under all circumstances l Unachievable Intruders (crackers) attempt t breach security Threat is ptential security vilatin Attack is attempt t breach security Attack can be accidental r malicius Easier t prtect against accidental than malicius misuse Security Vilatin Categries Breach f cnfidentiality Unauthrized reading f data Breach f integrity Unauthrized mdificatin f data Breach f availability Unauthrized destructin f data CREC, Dept. f CSE Page 111

115 Theft f service Unauthrized use f resurces Denial f service (DOS) Preventin f legitimate use Security Vilatin Methds Masquerading (breach authenticatin) Pretending t be an authrized user t escalate privileges Replay attack As is r with message mdificatin Man-in-the-middle attack Intruder sits in data flw, masquerading as sender t receiver and vice versa Sessin hijacking Intercept an already-established sessin t bypass authenticatin Standard Security Attacks CREC, Dept. f CSE Page 112

116 Security Measure Levels Security must ccur at fur levels t be effective: Physical 4 Data centers, servers, cnnected terminals Human 4 Avid scial engineering, phishing, dumpster diving Operating System 4 Prtectin mechanisms, debugging Netwrk 4 Intercepted cmmunicatins, interruptin, DOS Security is as weak as the weakest link in the chain Prgram Threats Many variatins, many names Trjan Hrse Cde segment that misuses its envirnment Explits mechanisms fr allwing prgrams written by users t be executed by ther users Spyware, pp-up brwser windws, cvert channels Up t 80% f spam delivered by spyware-infected systems Trap Dr Specific user identifier r passwrd that circumvents nrmal security prcedures Culd be included in a cmpiler Lgic Bmb Prgram that initiates a security incident under certain circumstances Stack and Buffer Overflw Explits a bug in a prgram (verflw either the stack r memry buffers) Failure t check bunds n inputs, arguments Write past arguments n the stack int the return address n stack When rutine returns frm call, returns t hacked address n Pinted t cde laded nt stack that executes malicius cde CREC, Dept. f CSE Page 113

117 Unauthrized user r privilege escalatin Viruses Cde fragment embedded in legitimate prgram Self-replicating, designed t infect ther cmputers Very specific t CPU architecture, perating system, applicatins Usually brne via r as a macr n Visual Basic Macr t refrmat hard drive Sub AutOpen() Dim FS Set FS = CreateObject( Scripting.FileSystemObject ) vs = Shell( c:cmmand.cm /k frmat c:,vbhide) End Sub Virus drpper inserts virus nt the system Many categries f viruses, literally many thusands f viruses File / parasitic Bt / memry Macr Surce cde Plymrphic t avid having a virus signature Encrypted Stealth Tunneling Multipartite Armred CREC, Dept. f CSE Page 114

118 A Bt-sectr Cmputer Virus System and Netwrk Threats Sme systems pen rather than secure by default Reduce attack surface But harder t use, mre knwledge needed t administer Netwrk threats harder t detect, prevent Prtectin systems weaker Mre difficult t have a shared secret n which t base access N physical limits nce system attached t internet 4 Or n netwrk with system attached t internet Even determining lcatin f cnnecting system difficult 4 IP address is nly knwledge Wrms use spawn mechanism; standalne prgram Internet wrm Explited UNIX netwrking features (remte access) and bugs in finger and sendmail prgrams Explited trust-relatinship mechanism used by rsh t access friendly systems withut use f passwrd CREC, Dept. f CSE Page 115

119 Grappling hk prgram upladed main wrm prgram 4 99 lines f C cde Hked system then upladed main cde, tried t attack cnnected systems Als tried t break int ther users accunts n lcal system via passwrd guessing If target system already infected, abrt, except fr every 7 th time The Mrris Internet Wrm Prt scanning Autmated attempt t cnnect t a range f prts n ne r a range f IP addresses Detectin f answering service prtcl Detectin f OS and versin running n system nmap scans all prts in a given IP range fr a respnse nessus has a database f prtcls and bugs (and explits) t apply against a system Frequently launched frm zmbie systems 4 T decrease trace-ability CREC, Dept. f CSE Page 116

120 Denial f Service Overlad the targeted cmputer preventing it frm ding any useful wrk Distributed denial-f-service (DDOS) cme frm multiple sites at nce Cnsider the start f the IP-cnnectin handshake (SYN) 4 Hw many started-cnnectins can the OS handle? Cnsider traffic t a web site Accidental CS students writing bad frk() cde Purpseful extrtin, punishment Cryptgraphy as a Security Tl Bradest security tl available Internal t a given cmputer, surce and destinatin f messages can be knwn and prtected 4 OS creates, manages, prtects prcess IDs, cmmunicatin prts Surce and destinatin f messages n netwrk cannt be trusted withut cryptgraphy 4 Lcal netwrk IP address? Cnsider unauthrized hst added 4 WAN / Internet hw t establish authenticity Nt via IP address Cryptgraphy Means t cnstrain ptential senders (surces) and / r receivers (destinatins) f messages Based n secrets (keys) Enables 4 Cnfirmatin f surce 4 Receipt nly by certain destinatin 4 Trust relatinship between sender and receiver CREC, Dept. f CSE Page 117

121 Secure Cmmunicatin ver Insecure Medium Encryptin Encryptin algrithm cnsists f Set K f keys Set M f Messages Set C f ciphertexts (encrypted messages) A functin E : K (M C). That is, fr each k ΠK, E(k) is a functin fr generating ciphertexts frm messages 4 Bth E and E(k) fr any k shuld be efficiently cmputable functins A functin D : K (C M). That is, fr each k ΠK, D(k) is a functin fr generating messages frm ciphertexts 4 Bth D and D(k) fr any k shuld be efficiently cmputable functins An encryptin algrithm must prvide this essential prperty: Given a ciphertext c ΠC, a cmputer can cmpute m such that E(k)(m) = c nly if it pssesses D(k) CREC, Dept. f CSE Page 118

122 Thus, a cmputer hlding D(k) can decrypt ciphertexts t the plaintexts used t prduce them, but a cmputer nt hlding D(k) cannt decrypt ciphertexts Since ciphertexts are generally expsed (fr example, sent n the netwrk), it is imprtant that it be infeasible t derive D(k) frm the ciphertexts Symmetric Encryptin Same key used t encrypt and decrypt E(k) can be derived frm D(k), and vice versa DES is mst cmmnly used symmetric blck-encryptin algrithm (created by US Gvt) Encrypts a blck f data at a time Triple-DES cnsidered mre secure Advanced Encryptin Standard (AES), twfish up and cming RC4 is mst cmmn symmetric stream cipher, but knwn t have vulnerabilities Encrypts/decrypts a stream f bytes (i.e., wireless transmissin) Key is a input t psued-randm-bit generatr Asymmetric Encryptin 4 Generates an infinite keystream Public-key encryptin based n each user having tw keys: public key published key used t encrypt data private key key knwn nly t individual user used t decrypt data Must be an encryptin scheme that can be made public withut making it easy t figure ut the decryptin scheme Mst cmmn is RSA blck cipher Efficient algrithm fr testing whether r nt a number is prime N efficient algrithm is knw fr finding the prime factrs f a number Frmally, it is cmputatinally infeasible t derive D(k d, N) frm E(k e, N), and s E(k e, N) need nt be kept secret and can be widely disseminated E(k e, N) (r just k e ) is the public key D(k d, N) (r just k d ) is the private key N is the prduct f tw large, randmly chsen prime numbers p and q (fr example, p and q are 512 bits each) CREC, Dept. f CSE Page 119

123 Encryptin algrithm is E(k e, N)(m) = m ke md N, where k e satisfies k e k d md (p 1)(q 1) = 1 The decryptin algrithm is then D(k d, N)(c) = c kd md N Asymmetric Encryptin Example Fr example. make p = 7and q = 13 We then calculate N = 7 13 = 91 and (p 1)(q 1) = 72 We next select k e relatively prime t 72 and< 72, yielding 5 Finally,we calculate k d such that k e k d md 72 = 1, yielding 29 We hw have ur keys Public key, k e, N = 5, 91 Private key, k d, N = 29, 91 Encrypting the message 69 with the public key results in the cyphertext 62 Cyphertext can be decded with the private key Public key can be distributed in cleartext t anyne wh wants t cmmunicate with hlder f public key Authenticatin Cnstraining set f ptential senders f a message Cmplementary and smetimes redundant t encryptin Als can prve message unmdified Algrithm cmpnents A set K f keys A set M f messages A set A f authenticatrs A functin S : K (M A) 4 That is, fr each k ΠK, S(k) is a functin fr generating authenticatrs frm messages 4 Bth S and S(k) fr any k shuld be efficiently cmputable functins A functin V : K (M A {true, false}). That is, fr each k ΠK, V(k) is a functin fr verifying authenticatrs n messages 4 Bth V and V(k) fr any k shuld be efficiently cmputable functins CREC, Dept. f CSE Page 120

124 Fr a message m, a cmputer can generate an authenticatr a ΠA such that V(k)(m, a) = true nly if it pssesses S(k) Thus, cmputer hlding S(k) can generate authenticatrs n messages s that any ther cmputer pssessing V(k) can verify them Cmputer nt hlding S(k) cannt generate authenticatrs n messages that can be verified using V(k) Since authenticatrs are generally expsed (fr example, they are sent n the netwrk with the messages themselves), it must nt be feasible t derive S(k) frm the authenticatrs Authenticatin Hash Functins Basis f authenticatin Creates small, fixed-size blck f data (message digest, hash value) frm m Hash Functin H must be cllisin resistant n m Must be infeasible t find an m m such that H(m) = H(m ) If H(m) = H(m ), then m = m The message has nt been mdified Cmmn message-digest functins include MD5, which prduces a 128-bit hash, and SHA-1, which utputs a 160-bit hash Authenticatin MAC Symmetric encryptin used in message-authenticatin cde (MAC) authenticatin algrithm Simple example: MAC defines S(k)(m) = f (k, H(m)) Authenticatin Digital Signature 4 Where f is a functin that is ne-way n its first argument k cannt be derived frm f (k, H(m)) 4 Because f the cllisin resistance in the hash functin, reasnably assured n ther message culd create the same MAC 4 A suitable verificatin algrithm is V(k)(m, a) ( f (k,m) = a) 4 Nte that k is needed t cmpute bth S(k) and V(k), s anyne able t cmpute ne can cmpute the ther Based n asymmetric keys and digital signature algrithm CREC, Dept. f CSE Page 121

125 Authenticatrs prduced are digital signatures In a digital-signature algrithm, cmputatinally infeasible t derive S(k s ) frm V(k v ) V is a ne-way functin Thus, k v is the public key and k s is the private key Cnsider the RSA digital-signature algrithm Similar t the RSA encryptin algrithm, but the key use is reversed Digital signature f message S(k s )(m) = H(m) ks md N The key k s again is a pair d, N, where N is the prduct f tw large, randmly chsen prime numbers p and q Verificatin algrithm is V(k v )(m, a) (a kv md N = H(m)) 4 Where k v satisfies k v k s md (p 1)(q 1) = 1 Key Distributin Delivery f symmetric key is huge challenge Smetimes dne ut-f-band Asymmetric keys can prliferate stred n key ring Even asymmetric key distributin needs care man-in-the-middle attack Digital Certificates Prf f wh r what wns a public key Public key digitally signed a trusted party Trusted party receives prf f identificatin frm entity and certifies that public key belngs t entity Certificate authrity are trusted party their public keys included with web brwser distributins They vuch fr ther authrities via digitally signing their keys, and s n User Authenticatin Crucial t identify user crrectly, as prtectin systems depend n user ID User identity mst ften established thrugh passwrds, can be cnsidered a special case f either keys r capabilities Passwrds must be kept secret Frequent change f passwrds Histry t avid repeats CREC, Dept. f CSE Page 122

126 Use f nn-guessable passwrds Lg all invalid access attempts (but nt the passwrds themselves) Unauthrized transfer Passwrds may als either be encrypted r allwed t be used nly nce Des encrypting passwrds slve the expsure prblem? 4 Might slve sniffing 4 Cnsider shulder surfing 4 Cnsider Trjan hrse keystrke lgger Passwrds Encrypt t avid having t keep secret But keep secret anyway (i.e. Unix uses superuser-nly readably file /etc/shadw) Use algrithm easy t cmpute but difficult t invert Only encrypted passwrd stred, never decrypted Add salt t avid the same passwrd being encrypted t the same value One-time passwrds Use a functin based n a seed t cmpute a passwrd, bth user and cmputer Hardware device / calculatr / key fb t generate the passwrd 4 Changes very frequently Bimetrics Sme physical attribute (fingerprint, hand scan) Multi-factr authenticatin Need tw r mre factrs fr authenticatin Implementing Security Defenses 4 i.e. USB dngle, bimetric measure, and passwrd Defense in depth is mst cmmn security thery multiple layers f security Security plicy describes what is being secured Vulnerability assessment cmpares real state f system / netwrk cmpared t security plicy Intrusin detectin endeavrs t detect attempted r successful intrusins CREC, Dept. f CSE Page 123

127 Signature-based detectin spts knwn bad patterns Anmaly detectin spts differences frm nrmal behavir 4 Can detect zer-day attacks False-psitives and false-negatives a prblem Virus prtectin Auditing, accunting, and lgging f all r specific system r netwrk activities Firewalling t Prtect Systems and Netwrks A netwrk firewall is placed between trusted and untrusted hsts The firewall limits netwrk access between these tw security dmains Can be tunneled r spfed Tunneling allws disallwed prtcl t travel within allwed prtcl (i.e., telnet inside f HTTP) Firewall rules typically based n hst name r IP address which can be spfed Persnal firewall is sftware layer n given hst Can mnitr / limit traffic t and frm the hst Applicatin prxy firewall understands applicatin prtcl and can cntrl them (i.e., SMTP) System-call firewall mnitrs all imprtant system calls and apply rules t them (i.e., this prgram can execute that system call) Netwrk Security Thrugh Dmain Separatin Via Firewall CREC, Dept. f CSE Page 124