CIS 5373 Systems Security Topic 3.1: OS Security Basics of secure design Endadul Hoque
Slide Acknowledgment Contents are based on slides from Ninghui Li (Purdue), John Mitchell (Stanford), Dan Boneh (Stanford) 2
What security goals does an operating system provide? Originally: time-sharing computers: enabling multiple users to securely share a computer Separation and sharing of processes, memory, files, devices, etc. What is the threat model? Users may be malicious, users have terminal access to computers, software may be malicious/buggy, and so on Security mechanisms Memory protection Processor modes User authentication File access control 3
What security goals does an operating More recent past and present: Networked desktop computers: ensure secure operation in networked environment New threat? Attackers coming from the network. Network-facing programs on computers may be buggy. Users may be hurt via online communication. Security mechanisms Authentication; Access Control Secure Communication (using cryptography) Logging & Auditing system provide? Intrusion Prevention and Detection Recovery 4
What security goals does an operating Present and near future: mobile computing devices New threat? Apps (programs) may be malicious or questionable. More tightly connected with personal life of the owner. Security mechanisms? Isolation of each app. system provide? Help users assess risks of apps. Risk communication. 5
Principles of Secure Design Compartmentalization Isolation Principle of least privilege Defense in depth Use more than one security mechanism Secure the weakest link Fail securely Keep it simple 6
Principles of Least Privilege Principle of Least Privilege A system module should only have the minimal privileges needed for its intended purposes What s a privilege? Ability to access or modify a resource Assumes compartmentalization and isolation Separate the system into isolated compartments Limit interaction between compartments 7
Monolithic design Network User input File system System Network User device File system 8
Monolithic design Network User input File system System Network User device File system 9
Monolithic design Network User input File system System Network User device File system 10
Component design Network User input File system Network User display File system 11
Component design Network User input File system Network User display File system 12
Component design Network User input File system Network User display File system 13
Example: Mail Agent Requirements Receive and send email over external network Place incoming email into local user inbox files Sendmail Traditional Unix Monolithic design Historical source of many vulnerabilities Qmail Compartmentalized design 14
Memory Protection: Access Control to Memory Ensures that one user s process cannot access other s memory fence relocation base/bounds register segmentation paging Operating system and user processes need to have different privileges Reading: http://flylib.com/books/en/4.270.1.46/1/ 15
CPU modes (aka Processor modes or privilege) System mode (privileged mode, master mode, supervisor mode, kernel mode) Can execute any instruction Can access any memory locations, e.g., accessing hardware devices, Can enable and disable interrupts, Can change privileged processor state, Can access memory management units, Can modify registers for various descriptor tables Reading: http://en.wikipedia.org/wiki/cpu_modes 16
CPU modes (aka Processor modes or User mode privilege) Access to memory is limited, Cannot execute some instructions Cannot disable interrupts, Cannot change arbitrary processor state, Cannot access memory management units Transition from user mode to system mode can only happen via well defined entry points, i.e., through system calls Reading: http://en.wikipedia.org/wiki/cpu_modes 17
System Calls Guarded gates from user mode (space, land) into kernel mode (space, land) use a special CPU instruction (often an interruption) to transfer control to predefined entry point in more privileged code allows the more privileged code to specify where it will be entered as well as important processor state at the time of entry the higher privileged code, by examining processor state set by the less privileged code and/or its stack, determines what is being requested and whether to allow it Reading: http://en.wikipedia.org/wiki/system_call 18
Kernel space vs User space Part of the OS runs in the kernel model known as the OS kernel Other parts of the OS run in the user mode, including service programs (daemon programs), user applications, etc. they run as processes they form the user space (or the user land) Why is root user more powerful than normal users? 19
Privilege Levels Security is often achieved by running control/protection code at a higher privilege level Components running at the same level can be isolated by a higher-privilege component If attack and defense are at the same level, then it is an arms race and there can be no guarantee 20
OS Security ISOLATION THE CONFINEMENT PRINCIPLE 21
Running untrusted code We often need to run buggy/unstrusted code: programs from untrusted Internet sites: apps, extensions, plug-ins, codecs for media player exposed applications: pdf viewers, outlook legacy daemons: sendmail, bind Honeypots Goal: if application misbehaves kill it 22
Approach: Confinement Confinement: ensure misbehaving app cannot harm rest of system Can be implemented at many levels: Hardware: run application on isolated hw (air gap) app 1 app 2 Network 2 air gap network 1 difficult to manage 23
Approach: Confinement Confinement: ensure misbehaving app cannot harm rest of system Can be implemented at many levels: Virtual machines: isolate OS s on a single machine app1 app2 OS 1 OS 2 Virtual Machine Monitor (VMM) 24
Approach: Confinement Confinement: ensure misbehaving app cannot harm rest of system Can be implemented at many levels: Process: System Call Interposition Isolate a process in a single operating system process 2 process 1 Operating System 25
Approach: Confinement Confinement: ensure misbehaving app cannot harm rest of system Can be implemented at many levels: Threads: Software Fault Isolation (SFI) Isolating threads sharing same address space Application: e.g. browser-based confinement 26
Implementing Confinement Key component: reference monitor Mediates requests from applications Implements protection policy Enforces isolation and confinement Must always be invoked: Every application request must be mediated Tamperproof: Reference monitor cannot be killed or if killed, then monitored process is killed too Small enough to be analyzed and validated 27
System call interposition Observation: to damage host system (e.g. persistent changes) app must make system calls: To delete/overwrite files: unlink, open, write To do network attacks: socket, bind, connect, send Idea: monitor app s system calls and block unauthorized calls Implementation options: Completely kernel space (e.g. GSWTK) Completely user space (e.g. program shepherding) Hybrid (e.g. Systrace) 28
System call interposition Linux ptrace: process tracing process calls: ptrace (, pid_t pid, ) and wakes up when pid makes sys call. monitored application (browser) monitor user space open( /etc/passwd, r ) OS Kernel Monitor kills application if request is disallowed 29
System call interposition Complications: If app forks, monitor must fork Forked-monitor monitors forked-app cd( /tmp ) open( passwd, r ) cd( /etc ) open( passwd, r ) If monitor crashes, app must be killed Monitor must maintain all OS state associated with app current-working-dir (CWD), UID, EUID, GID When app does cd path monitor must update its CWD otherwise: relative path requests interpreted incorrectly 30
Virtual Machines VM2 Apps Apps VM1 Guest OS 2 Guest OS 1 Virtual Machine Monitor (VMM) Host OS Hardware 31
VMM security assumption VMM Security assumption: Malware can infect guest OS and guest apps But malware cannot escape from the infected VM Cannot infect host OS Cannot infect other VMs on the same hardware Requires that VMM protect itself and is not buggy VMM is much simpler than full OS but device drivers run in Host OS 32
Problem: Covert channels Covert channel: unintended communication channel between isolated components Can be used to leak classified data from secure component to public component Classified VM secret doc malware covert channel VMM Public VM listener 33
An example covert channel Both VMs use the same underlying hardware To send a bit b {0,1} malware does: b = 1: at 1:00am do CPU intensive calculation b = 0: at 1:00am do nothing At 1:00am listener does CPU intensive calc. and measures completion time b = 1 completion-time > threshold Many covert channels exist in running system: File lock status, cache contents, interrupts, Difficult to eliminate all 34
VMM introspection protecting IDS/AV systems Intrusion detection / Anti-virus Runs as part of OS kernel and user space process Kernel root kit can shutdown protection system Common practice for modern malware Standard solution: run IDS system in the network Problem: insufficient visibility into user s machine 35
VMM introspection protecting IDS/AV systems Better: run IDS as part of VMM (protected from malware) VMM can monitor virtual hardware for anomalies VMI: Virtual Machine Introspection Allows VMM to check Guest OS internals malware IDS Infected VM Guest OS VMM Subversion of VMI is possible Hardware 36
Coming up Access control 37