T : Malware Analysis and Antivirus Technologies Windows Operating System

Size: px

Start display at page:

Download "T : Malware Analysis and Antivirus Technologies Windows Operating System"

Paul Cummings
5 years ago
Views:

1 T : Malware Analysis and Antivirus Technologies Windows Operating System Antti Tikkanen, Protecting the irreplaceable f-secure.com

2 Lecture Agenda 1. Applications on Windows 2. Processes and Threads 3. Windows Architecture 4. System Mechanisms 5. Management Mechanisms 6. Memory Management 7. File Systems 8. Security Mechanisms 9. Driver Basics 2

3 Applications on Windows 3

4 Executable Format Object files and executables follow the PE (Portable Executable) file format Full specification available online firmware/pecoff.mspx More on this in Reverse Engineering III 4

5 Windows Executables Filename extension hints to the executable type EXE = executable application, anything from a DOS executable to 64-bit Windows applications DLL = Dynamic link library, a set of callable routines compiled together as a loadable file SYS = Driver OBJ = Object file, input to the linker Note that Windows does not really care much about the file extension You can execute britneyspears.jpg just fine! All of these follow the PE specification 5

6 Windows API Windows API (aka. Win32 API) is the interface to the operating system for applications Exposed by a set of system libraries: kernel32.dll, user32.dll, Several subcategories Administration and management (WMI, ) Diagnostics (event logging, ) Networking Security System services (processes, threads, registry ) MSDN is the reverse engineers best friend for Windows binaries 6

Native API Undocumented interface to OS functionality One level below Windows API Some low-level functionality only available through Native API

7 Native API Undocumented interface to OS functionality One level below Windows API Some low-level functionality only available through Native API Examples of interesting functions NtSetSystemInformation NtQuerySystemInformation NtQueryDirectoryFile See Windows NT/2000 Native API Reference by Nebbett 7

8 Windows Architecture 8

9 Simplified Windows Architecture System Support Processes Service Processes User Applications Environment Subsystems Subsystem DLL s NTDLL.DLL Kernel Executive Device Drivers Windowing & Graphics HAL 9

10 System Mechanisms 10

11 Kernel-mode & user-mode 0xFFFFFFFF System-space (Ring 0) Int 0x2e / Sysenter 0x User-space User-space (Ring User-space 3) (Ring 3) (Ring 3) 11

12 System Service Dispatching Interrupt Read the file Return to caller Call NtReadFile Dismiss interrupt Nt! NtReadFile Nt! KiSystemService Sysenter Return to caller Call NtReadFile Return to caller ReadFile( ) Ntdll.dll! NtReadFile Kernel32.dll! ReadFile Application 12

13 System Service Dispatching 13

14 System Service Dispatching 14

15 Processes & Threads 15

16 Processes Abstraction of an executing program A process has A private virtual address space (user-mode address space) A base image (the main executable) A set of open handles to OS resources An access token ( who is this process, what can it do ) A process ID (PID) One or more threads Job = A group of processes that can be managed as a unit 16

17 Protected Processes (Vista and later) By default, a process with a token containing the debug privilege can obtain any access to all other processes Admins should have control over the machine, right To support playback of high-quality digital content, Vista introduced Protected Processes Determined by a flag in the EPROCESS structure of the kernel process object Process image file must be signed with a special certificate Protected processes cannot be manipulated from other processes Can be terminated, though Examples: audiodg.exe, werfault.exe 17

18 Threads A thread is what gets scheduled for execution on the CPU A thread has A context (the state of execution) A list of exception handlers Two stacks, one for user-mode and one for kernel-mode A thread ID (TID) An access token Thread-Local Storage (TLS), a per-thread storage area Fiber = Manually scheduled unit of execution, shares the context of its owning thread 18

19 Processes & Threads 19

20 Case Study: FU Rootkit The FU rootkit hides processes by unlinking them from the kernels process list Why do those processes still get execution time? 20

21 Case Study: DLL Injection A technique to run code in the context of another process by forcing it to load a DLL Motivation 1. Stealth: no extra processes in process list 2. Data capture using API hooks 3. Making reverse engineering more difficult 4. Avoiding HIPS features Various ways to accomplish DLL injection on Windows Registry: AppInit_DLLs Using remote threads SetWindowsHookEx From the kernel using APC s 21

22 What does this code do? 22

23 TEB & PEB TEB = Thread Environment Block Container for thread-specific things like the exception handler list, stack pointer, Windows uses the fs segment to store it (offset 0x18 has pointer to self) mov eax, fs[18] PEB = Process Environment Block Container for process-specific things like the list of loaded modules TEB has a pointer to PEB at offset 0x30 Important when reversing code that Enumerates loaded modules (Peb.Ldr) Checks for an attached debugger (PEB.BeingDebugged) Installs an exception handler (TEB.NtTib.ExceptionList) 23

24 Example: Checking For a Debugger ; Call IsDebuggerPresent() call [IsDebuggerPresent] test eax, eax ; Do the same by checking PEB mov eax, large fs:18h ; Offset 18h has self-pointer to TEB mov eax, [eax+30h] ; Offset 30h has pointer to PEB movzx eax, byte ptr [eax+2] ; PEB.BeingDebugged test eax, eax 24

25 Example: Installing an Exception Handler ; Install a SEH exception handler push offset_my_handler ; pointer to our handler push fs:[0] ; pointer to old exception record mov fs:[0], esp ; update TEB.NtTib.ExceptionList 25

26 Memory Management 26

27 Memory Manager Each process on Windows sees a large, continuous address space The memory manager has two important tasks 1. Mapping access to virtual memory into physical memory 2. Paging contents of virtual memory to disk as physical memory runs out, and paging data back to memory when it s needed 27

28 Virtual Memory Each process has private, virtual address space Paging = the process of transferring memory contents to and from disk Amount of virtual memory can exceed physical memory 28

29 Virtual Memory (x86) Total of 4 GB virtual memory By default, only lower 2 GB accessible from user-mode Upper 2 GB reserved for kernel-mode and shared between all processes 29

30 Management Mechanisms 30

31 Registry A directory for storing settings and configuration data for the Operating System and applications Think of it as a huge.ini file Basic concepts Hive Key Value Hives are just files, most under %SystemRoot%\System32\Config 31

32 Registry Hive Format 32

33 Registry Root Keys HKEY_LOCAL_MACHINE (HKLM) System-wide information HKEY_USERS (HKU) User-specific settings for all accounts HKEY_CURRENT_USER (HKCU) A link to the current user under HKEY_USERS HKEY_CLASSES_ROOT (HKCR) File associations and COM registration, links to HKLM\Software\Classes HKEY_PERFORMANCE_DATA HKEY_CURRENT_CONFIG Current HW profile, links to HKLM\System\CurrentControlSet\Hardware Profiles\Current 33

34 Registry and Malware Malware wants to survive a reboot Registry is the most common way of starting after reboot Hundreds of launchpoints HKLM\Software\Microsoft\Windows\CurrentVersion\Run:MyApp HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\explorer.exe:Debugger Malware also wants to change (security) settings for other components Windows Firewall, IE extensions and settings, Windows File Protection, The registry is also a great source for forensic data, for example: HKEY_CURRENT_USER\Software\Microsoft\Windows\ShellNoRoam\MUICache HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\ UserAssist 34

35 Services Services are background processes that usually perform a specific task and require no user-interaction For example, Automatic Updates Controlled by the Service Control Manager (SCM), services.exe Configuration data under HKLM\System\CurrentControlSet\Services Different types of services Kernel drivers Separate process Shared process (hosted by svchost.exe) 35

36 Demo: Services 1. Which process is hosting the Automatic Updates service? 2. What file implements the service? DEMO 36

37 File Systems 37

38 Windows File System Formats CDFS (cdfs.sys) CD-ROM filesystem, considered legacy UDF (udfs.sys) Universal Disk Format, the standardized format for optical storage FAT12, FAT16, FAT32 (fastfat.sys) Legacy filesystem, mostly replaced by NTFS exfat (aka. FAT64) Adds functionality on top of FAT (file size limit increase, ACL s) NTFS 38

39 NTFS Native file system for Windows Support for advanced features ACL s on files and directories Alternate Data Streams Disk quotas Sparse files Compression and encryption Soft and hard links Transactional semantics 39

40 Demo: Alternate Data Streams DEMO 40

41 Security Mechanisms 41

42 Security & Objects Almost everything on Windows is an object Process, thread, desktop, mutex, Basic concepts Security Identifier (SID) is a unique ID for any actor S A token identifies the security context of a process Member of Administrators group, can shut down OS Security Descriptor specifies who can do what to an object Owner Discretionary Access Control List (DACL) Privileges 42

43 Access Check 43

44 Driver Basics 44

45 I/O Subsystem A set of components in the kernel that manage and provide access to hardware devices I/O Manager Plug and Play Manager Power Manager Key concepts Driver Device I/O requests 45

46 I/O Manager The core of the I/O system Provides a framework for other components to have device independent I/O services Responsible for dispatching the service requests to the appropriate device drivers for further processing Packet-driven (IRP s, I/O request packets) Handles creation and destruction of IRP s Offers uniform interface for drivers that handle IRP s 46

47 Driver Basics Drivers are loadable kernel-mode components Code in drivers gets executed in different contexts: 1. In the user thread that initiated I/O 2. A system thread 3. As a result of an interrupt (any thread) Different types of drivers: file system drivers, protocol drivers, hardware drivers Layered driver model 47

48 Reverse Engineering Drivers Interesting elements 1. Initialization routine (DriverEntry) The entrypoint of a driver Sets up global variables, initializes handler functions 2. Add-device routine Called by PnP manager for PnP drivers when a new device appears 3. Dispatch routines Main functionality of most drivers ( read, write, close ) In many cases the interesting stuff is here 48

Chapter 3: Malicious Code Environments, from 3.1 through 3.

49 Suggested Material Sysinternals tools Process Monitor Process Explorer Autoruns The Art of Computer Virus Research and Defense Chapter 3: Malicious Code Environments, from 3.1 through 3.6 Chapter 12: Memory Scanning and Disinfection Windows Internals, 5 th edition 49

Malware Analysis and Antivirus Technologies: Windows Operating System

Malware Analysis and Antivirus Technologies: Windows Operating System Protecting the irreplaceable f-secure.com Lecture Agenda 1. Applications on Windows 2. Processes and Threads 3. Windows Architecture