J. A. Drew Hamilton, Jr., Ph.D. Director, Center for Cyber Innovation Professor, Computer Science & Engineering

Transcription

1 J. A. Drew Hamilton, Jr., Ph.D. Director, Center for Cyber Innovation Professor, Computer Science & Engineering CCI Post Office Box 9627 Mississippi State, MS Voice: (662) Fax: (662)

2 Time Machines Lesson 4 Reference: Farmer & Venema 2

3 A Timing Analysis Case Study On August 20th of 2001, Barney, a harmless Linux computer previously used as a group's multimedia juke box, was found to have an ssh daemon (a program that enables encrypted network logins) listening for connections on a very strange TCP port. When no one confessed to installing the program it seemed clear that Barney had been compromised. In a rush to help the situation, Barney's administrators created a backup of all the directories that appeared to contain suspicious files. An alert was then sent to the corporate computer security staff. It took three days, but finally the security team quarantined the computer, The Coroner's Toolkit (TCT) was unpacked ( the suspect disk drive examined, and a story started to unfold. The team knew what had happened, but wanted to know when and, if possible, why, it did. 3

4 System Times MACtimes are a shorthand way to refer to the three time attributes - mtime, atime, and ctime - that are attached to any file or directory in UNIX, Linux, and other file systems. Microsoft's file systems have four similar times - ChangeTime, CreationTime, LastAccessTime, and LastWriteTime. Linux also has the dtime attribute which is set when a file or directory has been deleted. In particular this doesn't affect files in the visible file system, only deleted files.. Atime refers to the last time the file or directory was accessed. Mtimes, in contrast, are changed by modifying a file's contents. The ctime attribute keeps track of when the content or meta information about the file has changed - the owner, group, file permission, etc. Ctime may also be used as an approximation of when a file was deleted. For all of these attributes, however, it is crucial to note the word "last" - MACtimes only keep track of the last time a file is disturbed; once it has been changed historical MACtime data is impossible to uncover. 4

5 Reviewing Time Measurement On UNIX systems these times may be viewed by th ls command (see the ls man page for more details) and on NTFS by various 3rd party tools. In real situations, however, it's often easier to use aq mactime tool or to simply use the lstat() system call (which mactime itself uses) as evidenced by this simple Perl code fragment: ($dev, $inode, $mode, $nlink, $uid, $gid, $rdev, $size, $atime, $mtime, $ctime, $blksize, $blocks) = lstat($filename);print "$filename (MAC): $mtime, $atime,$ctime n MACtimes returned by the Perl lstat() function call are displayed as the number of seconds since January 1st, 1970, 00:00:00 UTC. NTFS keeps file times in 100 nanosecond chunks since Jan 1st, 1901; lstat() converts this. 5

6 MACtimes from Barney This looks very similar to UNIX's ls -l output. The difference here is the inclusion of the "MAC" column. This shows which of the three file time attributes (mtime, atime, and ctime) correspond to the dates and times in the first column No atimes -- files were copied for safekeeping. OOV says that more ephemeral data should be harvested before more stable data, and in this case reading (or copying) a file will change the atime attribute to the time the file was reads 6

7 Altering Time lstat()'ing a file does not change the MACtimes, opening a directory for reading will change the atime, so you must be certain to lstat() directories before opening them and examining their contents. Be cautious if using GUI-based file system management tools - many of these change the atime even when only listing files, as they read the file to figure out what icon should be displayed in the file listing. Digital hashes of file content are very useful for a variety of forensic or administrative purposes, but must be done after the lstat() because reading a file changes the atime of that file. If doing a serious investigation you'll ideally want to work from a duplicate of the media rather than using the original data. Failing that, mount the media read-only or at the very least turn off atime updates so that you don't inadvertently destroy or alter the data and come up with incorrect conclusions. 7

8 Altering time stamps MACtimes can only report on the last time a file has been disturbed and hence have no way of reporting on the historical activity of a file or directory. A program could run a thousand times and you'd only see evidence of a single occurrence. Another limitation is that MACtimes only show you the result of an action - not who did it. MACtimes also degrade over time, displaying a sort of digital Alzheimer's. MACtimes are less useful on busy multi-user systems, because user activity becomes difficult to distinguish from intruder activity. MACtimes also don't help much when normal system activity resembles the kind of trouble that you wish to investigate. Finally, MACtimes are easily forged. UNIX systems have the touch command that can change atimes and mtimes. Both Microsoft's NTFS and UNIX file systems can also use the utime() system call to change those two times. 8

9 Argus (the network Audit Record Generation and Utilization System) Argus is software that reports on the network status and traffic that it listens to. In the Barney case study there were two things to look for: connections to the rogue ssh daemon (the port the program was listening to, TCP 33332, was unusual enough that it could be readily spotted even in large quantities of data; ironically if the intruder had simply placed it on ssh's normal port, it might have never been noticed) a file transfer that might have placed the tar file onto Barney. In this example Barney's IP address was and the intruder came from Finding the first session to the new ssh daemon was easy - it lasted 17 minutes, as seen by this slightly edited Argus output. Argus appends the port number to the IP address, and the "ssefc" status flags indicate a complete TCP connection: 9

10 Spotting Further Connections Just prior to the ssh connection the intruder entered from a second system and downloaded something to Barney with FTP (an FTP server uses TCP ports 20 and 21 to send data and receive commands.) from Possibly the ssh tar file that was downloaded earlier. Comparing the various sources of data revealed that the time on the Argus system and Barney's differed by some 17 minutes (purely coincidental to the duration of the initial ssh connection). Clock skews such as this are very common and can provide endless amounts of frustration when trying to correlate evidence from different sources. 10

11 Connecting to port In this example Barney's IP address was and the intruder came from Finding the first session to the new ssh daemon was easy - it lasted 17 minutes, as seen by this slightly edited Argus output. Argus appends the port number to the IP address, and the "ssefc" status flags indicate a complete TCP connection: 11

12 Back Tracing Just prior to the ssh connection the intruder entered from a second system and downloaded something to Barney with FTP (an FTP server uses TCP ports 20 and 21 to send data and receive commands.) from This is quite possibly the ssh tar file that was downloaded earlier. 12

13 Clock Drift Comparing the various sources of data revealed that the time on the Argus system and Barney's differed by some 17 minutes (purely coincidental to the duration of the initial ssh connection). Clock skews such as this are very common and can provide endless amounts of frustration when trying to correlate evidence from different sources. If we scan the Argus logs further back we see the computer at scanning the network for back doors on TCP port 110 (the POP3 mail service) and TCP port 21 (the ftp port.) We note that all the connections are from TCP source port presumably such an unusual occurrence is not merely a coincidence. A connection lasting four and a half minutes to ftp suggests that there might have been a back door previously installed on Barney ("sr" status flags mean a connection has been refused): 13

14 Checking Connections from port The unusual port numbers used by the attackers warranted additional searching, and certainly finding additional connections from TCP port was easy enough. Not only did we find the above traffic but another suspicious trail involving the same Barney machine, starting almost a year earlier - August 22, Barney was apparently compromised through the name daemon port (TCP port 53) by what was probably a server vulnerability. 14

15 Barney Incident Timeline 15

16 Checking Log Files - pcat Many times, you would like to examine or search a part of a system rather than a relatively large subsection. TCT's pcat command, which captures the raw memory contained in a process, can be used to find any date strings within the currently running syslogd process: 16

17 Verification of logs - 2nd source This shows what is currently in the processes' memory - some log entries span several months! While how much and what kind of data is in a running process varies wildly from system to system, process to process, and the activity levels of the computer in question, this can be an invaluable source of information. Here the log entries could be checked against what is in the actual system logs - if the entries in memory are not present in the log file, then something is amiss. TCT's Lazarus automatically categorizes data based on the content that it finds, and may be useful in finding not only time-based data but giving form to arbitrary content in the program. 17

18 DNS & Time DNS has several types of records: PTR (Pointer records, which map an IP number to a host name), A (Address records, which map the computer's name to an IP number), MX (Mail Exchange records, which tell mail agents where should be sent to.) Bind maintains an in-memory cache of recent lookup results. On request it can dump this cache in an orderly manner. The request is made via the ndc or rndc command, or by sending a SIGINT signal (e.g. "kill -SIGINT bind-pid"). And while Bind doesn't keep the explicit time for each of the requests, it does display the time the data has left in the cache (this is called its Time To Live, or TTL) before it will discard the data. Figure below shows a snippet from an rndc dump of the Bind program with this. 18

19 Timing Query -- TTL If you were able to obtain the real TTL value and subtract Bind's time left for a specific request in the cache you will - in theory - know how long ago the query happened. We can get TTLs from the Internet for any DNS resource record, using the host command. If you were running your own caching name server it would save the TTL (10800, in this case) and subsequent requests would show the TTL counter decrementing (normally this value should remain the same from query to query.) 19

20 How DNS Times are Generated In order to get a definitive value for a TTL you must ask an authoritative name server and look at the TTL that comes back with it, or, if using your own server ensure that you clear its cache first, as shown below. Parse output using scripting languages. 20

21 Fragments of BIND s Processed Memory Cache The A records here are when our SMTP mailer wanted to send mail to another site needed to look up the IP address from the host name. The PTR record was when a computer was probing our ssh daemon, which logged the IP address along with the resolved host name. 21

22 Journaling file systems and MACtimes Journaling file systems are not new -- MAC OS X, Linux and Microsoft Windows. Examples include Ext3fs, JFS, NTFS, Reiserfs, XFS, and others. With a journaling file system, part of all of the disk updates are first written to a journal file before they are committed to the file system itself [Robbins, 2001]. Journaling significantly improves recovery from a system crash. Depending on what optimizations the file system is allowed to make, journaling does not need not to cause loss of performance. Every non-trivial file system operation such as creating or appending a file results in a sequence of disk updates that affect both file data (content) and file metadata (the location of file content, and what files belong to a directory). 22

23 Leveraging Journaled File Systems When a sequence of updates is interrupted due to a system crash, non-journaling file systems such as FFS, EXT2FS or Windows FAT can leave their file metadata in an inconsistent state. FFS versions with soft metadata updates avoid this consistency problem by carefully scheduling their disk updates, so that most of the file system check can be run in the background while the system boots up [McKusick, 2004]. The recovery process involves programs such as fsck or scandisk, and can take several hours with large file systems. Compared to this, recovery with a journaling file system is almost instantaneous: it can be as simple as replaying the "good" portion of the journal to the file system, and discarding the rest. While journaling file systems differ widely in the way they manage their information, conceptually they are very easy to understand. There are two major flavors: those that journal metadata only, and those that journal both data and metadata. We will look only at MACtimes, i.e. metadata, although journaled file content has great forensic potential, too. 23

24 Journaling saves repeated accesses 24

25 Reading Journaled MACtimes Check the filesizes of /var/log/cron and /var/log/sa/sa19. Locating the journal (linux example) Linux tune2fs shows that the journal is stored as a regular file with inode number 8 Does not show that the journal has a fixed size of 32 MBytes. One way to save the content of the journal is with TCT's icat command. Save it to a different file system, otherwise the journal may end up destroying itself with its own content. linux# icat /dev/hda1 8 >journalfile 25

26 Linux debugfs The Linux debugfs file system debugger can examine the file system journal. In order to examine a saved journal file, specify "-f journalfile" on the logdump command line. As with all tools that are used for unintended purposes, debugfs can produce unexpected results at times. Some judgment is needed when interpreting the results. The amount of MACtime history that can be recovered from a file system journal depends on the type and amount of activity in the file system, as well as file system implementation details. File systems such as Ext3fs that journal both data and metadata, the amount of recoverable MACtimes can be small. Systems with little activity can have records that go back by as much as an entire day or more. In such cases, watching a file system journal can be like watching a tree grow one ring at a time. The command below dumps recent access times for the /etc/ passwd file: 26

27 Foibles of time Hours, minutes, seconds Time zones Multiple implementations Implementation issues Accuracy Drift Synchronization The Network Time Protocol ([NTP, 2004]) and other time synchronization efforts are useful, but will not solve all problems. Systems that act as a central repository for logs often will get log messages from systems in other time zones - yet log the activity in the local time zone. When computers physically move to another time zone, clocks go bad, intruders attempt to inject false or spurious times into your logging mechanisms, systems lose power, backup clock batteries lose power, etc. 27

28 Conclusion No other form of forensic data is more interesting, frustrating, relied upon and untrustworthy than time. Provably accurate or consistent time can be extraordinarily difficult to obtain, and should generally only be relied upon when several events or points of view are correlated. Some forms of time data recovery and processing are difficult to automate and impractical for general use - the system will often only give its secrets out under duress or brute force. Additional research is needed on investigating, documenting, and providing methods to collect forensic timing data. Time data is vital to a forensic investigation. Time analysis provides the opportunity to detect modified or deleted records, great care and effort should be taken to try and uncover the gems of time scattered through the system. 28