Log File Modification Detection and Location Using Fragile Watermark

Transcription

1 Log File Modification Detection and Location Using Fragile Watermark Liang Xu and Huiping Guo Department of Computer Science California State University at Los Angeles Los Angeles, CA, USA Abstract- In this paper, a novel algorithm is proposed to protect the integrity of log files. Unlike other existing schemes, the proposed algorithm can detect and locate any malicious modifications made to the log files. Furthermore, massive deletion of continuous data can be classified and identified. Security analysis shows that the algorithm can detect modifications with high probability which is verified by the experimental results. In real application, the proposed algorithm can be built into the generation procedure of the log files, so no extra process is needed to embed the watermark for the log files. Keywords: log file; modification detection; modification location; fragile watermarking 1. Introduction Nowadays logs are widely used to keep records of valuable information. Some examples are web server logs[1] and database logs[2]. Many log files contain records related to computer security, the concern of log file security has increased greatly because of the ever-increasing threats against systems and networks[3]. Lots of implementations have been created which place a greater emphasis on log file security. Most have been based on a proposed standard, RFC 3195, which was designed specifically to improve the security of syslog[4], a protocol presenting a spectrum of service options for provisioning an event-based logging service over a network. Among these implementations, transmission damage detection and message digest are aimed to protect the integrity of the log files. The transmission damage detection mainly focuses communication security. If log messages are damaged in transit, mechanisms built into the link layer as well as into the IP[5]and UDP[6] protocols will detect the damage and simply discard a damaged packet. When message digest method is used, log file integrity checking involves calculating a message digest for each file and storing the message digest securely to ensure that changes to archived logs are detected. The most commonly used message digest algorithms are MD5 and keyed Secure Hash Algorithm 1 (SHA-1). Although the transmission damage detection and the message digest method detect the modification of the log files, neither of them is able to locate the modifications. Once authentication is detected false, the whole message packet or the whole log file will be discarded. The usability of the log file is rather low once a modification is detected. The algorithm proposed in this paper not only detects any malicious modification but also locate the modifications. Except those located modification areas, the rest of the log file is still trustable. In the case of malicious modification, the usability of the log file is greatly improved. 2. Related Work In paper [7], Dr Guo and her colleagues proposed a novel algorithm of applying fragile watermarking in verifying the integrity of numerical and categorical data streams at the application layer. In their algorithm, watermarks are chained and embedded directly to the least significant digits of data groups to protect the authentication of data streams. Illegal modifications to the watermark embedded data streams can be effectively detected and located. In this paper, we extend Dr Guo s algorithm to the application of log file generation and storage. The algorithm is alternated accordingly to the specific application of log file generation. 3. Proposed Algorithms In our proposed algorithm, fragile watermarks are calculated and embedded on the fly of the generation of log files. No extra process is required for the watermark calculation and embedding. The embedded watermark can detect as well as locate any modifications made to the log file. 3.1 Algorithm assumptions and advantages In the proposed security log file scheme, we assume the log files are in human readable form composed of clear text. Different from numeric data streams which are widely used in multimedia data, most log files are text documents composed of character lines. We consider each separated text line as a single log data element.

2 We also assume a small buffer at the log file generator side which is used to store a group of log file data. As our scheme is group based, we collect the generated log entries in the same order as they are generated into the buffer until they form a complete group. Watermarks are calculated and embedded into the buffered log data, the watermarked data are then written to the log file storage. In this process, watermarks are calculated and embedded at the same stage as log file generation. No extra round of process is required. Our algorithm is designed to detect and locate modifications of log files using fragile watermark. It possesses the following advantages: Distortion free: The embedded watermark does not change the log file content. The information of the log file remains exactly the same meaning after the watermark embedding. Data is not distorted in any means. Invisibility: The embedded watermark does not degrade the perceptual quality of the log file. The watermark embedded log file remains in clear text format and conveys the exact same message as the log file generated without employing our scheme. Actually the marking is barely observable and the embedded watermark is invisible. Blind verification: The original unmarked log data are not - required for watermark verification. Location: The algorithm can detect as well as locate the modification to a log file. Any modifications, including changing, deletion or insertion of log entries, can be narrowed and located to specific groups. 3.2 Watermark embedding Table 1 gives the notations used in watermark embedding algorithm. The algorithm collects continuously the generated log data elements, i.e., log text lines, into a group and embeds the watermark. The watermark embedding mainly consists of two parts of processes, grouping and embedding. For each collected single log data line ei, we compute a secure hash hi. We check whether this data element i is a synchronization point based on its hash value hi. A log data element ei is defined as a synchronization point if and only if hi mod m = = 0 Table 1 Notations e i h i m L H(i) HASH WH W Watermark embedding notations Descriptions A single log data line secure hash of a single log data line Secure parameter Lower bound of the group size Group hash value Secure hash function Watermark hash value extracted watermark Note that throughout the whole algorithm, we trim the log line ei when we compute the secure hash hi. Since we embed the watermark bit at the end of each line as a space for watermark bit 1 and none for watermark bit 0. And whether the last character of a log line is a space will be used to calculate the watermark. The secure parameter m and the lower bound of the group size L decide how the log data elements are grouped. We only consider a group is formed and proceed to the embedding process if the log data element ei is a synchronization point and the number of elements in the group is larger than L. Otherwise, the log data element is buffered until a group is formed. The parameter L, the lower bound of the group size, is set to prevent small groups from forming for security reasons. HASH(H(i-2) H(i-1)) HASH(H(i-1) H(i)) W i-1 W i... Group i-2 Group i-1 Group i Group i+1 data Synchronization point Figure 1. Watermark embedding... As shown is figure 1, the watermark embedding is group based, all data between two synchronization points (providing the group size is larger than L), including the last synchronization point, form a group. A group hash value H(i) is computed on all the individual hash values hi of data elements within the group. Then a watermark W is constructed based on both the previous group hash value H(i-1) and the current group hash value H(i). We cut the length of the watermark (number of bits of the watermark) as the same as the number of data elements in the current group. The watermark is then embedded to the current group of data by inserting a space to the end of the trimmed log element line if the watermark bit is 1. In this way the embedded watermarks are actually chained. Even if a whole group of data is deleted, the deletion is still detectable because of the forward chain.

3 Algorithm 1 Watermark embedding 1: clear buff 2: fillbuff(buff) //See Algorithm 2 3: H0 = getgouphashe(buff) //See Algorithm 3 4: while true do 5: fillbuff(buff) 6: H1 = getgouphash(buff) 7: WatermarkEmbed(buff, H0, H1 ) //See Algorithm 4 8: Write data in buff into log file 9: Clear buff 10: H0 = H1 11: end while Algorithm 2 fillbuffer(buff) 1: k 0 2: while receive an incoming data element e i do 3: buff(k) = e i // collect and buff log element line e i 4: k++ 5: if (e i is a synchronization point) and (k >= L) then 6: return 7: end if 8: end while Algorithm 3 getgrouphash(buff,k) 1: k <-- number of log data elements in buff 2: for each data element e i in buff do 3: h i = HASH( e i ) //log data element line e i is trimmed. 4: end for 5: H = HASH2( h 1, h 2,..., h k) 6: return H Algorithm 4 watermarkembed(buff, H0, H1) 1: WH = HASH2(H 0, H 1 ) 2: k number of data element in buff 3: W = extractbits(wh) //See Algorithm 5 4: for i = 1 to k do 5: if w(i) == 1 then 6: Insert a space at the end of log data element line buff(i) 7: else if w(i) == 0 then 8: Do nothing. 9: end for Algorithm 5 extractbits(wh, k) 1: if length(wh) >= k then 2: W = concatenation of first k selected bits from WH 3: else 4: m = k - length(wh) 5: W = concatenation of WH and extractbits(wh, m) 6: end if 7: return W The above algorithms 1 5 illustrate how the watermark embedding process works. We will need one buffer in the process to collect and temporarily store a group of log data. As shown in algorithm 2, the buffer is filled until the log element line e i is a synchronization point and the current number of element lines in the buffer is not less than L, a group is then formed. Algorithm 3 calculates the hash value h i for each individual element line, and calculates the group hash value based on the hash value h i of all the individual elements in the group. Algorithm 5 extracts the same number of bits as the number of data elements in the group from a given watermark hash value WH, the extracted bits form the watermark to be embedded to the group data. Algorithm 4 computes the watermark hash value based on the group hash value of the previous data group, H 0, and the group hash value of the current group, H 1, watermark is extracted from the resulting watermark hash value and embedded to the current group of data. Algorithm 1 repeats the watermark embedding process infinitely. 3.3 Watermark verification As in the watermark embedding, watermark verification uses synchronization points to group log data elements into a group. A watermark which is constructed from the group hash value of the previous and the current group is checked against the extracted watermark from the current group. If the two watermarks match, no modification will be detected; the preliminary verification value of the current group is true. Table 2 Notations G pv1 pv0 V1 V0 V(-1) Watermark verification notations Descriptions A group Preliminary verification value of the current group G1 Preliminary verification value of the previous group G0 Final verification value of the current group G1 Final verification value of the previous group G0 Final verification value of the group before the previous group, i.e., G(-1) Table 2 gives the notations used in watermark verification algorithm. To verify the integrity of the log file, we need one buffer to collect and temporarily store a current group of data G1; we also need two pointers to point the beginning position and ending position of the previous group G0. The latter is used to locate the modifications made to the previous group. The watermark verification algorithms use the same initial group hash value H 0, secret parameter m, and the group size lower bound L as the watermark embedding algorithms. We also define two verification values here, the preliminary verification value and the final verification value. Notation pv0 indicates the preliminary verification value for the previous group and pv1 indicates the preliminary verification value for the current group; Similarly, V0 denotes the final verification value for the previous group and V1 denotes the final verification value for the current group. V(-1) denotes the final verification value we finalized at an earlier time for the group before the previous group. The preliminary verification value pv for a group, which results from the matching check between the watermark constructed and the watermark extracted, may be different from the final verification result V of that group. The final verification result for a group V indicates wheather the group is authentic. It is decided on the following watermark verification rational table.

4 Table 3 Watermark verification rationale Cases Precondition Rational Results Note pv0 pv1 V(-1) V0 V1 1 true true true true true No modification detected 2 true false true true false 3 false true false true true 4 false true true missing group(s) true true One or more groups may be missing before G0 5 false false true false? V1 to be finalized at next stage using next group G2 6 false false false false? V1 to be finalized at next stage using next group G2 Table 3 shows the rational of watermark verification. Since the watermark embedded in the current group G1 is a chained watermark computed from the value of the previous group G0 and the current group G1, if the watermark constructed from G0 and G1 matches the watermark extracted from G1, a positive preliminary verification result for the current group (pv1 = true) indicates the originality of the previous group data G0 and the current group data G1 (V0 = V1 = true). The situation is more complex if the two watermarks do not match. In this case, we can only say the preliminary verification for the current group is false (pv1 = false). We will still need to investigate the integrity of the next group before ascertaining the final verification result of the current group (forward check). Similarly, the final verification result for the previous group (V0) depends not only on the preliminary value of its own (pv0) but also on the preliminary verification result of the current group (pv1). If the preliminary verification for the current group is positive (pv1 = true), which also indicates a positive verification for the previous group (V0 = true), the false preliminary verification pv0 means either one or more group are missing between the G(-1) and G0, or the false verification result is brought forward from the false verification of G(-1). There are some interesting cases where preliminary verification is false. This may be due to modifications in the previous group or those in the current group, or missing groups. In Table 3, one or more groups are missing before G0 in the case 4. Since in this case the verifications of both the previous group V0 and the group before previous group V(-1) are true, which excludes the possibility of modifications in both G(-1) and G0, the only explanation for the false pv0 is that one or more entire groups between G(- 1) and G0 have been deleted. It is these deletions that cause the preliminary verification of the previous group pv0 to be false. Because watermarks are chained together, no matter how many groups are deleted, the deletions can be correctly detected. Algorithm 6 Watermark verification 1: clear buff 2: pv0,v0, V(-1) true 3: fillbuff(buff) //See Algorithm 2 4: H0 = getgouphash(buff) //See Algorithm 3 5: while true do 6: fillbuff(buff) 7: H1 = getgouphash(buff) //See Algorithm 3 8: WatermarkVerify(buff, H0, H1, pv0, V0, V(-1)) // See Algorithm 7 9: Clear buff 10: H0 H1 11: pv0 pv1 12: V(-1) V0 13: V0 V1 14:end while Algorithm 7 watermarkverify(buff, H0, H1,pV0,V0, V(-1)) 1: WH = HASH(H0,H1) 2: k number of data elements in buff 3: W1 = extractbits(wh, k) //See Algorithm 5 4: for i = 1 to k do 5: if end of log element line buff(i) is a space 6: W2(i) 1 7: else 8: W2(i) 0 9: end for 10: IF (W1 == W2) then 11: V1 = pv1 = V0 = true 12: else 13: pv1 = false 14: V1 = V0 & pv1 15: end if 16: if V0 == false then 17: The previous group may be tampered 18: end if 19: if pv0 == false && V0 == true && V(-1) == true then 20: One or more groups between the previous group and the current group may be missing 21: end if The algorithms 6 and 7 describe the watermark verification process. A group of log data lines are filled into the buffer, the group hash value H1 is computed. Algorithm 7 verifies the watermark extracted from a group of log data through watermark matching. Watermark hash value WH is computed based on the current group hash value H1 and the previous group hash value H0. (For the first group of data, the initial watermark hash value WH0 is introduced.) Watermark W1 is constructed from WH. Meanwhile, watermark W2 is extracted from the group of data lines. If W1 equals to W2, the watermark verification for the current group succeeds and the verification result is set to true; otherwise, the verification fails and a false preliminary verification result is set to the current group. Algorithm 6 repeats the above process for an infinite log file check.

5 4. Implementations To directly create the watermarked security log file, we integrate our watermark embedding mechanism with the logging service. Figure 2 displays the integration structure of integrating watermark embedding mechanism into the log4j logging. Based on this structure, the printing methods of a logger instance in log4j will output the log data to a socket. The watermark embedding mechanism collects the log data element lines from the socket and invokes the watermark embedding method. Thus the watermark embedding process is triggered and the watermarked log data are produced. The watermarked log data stream will finally be written to the predefined destination through the appending method of an appender and the write method of the output stream in log4j. Log4j Logger.printing() Output to socket Appender.appending() Outputstream.write() SOCKET Watermark Embedding Collect stream lines from socket watermarkembedding() Figure 2. Integration with log4j 5. Experimental Results and Conclusion 5.1 Experiment scenarios and results We test the validity of the mechanisms under various scenarios. We choose log files of different types and different sizes and embed watermarks into the log files using the watermark embedding mechanism. Then we modify the watermarked log file in various ways manually or using the log file modification program. The watermark verification mechanism is applied to the manipulated log files to detect and locate the modifications made to the log files Successful verification without modification First, we verify the original watermarked log file without any modification. No modification is detected in this scenario. The verification result turns out to be successful Single data element modification We change, delete or insert a single data element line in the watermarked log file and then run the verification process. The results show that if a regular data line is modified, the modification is easily detected and located to the modified group. If the modified line happens to be a synchronization point (this may happen at a less possibility rate), the grouping of the data is affected, the detection will locate the modification to the group before the synchronization point (including the synchronization point) and the group after the synchronization point Multiple data elements modification We change, delete or insert multiple data element lines in the watermark embedded log file and then run the verification process. The same as single data element modification, if regular data lines (not synchronization points) are modified, as long as the grouping of data is not affected, the modifications are easily detected and located to the modified groups. The detection becomes more complex if the grouping of data is changed because of the modification or change of synchronization points. If the modification is related to synchronization points, the detection mechanism locates the modification to the group before the modified synchronization point (including the synchronization point) and the group after the synchronization point. The experimental results also show that the mechanism is able to detect and locate the modifications when one or more entire groups are deleted or inserted although these cases rarely happen in real world. If the inserted two or more groups are continuous groups with well-embedded watermarks (this can be made by repeating continuous groups of data), the classification for the insertion fails which is mistakenly classified as missing groups. But the mechanism is still able to locate the modification before the inserted group and before the group after. 5.2 Security analysis There is a probability that the log file watermark mechanism fails to detect a modification made to the log file. In the case that the extracted watermark from the modified group happens to match the watermark constructed from the modified group and the group before, the preliminary verification of the modified group will succeed, the modification will not be detected. Assume the size of a group is l, after a modification is made to the group, the probability that the preliminary verification of the group will succeed (i.e., the false negative rate) is as 1/2 l <= 1/2 L L is defined as the lower bound of group size. The group false negative rate monotonically decreases with the value of L. With group size fixed, the false negative rate of any affected group in an attack remains the same no matter how many

6 elements in the group are changed. Therefore, we consider the attack at group level and assume that g groups are affected in attacks. The overall false negative rate, which is the probability that at least one affected group is verified successfully, can be computed as 1- (1-1/2 L ) g The overall false negative rate is monotonic increasing with g and decreasing with L. Providing a relative large L and small g, which is as in most cases in the real world, we can limit the overall false negative rate to a rather small rate. During the experiment, we noticed that the lower bound value of group size (i.e., L) has an impact on the number of affected groups (i.e., g) in simulated attacks. With a smaller L value, there are generally fewer data lines in an average group, the modification of single element and especially multiple data elements is more likely to affect the grouping of data. Thus, with a smaller size for each group, the modification location accuracy for a group increases while more groups may be affected and be located in overall modification locations. 5.3 Performance analysis Table 4 Watermark embedding time Log file size Generation time without watermarking Generation time with watermarking Time difference (watermarking time) 1 KB 2 ms 2 ms KB 5 ms 6 ms 1 ms 12.1 KB 9 ms 11 ms 2 ms 133 KB 43 m 48 ms 5 ms 367 KB 67 ms 74 ms 7 ms 463 KB 74 ms 84 ms 10 ms 1,285 KB 160ms 177 ms 17 ms 6,545 KB 400 ms 450 ms 50 ms 10,241 KB 955 ms 1105 ms 150 ms 27,179 KB 1510 ms 1676 ms 166 ms that the watermarking process doesn t cost much extra time compared to the original log generation. It costs about one tenth more time to generate a watermarked log file than to produce the original log file, which proves the feasibility of the proposed scheme. 5.4 Conclusion This paper gives a naive way to protect the integrity of the log files. According to our scheme, modifications, including change, insertion and deletion, made to a log file can be detected and located to a small group area. The algorithm is described in details. Implementations and experimental results are discussed. We could embed the scheme into the generation procedure of the log files thus no extra process is needed to embed the watermark for the log files. The algorithm can be applied to any text files. For any text files, fragile watermarks can be calculated and embedded, thus any modifications made to the text files can be detected and located to small areas using the watermark verification mechanism described in this paper. 6. References [1] M. Bruce, (1999, Jul.). A Brief Introduction to Server Logs. [Online]. Available: [2] Wikimedia Foundation, Inc., (2011, Feb.). Transaction log. [Online]. Available: [3] K. Kent, M. Souppaya, "Guide to Computer Security Log Management", National Institute of Standards and Technology, Technology Adminstration, U.S. Department of Commerce, Special Publication [4] C. Lonvick, "The BSD Syslog Protocol", RFC 3164, August [5] J. Postel, "Internet Protocol", STD 5, RFC 791, September [6] F. Baker, "Requirements for IP Version 4 Routers", RFC 1812, June With watermarking Without watermarking [7] H. Guo, Y. Li, A. Liu and S. Jajodia, A fragile watermarking scheme for detecting malicious modifications of database relations, Information Sciences, vol. 176, pp , May 2006 Figure 3. Watermark embedding time Table 4 gives approximate time to watermark log files in different sizes using our watermark embedding mechanism. Figure 3 illustrates the time increase chart for the log generation without watermarking and with watermarking as the log file size increases. Both the table and the figure show