Comparison of Existing Dictionary Based Data Compression Methods for English and Gujarati Text # MCA,SRIMCA

Transcription

1 Comparison of Existing Dictionary Based Data Compression Methods for English and Gujarati Sandip Maniya #1, Jikitsha Sheth *2, Dr. Kalpesh Lad *3 # MCA,SRIMCA Uka Tarsadia University, Maliba Campus, Dist. Surat, GUJ, INDIA 1 10mca015@srimca.edu.in * SRIMCA Uka Tarsadia University, Maliba Campus, Dist. Surat, GUJ, INDIA 2 jikitsha.sheth@srimca.edu.in 3 kalpesh.lad@srimca.edu.in Abstract Data compression is a common requirement for most of the computerized applications. There are number of data compression algorithms, which are dedicated to compress different data formats. Even for a single data type there are number of different compression algorithms, which uses different approaches. This paper examines dictionary based data compression algorithms and compares their performance. The performance of selected s is examined with respect to compressing the English as well as Gujarati text data. Keywords: Data compression, Lossless Compression, Lossy Compression Introduction Data compression is the process of encoding an input data stream into another data stream that has a smaller size [1]. The main objective of data compression is to reduce the physical capacity of the data. It is important in the computing process because it helps to reduce the space occupied by a, which normally leads to the reduction in the time taken to access the [4]. s which may be compressed include text, images, video/speech and sound. It applied at several areas of IT including neural networks, bioinformatics, database management systems, wireless systems, error detection and correction codes, fractals etc [4]. Data compression technique can be broadly divided into two major families: Lossless compression and Lossy compression [2]. Lossless compression techniques are used to compress medical images, text and images preserved for legal reasons, computer executable and so on [10]. Lossy compression techniques reconstruct the original message with loss of some information. It is not possible to reconstruct the original message using the decoding process, and is called irreversible compression [11]. Most data compression s consist of one or a combination of data transform, quantization, and coding [8]. There are many known s for data compression. They are based on different ideas, are suitable for different types of data, and produce different results, but they are all based on the same principle, namely they compress data by removing redundancy from the original data in the source chiefeditor@ijrcct.org Page 294

2 [2]. Authors focus on dictionary based compression s for Gujarati text compression. There are billions of internet users in Asia; but there is very less content in local languages to support them, Just 13.83% of all content on the Internet was in Asian languages [13]. And from that only 0.03% web contents accounted other than Chinese, Japanese and Korean Asian languages including Gujarati [13]. According to data, Gujarati text is very less available in electronic form, many compression techniques have been proposed for English text [3]. Data Compression Methods Dictionary based compression s do not use a statistical model, nor do they use variable-size codes. Instead they select strings of symbols and encode each string as a token using a dictionary [2][10]. The dictionary holds strings of symbols, and it may be static or dynamic. The former is permanent, sometimes allowing the addition of strings but no deletions, whereas the latter holds strings previously found in the input stream, allowing for additions and deletions of strings as new input is being read. Authors studied BZip2,, LZMA, PPMd, RAR, ZIP and Coding data compression s which are given below Zip Archive 7-Zip is a free and open source archive. 7-Zip operates with the 7z archive format, but can read and write several other archive formats. 7-Zip began in 1999 and is actively developed by Igor Pavlov. 1. BZip2 [14]: It is a free and open source compressor that uses the Burrows Wheeler algorithm. It is developed and maintained by Julian Seward. It compresses most s more effectively than the older LZW (.Z) and Deflate (.zip and.gz) compression algorithms, but is considerably slower. LZMA is generally more space-efficient than bzip2 at the expense of slower compression speed, while having much faster decompression. 2. [15]: It is a simple container format that can include both uncompressed data and LZMA data, possibly with multiple different LZMA encoding parameters. supports arbitrarily scalable multithreaded compression and decompression and efficient compression of data which is partially incompressible. 3. LZMA [16]: The Lempel Ziv Markov chain algorithm (LZMA) is an algorithm used to perform lossless data compression. It has been under development since 1998 and was first used in the 7z format of the 7-Zip archiver. This algorithm uses a dictionary compression scheme somewhat similar to LZ77 and features a high compression ratio and a variable compression-dictionary size while still maintaining decompression speed similar to other commonly used compression algorithms. 4. PPMd [17]: Predictions are usually reduced to symbol rankings. The number of previous symbols, n, determines the order of the PPM model which is denoted as PPM (n). Unbounded variants where the context has no length limitations also exist and are denoted as PPM. If no prediction can be made based on all n context symbols a prediction is attempted with n 1 symbols. This process is repeated until a match is found or no more symbols remain in context. At that point a fixed prediction is made. 2. Archive chiefeditor@ijrcct.org Page 295

3 is a shareware archiver and data compression utility. It is one of the few applications that are able to create RAR archives natively, because the encoding is held to be proprietary. It supports multithreaded compression, ability to create self-extracting and multi-volume archives, and NTFS system and Unicode in names. It support for RAR and ZIP archives, and unpacking of ARJ, LZH, TAR, GZ, ACE, UUE, BZ2, JAR, ISO, EXE, 7z, and Z archives[19]. 1. RAR [18]: RAR stands for Roshal Archive. It is a proprietary archive format that supports data compression, error recovery, and spanning. It was developed by a russian software engineer, Eugene Roshal, and is currently licensed by win.rar GmbH. The name extension used by RAR is.rar for the data volume set and.rev for the recovery volume set. The RAR compression utility is proprietary, with a closed algorithm. 2. ZIP [18]: Zip is a format used for data compression and archiving. A zip contains one or more s that have been compressed, to reduce size, or stored as is. The zip format permits a number of compression algorithms. The format was originally created in 1989 by Phil Katz, and was first implemented in PKWARE's PKZIP utility, as a replacement for the previous ARC compression format by Thom Henderson. The zip format is now supported by many software utilities other than PKZIP. Microsoft has included built-in zip support in versions of Microsoft Windows since Apple has included built-in zip support in Mac OS X 10.3 and later. Zip s generally use the extensions ".zip" or ".ZIP" and the MIME media type application/zip. Zip is used as a base format by many programs, usually under a different name. 3. Coding Method [6][12] The code algorithm creates variable length code that is integral number of bits. The code have unique prefix codeword for symbol base on symbol probability distribution pi, where I = 1, 2, 3... N; the frequency distribution of all the symbols of the source is calculated in order to calculate the probability distribution. According to the probabilities, the codeword for each symbol are assigned. It assigns shorter codeword for higher probability symbols and longer codeword for smaller probability symbols. The codes are built from the bottom up, starting with the leaves of the tree and working progressively closer to root. Building decoding tree is done using completely different code is laid out as string of leaf nodes that are going to be connected by a binary tree. Each node has weight, which is simply occurrence or probability of symbol s appearance. Experiment and Result Authors applied the above discussed s on different Gujarati as well as English text. One of the factors that was kept under observation was size. After compression, the size were observed reduced as follows: chiefeditor@ijrcct.org Page 296

4 BZip2 LZMA PPMd Test1.txt Test2.txt Test3.txt Test4.txt Test5.txt Test6.txt Test7.txt Test8.txt Test9.txt Table 1.1 English Compressed Size The compression ratio based on the above table is as follows: BZip2 LZMA PPMd Test1.txt % 74.51% 74.51% 77.59% 75% 74.35% 38.79% Test2.txt % 78.26% 78.26% 81.02% 78.18% 77.79% 40.47% Test3.txt % 80.27% 80.27% 79.81% 79.08% 78.25% 40.91% Test4.txt % 94.99% 94.99% 90.15% 94.62% 80.80% 36.88% Test5.txt % 97.68% 97.68% 91.86% 97.48% 80.73% 36.70% Test6.txt % 98.89% 98.89% 92.65% 98.46% 80.72% 36.72% Test7.txt % 99.46% 99.46% 93.07% 99.36% 80.71% 36.66% Test8.txt % 99.75% 99.75% 93.29% 99.67% 80.70% 36.67% Test9.txt % 99.80% 99.80% 93.34% 99.72% 84.91% 36.71% Average Compression 85.89% 91.51% 91.51% 88.08% 91.28% 79.88% 37.83% Table 1.2 English Compressed Size It was observed that the average compression ratio of coding for English content text was poor compared to 7z variants, and. It was also observed that as the size of such increased, the comparative ratio percentage degraded. For example, for test1.txt, the 7z variants gave near to 75% while Page 297

5 encoding resulted to only half of that percentage (i.e near to 38%). For test9.txt the 7z variants gave near to 95% while encoding resulted to much less that half of that percentage(i.e near to 36%). As the size of increased, the performance of performed poor than rest the s under observation. In the following table, the size after compression of Gujarati text is described: BZip2 LZMA PPMd Test1.txt Test2.txt Test3.txt Test4.txt Test5.txt Test6.txt Test7.txt Test8.txt Test9.txt Table 1.3 Gujarati Compressed Size The compression ratio based on the above table is as follows: BZip2 LZMA PPMd Test1.txt % 82.82% 82.82% 83.20% 82.51% 81.52% 51.29% Test2.txt % 82.11% 82.11% 84% 81.52% 80.82% 52.07% Test3.txt % 91.27% 91.27% 86.57% 90.53% 81.07% 53.02% Test4.txt % 99.06% 99.06% 88.48% 98.84% 81.54% 53.46% Test5.txt % 99.51% 99.51% 88.58% 99.35% 81.56% 53.41% Test6.txt % 99.74% 99.74% 88.65% 99.61% 81.56% 53.42% Test7.txt % 99.86% 99.86% 88.68% 99.75% 81.56% 53.42% Test8.txt % 99.93% 99.93% 88.71% 99.84% 81.58% 53.42% Test9.txt % 99.94% 99.94% 88.71% 99.84% 81.56% 53.42% Average Compression 93.78% 94.91% 94.91% 87.28% 94.64% 81.41% 52.99% Table 1.4 Gujarati Compressed Size chiefeditor@ijrcct.org Page 298

6 It was observed that the average compression ratio of coding for Gujarati content text was poor compared to 7z variants, and. It was also observed that as the size of such increased, the comparative ratio percentage degraded. For example, for test1.txt, the 7z variants gave compression near to 83% while encoding resulted to approximately only 51%. Gradually as size increased, it was observed that even for test9.txt where the 7z variants gave near to 95% compression, encoding resulted to 53% only. As the size of increased, the performance of performed poor than rest the s under observation. Unlike observed in case of English text compression, the compression ratio for Gujarati text remains stable, even if high changes are made to size factor. Conclusion In this paper authors studied and compared dictionary based compression s on English as well as Gujarati text. Dictionary based compression techniques provide good as well as lossless compression. encoding performs poor compared to 7z variants, and. outperforms other s as the size grows. For English text, the variation in compression ratio is observed as size changes, but such is not the case for compression with respect to Gujarati text. References: 1. Nirali Thakkar, Malay Bhatt, "Cascading of the PDLZW Compression Algorithm with Arithmetic Coding", International Journal of Computer Applications ( ), Volume 46 No.16, May David Salomon, "Data Compression-The Complete Reference", Springer-Verlag London Limited, S Sankar, Dr. S Nagarajan, "A Comparative Study: Data Compression on TANGLISH Natural Language ", International Journal of Computer Applications ( ), Volume 38 No.3,PP 33-37, January Dele Oluwade, "A Comparative Analysis and Application of the Compression Properties of Two 7-Bit Subsets of Unicode", Journal of Emerging Trends in Computing and Information Sciences, ISSN , VOL. 3, NO. 4, pp , April Ruey-Cheng Chen, Chiung-Min Tsai and Jieh Hsiang, "A Regularized Compression Method To Unsupervised Word Segmentation", Proceedings of the Twelfth Meeting of the Special Interest Group on Computational Morphology and Phonology, pages Mohd. Faisal Muqtida and Raju Singh Kushwaha, "Improvement in Compression Efficiency of Coding", International Journal of Computer Applications ( ), Volume 45 No.24,pp , May Sebastian Deorowicz, "Universal lossless data compression algorithms", Ph.D. Thesis, Silesian University of Technology Faculty of Automatic Control, Gliwice, Matthew J. Watson, Antonios Liakopoulos, Dragana Brzakovic, and Christos Georgakis,"A Practical Assessment of Process Data Compression Techniques", Ind. Eng. Chem. Res. 1998, 37, Kodituwakku and Amarasinghe, "Comparison of Lossless Data Compression Algorithms for Data", Indian Journal of Computer Science and Engineering, Vol 1 No Blelloch, E., Introduction to Data Compression, Computer Science Department, Carnegie Mellon University, Kesheng, Otoo and Arie, "Optimizing bitmap indices with efficient compression", ACM Trans. Database Systems, pp. 1-38, Basel Bani-Ismail, "Data Compression Techniques", College of Applied Studies, University of Bahrain. 13. Dmitry Shkarin, "PPM:one step to practicality", institute of Dynamics of Geoshperes, Moscow, Russia. 14. Boris Ryabko, "Compression-based s for density estimation for time series", Institute of Computational Technologies of Siberian Branch of Russian Academy of Science and Siberian State University of Telecommunications and Informatics, Novosibirsk, Russia chiefeditor@ijrcct.org Page 299