International Journal of Computer Engineering and Applications, A RELIABLE AGNOSTIC DATA COMPRESSION MODEL BETWEEN EDGE AND CLOUD IOT PLATFORM

Transcription

1 A RELIABLE AGNOSTIC DATA COMPRESSION MODEL BETWEEN EDGE AND CLOUD IOT PLATFORM S.Balachandar Application Delivery Lead Shell India Market Private Limited- Bangalore Dr. R.Chinnaiyan Professor, Department of Master of Computer Applications New Horizon College of Engineering, Bangalore ABSTRACT: Due to rapidly adopting different IOT devices at edge layer which translates the data into various formats like JSON, GeoJSON, XML, SenML, etc.. Processing & real time transfer of data from edge to cloud is becoming a mammoth task. Many times data files contains a format which is not parsed by the big data layer in cloud. It is essential to keep a data compressor and real-time migration agent which transfer the data from edge to cloud seamlessly in the defined compression formats (e.g. Avro, Parqueet, etc..). This paper deals with agnostic data compression between edge and cloud IoT platforms. Keywords: IOT Devices, Data format, Edge, Big Data, Data compression, data migration [1] INTRODUCTION There has been wide-ranging research about how to unify and migrate the data between IOT edge layer to IOT Cloud layer with a quick turnaround time and it should be seamless in terms of loading and processing the data at IT cloud layer which has big data tools like Hadoop, Spark, Flink, etc. the S. Balachandar and Dr. R. Chinnaiyan 139

2 A RELIABLE AGNOSTIC DATA COMPRESSION MODEL BETWEEN EDGE AND CLOUD IOT PLATFORM objective this proposal is to simplify the data migration between edge to cloud with a common compression format that should work between edge and cloud Big data layer at any time. 1.1 IOT Edge This is the local layer where all the IOT devices will pour the data in different formats and it will be controlled and integrated with rules and actions. The underlying data set will reside in Big Data (light weight) and NOSQL layer. 1.2 IOT Cloud This is the central layer where all the IOT edge layers will send the data in different formats and data will be ingested in to data lake which curates and transforms the data into different buckets, hence the data scientist and analyst who can build models and dashboards based on different business demand. [2] PROBLEM STATEMENT The proposed research work will deliver a data compression framework for IOT Edge which can store data in Hadoop (e.g. ORC, PARQUEET, etc..) and synchronized between IOT Edge and IOT Cloud seamlessly. To design and deliver reliable compression framework within time and budget plays a vital role in IOT Edge platform design and development. This issue is addressed in this proposed research. 2.1 Literature Survey Data compression is performed by a code that uses a formula or algorithm to determine how to shrink the size of the data. For instance, an algorithm or code may represent a string of binary digits0s and 1s with a smaller string of 0s and 1s by using a dictionary for the conversion between them, or the formula may insert a reference or pointer to a string of 0s and 1s that the program has already seen.word or Text compression can be as simple as removing all unneeded characters, inserting a single repeat character to indicate a string of repeated characters and substituting a smaller bit string for a frequently occurring bit string.data compression can reduce a text file to 50% or a significantly higher percentage of its original size.for data transmission, compression can be performed on the data content or on the entire transmission unit, including header data, it may be transmitted in a ZIP, GZIP, LZO or another compressed format. Compressing data can be a lossless or lossy process. Lossless compression enables the restoration of a file to its original state, without the loss of a single bit of data, when the file is uncompressed. Lossless compression is the typical approach with executables, as well as text and spreadsheet files, where the loss of words or numbers would change the information.lossy compression permanently eliminates bits of data that are redundant, unimportant or imperceptible. Lossy compression is useful with audio, video and images, where the removal of some data bits has little or no discernible effect on the representation of the content. The following are native compression format supported by Hadoop framework S. Balachandar and Dr. R. Chinnaiyan 140

3 [3] PROPOSED APPROACH This proposed research uses the following methodologies for evaluating the compression & migration framework of IOT Edge to IOT Cloud a. SensCompr, a dynamic lossy compression method specific for sensor datasets and it is easily realizable with standard compression methods. b. Algorithmic optimization At the highest level, algorithmic optimization provides the greatest benefit. The data compression algorithms are implemented from scratch which process input data and to generate output format based on IOT cloud s needs c. Real time or Near real-time migration of compressed data between edge and cloud servers of Big Data. [4] DATA COMPRESSION AND MIGRATION The role of data compression is essential when data grows unknowingly at edge server. The data store at edge may not able to keep beyond certain data limits and it will be start purged automatically. As soon we receive the data we can compress it in edge and cloud agreed format and start forward to cloud in the same compressed format (e.g. Avro, ORC). The data transformation at edge will happen smoothly with large compressed data set. The similar data set of compressed format will be available in cloud store, the data lake at cloud environment will leverage even larger amount of data than before. S. Balachandar and Dr. R. Chinnaiyan 141

4 A RELIABLE AGNOSTIC DATA COMPRESSION MODEL BETWEEN EDGE AND CLOUD IOT PLATFORM The compressor solution will help us to fix up the format before we do migration. Migration is the continuous process between edge and cloud and it will be synchronized based on certain intervals or real-time. [5] CONCLUSION The compressed data format will still allow to query the way the current system does. It doesn t require any specific parsing. Data synchronization between edge and cloud IOT platform is seamless. The migration of compressed data between edge and cloud IOT will save significant effort of data ingestion of different formats and consolidation. It significantly reduces disk I/O for small reads and increase throughput through large compressed files. REFERENCES 1. N. Hung, H. Jeung, K. Aberer, "An evaluation of model-based approaches to sensor data compression", IEEE Trans. on Knowl. and Data Eng., pp , V. Misra, V. K Goyal, L. R. Varshney, "Distributed scalar quantization for computing: High-resolution analysis and extensions", IEEE Trans. Inf. Theory, vol. 57, pp , Aug S. E. Hawkins, E. H. Darlington, "Algorithm for compressing time-series data", NASA Tech Briefs, pp. 7-8, June A. Zanella, N. Bui, A. Castellani, L. Vangelista, M. Zorzi, Internet of things for smart cities. IEEE Internet Things J. 1 (1), (2014)CrossRefGoogle Scholar 5. P. Neirotti, A.D. Marco, A. Cagliano, G. Mangano, F. Scorrano, Current trends in smart city initiatives: some stylised facts. Cities 38, (2014) 6. M. Naphade, G. Banavar, C. Harrison, J. Paraszczak, R. Morris, Smarter cities and their innovation challenges. Computer 44 (6), (2011) 7. M. Rahimi, J. Ren, C. Liu, A. Vasilakos, N. Venkatasubramanian, Mobile cloud computing: a survey, state of art and future directions. Mobile Netw. Appl. 19 (2), (2013) S. Balachandar and Dr. R. Chinnaiyan 142

5 8. N. Mitton, S. Papavassiliou, A. Puliafito, K. Trivedi, Combining cloud and sensors in a smart city environment. EURASIP J. Wirel. Commun. Netw (1), 1 10 (2012) 9. M. Satyanarayanan, The emergence of edge computing. Computer 50 (1), (2017) 10. Openfog reference architecture for fog computing, produced by the openfog consortium architecture working group. [Online]. Available: T. Zhang, A. Chowdhery, V. Bahl, K. Jamieson, S. Banerjee, The design and implementation of a wireless video surveillance system, in Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (ACM, New York, 2015), pp K.-D. Lee, M.Y. Nam, K.-Y. Chung, Y.-H. Lee, U.-G. Kang, Context and profile based cascade classifier for efficient people detection and safety care system. Multimed. Tools Appl. 63 (1), (2013). 13. R.Chinnaiyan, Dr.S.Somasundaram, Reliability Assessment of component based software systems using Test Suite - A Review, Journal of Computer Applications, Vol 1, No.4, Oct Dec R.Chinnaiyan, Dr.S.Somasundaram, An Experimental Study on Reliability Estimation of GNU Compiler Components A Review, International Journal of Computer Applications ( ) Volume 25 No.3, July R.Chinnaiyan, Dr.S.Somasundaram, Reliability of Component Based Software With Similar Software Components - A Review, i-manager's Journal on Software Engineering; Nagercoil5.2 (Oct-Dec 2010). 16. R Chinnaiyan, S Somasundaram, Monte Carlo Simulation For Reliability Assessment of Component Based Software Systems, i-manager's Journal on Software Engineering, R Chinnaiyan, S Somasundaram, RELIABILITY ESTIMATION OF COMPONENT BASED SOFTWARE SYSTEMS THROUGH MARKOV PROCESS, International Journal of Mathematics, Computer Sciences and Information Technology. 18. R.Chinnaiyan, S.Somasundaram(2010) Evaluating the Reliability of Component Based Software Systems,International Journal of Quality and Reliability Management, Vol. 27, No. 1., pp R.Chinnaiyan, S.Somasundaram(2011), An Experimental Study on Reliability Estimation of GNU Compiler Components - A Review, International Journal of Computer Applications,Vol.25, No.3, July 2011, pp R.Chinnaiyan, S.Somasundaram(2009) Reliability of Object Oriented Software Systems using communication variables a review,international Journal of Software Engineering, Vol.2, No.2,PP S. Balachandar and Dr. R. Chinnaiyan 143