DATA DEDUPLCATION AND MIGRATION USING LOAD REBALANCING APPROACH IN HDFS Pritee Patil 1, Nitin Pise 2,Sarika Bobde 3 1

Transcription

1 DATA DEDUPLCATION AND MIGRATION USING LOAD REBALANCING APPROACH IN HDFS Pritee Patil 1, Nitin Pise 2,Sarika Bobde 3 1 Department of Computer Engineering 2 Department of Computer Engineering Maharashtra Institute of Technology 3 Department of Computer Engineering Maharashtra Institute of Technology Pune, India ABSTRACT: Protection of user s is the main challenge for big data in security point. Big data contains huge amounts of personal identifiable information and therefore privacy of users is a huge concern. However, encrypted data introduce new challenges for cloud data deduplication, which becomes crucial for big data storage and processing in cloud. Traditional deduplication schemes cannot work on encrypted data. Existing solutions of data deduplication suffer from security weakness. They cannot support access control and revocation mechanism. In this paper, we propose a scheme for data deduplication which is used to reduce space, help in storage utilization. It also support migration concept with load rebalancing technique. The proposed system integrates data deduplication with access control and revocation. The results show the superior efficiency and effectiveness of the scheme for potential practical deployment, especially for big data deduplication in HDFS framework. Keywords: Access Control, Big data, Data-duplication, HDFS [1] INTRODUCTION Our aim is to minimize repetitive information and augment space funds. A strategy which has been generally embraced is cross-client deduplication. The basic thought behind deduplication is to store copy information (either documents or pieces) just once. Accordingly, if a client needs to transfer a record (piece) which is now put away, the cloud supplier will add the client to the proprietor rundown of that document (square). In existing surveys we find some issues mentioned as below - 1) Data redundancy issue generated in cloud storage. 2) Some systems can be take very long time for data processing 3) No data security mechanism yet develop in any system. After finding all these issues in existing approaches we develop a system which can work with parallel processing finding a duplication issue. Data de-duplication is a special data compression technique which eliminates duplicate copies of repeating data in storage. The technique is preferable to improve storage utilization and also applied for network data transfers to reduce the number of bytes that must be sent. Instead of keeping multiple data copies with the same content, de-duplication eliminates redundant data by keeping only one physical copy and referring other redundant data to that Sonali Gupta and Komal Kumar Bhatia 1

2 DATA DEDUPLCATION AND MIGRATION USING LOAD REBALANCING APPROACH IN HDFS copy. De-duplication can take place at either the file level or at the block level. For file level deduplication, it eliminates duplicate copies of the same file. For block level, it eliminates duplicate blocks of data that occur in non-identical files. The simple idea on the side of deduplication is to store duplicate data (either files or blocks) only once. Therefore, if a user wants to upload a file (block) which is already stored, the HDFS framework will add the user to the owner list of that file (block). A Hadoop frame-worked application works in an environment that provides distributed storage and computation across clusters of computers. Hadoop is designed to scale up from single server to thousands of machines, each offering local computation and storage. The Hadoop Distributed File System (HDFS) is based on the Google File System (GFS) and provides a distributed file system that is designed to run on large clusters (thousands of computers) of small computer machines in a reliable, fault-tolerant manner. HDFS uses a master/slave architecture where master consists of a single Name Node that manages the file system metadata and one or more slave Data Nodes that store the actual data. [2] CHALLENGES IN EXISTING SYSTEM 1) In most of the existing data de-duplication system, the private cloud is involved as a proxy to allow data owner/users to securely perform duplicate check with differential privileges. 2) In existing system apply de-duplication on only without encrypted data. 3) The data owners only outsource their data storage by utilizing public cloud while the data operation is managed 4) In existing system migration concept is not introduced. 2.1 Advantages of Proposed System The proposed system can flexibly support access control on encrypted data with deduplication. The proposed system can have Low Cost of Storage. The proposed system can efficiently perform big data deduplication with load rebalancing technique. [3] SYSTEM ARCHITECTURE In the proposed research work we design and implement a system which will provide the parallel processing to detect the data de-duplication problem in big data environment. The system also provides benefit access control of data management and proxy revocation of system. It also support migration concept. Sonali Gupta and Komal Kumar Bhatia 2

3 Figure 1: System architecture of proposed System There are basically two types of data de-duplication as: i) File-level de-duplication: This de-duplication technique is commonly called as single instance storage; file-level data deduplication compares a file that has to be archived or backup that has already been stored by checking all its attributes against the index. ii) Block-level de-duplication: Block-level data de-duplication operates on the basis of sub-file level. As the name implies, that the file is being broken into segments blocks or chunks that will be examined for previously stored information vs redundancy. The proposed system accepts input as file, uploads the data in an encrypted form after that it checks the data deduplication and performs the operations such as modification, deletion and download of data. System also handles data owner management. [3.1] DATA OWNER First data owner can upload the text file from at the same system can take all files from data nodes and check the duplication with given file. If the VCS score is greater than threshold system can denote this file as duplicate. If the file is not duplicate then job manager first check the each server load and find the trustworthy of them base on CPU as well as memory load. Then the encrypted data can distribute into hdfs, and store the file tokens and other details into hash table. [3.2] USER GROUP In case that an authentic data owner uploads the data later than the data holder, the job manager can manage to preserve the data encrypted by the authentic data owner at the HDFS. But at the same time manager can use the hash table efficient data retrieval, that can be reduce the time and cost also. Sonali Gupta and Komal Kumar Bhatia 3

4 DATA DEDUPLCATION AND MIGRATION USING LOAD REBALANCING APPROACH IN HDFS [3.2] HDFS FRAMEWORK Hadoop Distributed File System maintains consistency of data distributed across the large number of data nodes. Large files are distributed across the cluster and managed via a metadata server known as the primary name node. HDFS maintains a number of high availability features including replication and rebalancing of data across nodes. It executes map-reduce and checks the similarity score with the help of VCS algorithm. In this module system will process all the execution in parallel, we used one hash table at the time of data insertion once data has insert into database it will make a history into the hash table. For the efficient retrieval we can use the hash table. [4] ALGORITHM The algorithms used in deduplication technique and the implementation process details are described below. [4.1] Equally Load Re-Balancer (migration) Algorithm Input : Each Node load base on current hitting Step 1: Initialize all data nodes which are connected to master node as n. Step 2: for each (I up to n ) Take each ith node server load. A[i] => ith Node load degree or hitting load. End for. Step 3: get total length of A. create the data requested chunks. K=A. length (); Step 4: Generate k mappers for distribute a data. Step 5: assign each chunks to each mapper. Step 6: Request to server for save a data. Step 7: end procedure. Output: Distributed data to each node. [4.2] Vector base cosine similarity (VCS) Input: Query Q, Threshold t Here the system have to find the similarity of two vectors: and, where and are the components of the vector (features of the document, or values for each word of the comment ) and the is the dimension of the vectors: Step 1: Read each row R from dataset D Step 2: for each (Column c from R) Step 3: score= Formula1(R,Q) If(score > t) Break; Sonali Gupta and Komal Kumar Bhatia 4

5 Early stop; Else step 2 continue End for Output: duplicate if returns 1 else unique [5] RESULT AND DISCUSSION The final results of the designed system are given below. From those results we get the detailed information to Check de-duplication and upload the files, checking for Duplication, file uploading, file downloading. Detailed procedure of the proposed system is given. For the results and comparative analysis we compare the system with some existing approaches like cloud base deduplication, KNN base duplication, the below graphs show the time required for retrieve the data with propose as well as existing. The below graph fig. 2 show the proposed system comparison with others. Here X axis shows the different techniques and Y shows the time required for execution in seconds with specific system. On the basis of fig.2 system can finally conclude the proposed system efficient than others System. Figure 2: Proposed system comparison with others Result analysis is carried out in following ways: System allows User Login to authenticated users only. System checks validation of different inputting fields of register users such as username and field etc. System allows uploading file in text format with corresponding server. System can provide security to data with encryption and decryption. [6] CONCLUSION In this paper we are introducing a data duplication scheme with HDFS framework which can provide parallel processing with minimum time complexity. The proposed scheme can flexibly support data update and sharing with deduplication even when the data holders are offline. Encrypted data can be securely accessed because only authorized data holders can obtain the symmetric keys used for data decryption. The proposed system also supports access control and revocation mechanism. It also support migration concept. Futurework includes slot configuration and jobordering for fast retrival and optimizing our design. Sonali Gupta and Komal Kumar Bhatia 5

6 DATA DEDUPLCATION AND MIGRATION USING LOAD REBALANCING APPROACH IN HDFS REFERENCES [1] Zheng Yan, Wenxiu Ding, Xixun Yu, Deduplication on Encrypted Big Data in Cloud, IEEE Transaction on Big Data, Vol. 2, No. 2, April-June 2016 [2] Jin Li, Yan Kit Li, Xiaofeng Chen, Patrick P. C. Lee and Wenjing Lou, A Hybrid Cloud Approach for Secure Authorized Deduplication, IEEE Transaction On Parallel And Distributed System,Vol.PP,No.99, [3] Zheng Yan, Wenxiu Ding, Xixun Yu, Deduplication on Encrypted Big Data in Cloud, IEEE Transaction on Big Data, Vol. 2, No. 2, April-June [4] Maneesha Sharma, Himani Bansal and Amit Kumar Sharma, Cloud Computing: Different Approach & Security Challenge, IJSCE, Volume-2, Issue-1, March [5] Kangchan Lee, Security Threats in Cloud Computing Environments, International Journal of Security and Its Applications, Vol. 6, No. 4, October, [6] Sashank Dara, Cryptography Challenges for Computational Privacy in Public Clouds, International Journal of Security and Its Applications, Volume 4, [7] Zhaocong Wen, Jinman Luo, Huajun Chen, Jiaxiao Meng, A Verifiable Data Deduplication Scheme in Cloud Computing, International Conference on Intelligent Networking and Collaborative Systems [8] Pasquale Puzio, Refik Molva, Melek Onen, Sergio Loureiro, ClouDedup: Secure Deduplication with Encrypted Data for Cloud Storage, IEEE 5 th International Conference on Cloud Computing Technology and Science [9] Sun, Z., Shen, J. & Yong, J, A novel approach to data de-duplication over the engineeringoriented cloud systems, Integrated Computer Aided Engineering, 20 (1), 45-57, [10] David Pointcheval, Asymmetric Cryptography and Practical Security, International Journal of Security and Its Applications, Volume 4, [11] Yogesh Kumar, Rajiv Munjal and Harsh Sharma, Comparison of Symmetric and Asymmetric Cryptography with Existing Vulnerabilities and Counter-measures, International Journal of Computer Science and Management Studies, Vol. 11, Issue 03, Oct Sonali Gupta and Komal Kumar Bhatia 6

7 [12] Mihir Bellare, Sriram Keelveedhi, Thomas Ristenpart, DupLESS: Server-Aided Encryption for De-duplicated Storage, IEEE Transactions on Information Forensics and Security, Vol.11, [13] Zhaocong Wen, Jinman Luo, Huajun Chen, Jiaxiao Meng, A Verifiable Data Deduplication Scheme in Cloud Computing, International Conference on Intelligent Networking and Collaborative Systems Sonali Gupta and Komal Kumar Bhatia 7