Transaction Analysis using Big-Data Analytics

Transcription

1 Volume 120 No , ISSN: (on-line version) url: Transaction Analysis using Big-Data Analytics Rajashree. B. Karagi 1, R. H. Goudar 2, 1,2 Dept of Computer Networking Engineering 1,2 Center for PG studies VTU, Belagavi. August 14, 2018 Abstract Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using onhand database system tools. Due to some properties of Bigdata, it is hard to analyze the data, thus Big-data analytics can be the alternate tool. In this paper, we are using Bigdata analytic tool as Hive. It is simple to write a query and easy to understand, the queries same as SQL. Why not SQL because it is a row-level data searching and it is used when the database is relatively small, does not analyze the complex data. For these reasons Hive tool used and it is helpful for storing wide range amount of data as well as process complex datasets. Analyzing data help the business managers make well- informed decisions to handle the company forward, better efficiency, raise the profits and achieve organizational goals. Key Words:Big-data; Big-data Analytics; Apache Hadoop; Apache Hive; Data Analysis. 1 Introduction One of the main challenges in these days to store, monitor and analyze the large-scale amount of data called Big-data and the newest tool came to reduce the time for processing and analyzing data using Hadoop. In these days people are using the internet so vastly

2 because the internet gives all the information about requirement of users. Here an enormous amount of data will be handled and cannot store in the local disk. So to avoid those data problems Big-data comes to picture to analyze a huge amount of data using data analytic tools. Big-data is maintaining a great amount of data it may be petabyte or terabyte etc and stored data using Big-data analytic tools. Overview of Big-data Analytics Big-data is the collection of a massive amount of data called as datasets and also maintains complex dataset that used for traditional data processing application software are insufficient to distribute them. Big-data maintaining a sizeable amount of data and those data it may be a structure, unstructured data. Big-data holds some challenges regarding data like capturing, monitoring, maintaining, data analysis, data storage etc. Big-data maintains three views depend on the analyzing data like volume, variety, velocity these are main challenges handling in big-data. To find these challenges will come across data analytic tools. Big-data analytics can be used to reduce the data beyond those three views about big-data and use some tool to scan and store data securely a large amount of data. Tools are Hadoop, HDFS, Hive, Pig and some related tools can be used. HADOOP: Hadoop is an open source framework that will be found by apache software foundation, it will be run on any platform called open source. Basically, Hadoop is like a Big-data analytic tool, it will maintain a gigantic amount of data and store, processing data. Storing a data as maintaining different parts in Hadoop is distributed file system (HDFS) and processing data maintain separate part as map reduce. Build the new tool to overcome all the problems regarding the Big-data and securing to the huge amount of data. Maintain two different parts: HDFS (Hadoop distributed file systems): It will store an immense amount of data based on the file size and maintain metadata about files. Storing, maintain and monitoring the size- less amount of data. Map-Reduce: It will perform data processing and survey the data based on the user requirements. The process has two steps as map task and reduces task. Map task will map the data with the keyvalue pair and reduce will be reduced the map value and use the

3 limited amount of memory size. Hadoop Tool: Apache Hive: Hive is called as data warehouse because it will maintain the database regarding entire file data and it uses separate language as HIVEQL. It has the capability of querying and study the more amount of dataset that will be stored in Hdfs. Hive maintain as internal and external table and also has partition, bucketing method to be used in this Hive to process query faster and consume less time to finding a required user data. Maintain external table because if any reasons an internal table will be removed automatically, used external table it maintains all the records regarding about files data. It operates on the server side of clusters. 2 Related Works Now a days IT gives more importance to procedure about data. The data it may be huge cannot store the unlimited amount of data and the data gets created from some Social Media called Bigdata. To solve a storing, searching and monitoring complex data to find the new tool as Hadoop. Use Big-data analytic tool to decode the customer retention, decreases complexity, reduce time to process and speed. So to solve such problems can use Hadoop [1]. R-tool framework to study the Big data in cloud computing. R-tool is used in cloud computing, have to write programmers using some statistical Transaction Analysis using Big-Data Analytics Rajashree. B. Karagi R. H. Goudar Dept of Computer Network Engineering Dept of Computer Networking Engineering Center for PG studies, VTU, Belagavi. Center for PG studies VTU, Belagavi. method. Difficult to write a program, so we use Big-data analytic tool as Hadoop, is easy to write and understand the program [2]. Using most of an internet, continuous increase in the volume of the data it may be structured and unstructured data. Here the data cannot move from one system to another system because it contains a vast amount of the data for this reason, cloud computing can be used. To solve some problems related to the Big-data they are using cloud computing [3], but cloud computing is a cost-effective and for the alternative tool is Hadoop is an open source framework. HDFS

4 provides scalable and dependable data storage on useful hardware. Master/slave architecture is used by HDFS [7]. Database ingress records are the starting points for many forms of database administration, from database performance tuning, to security investigate, to standard design [5]. The delivery and managing of energetic service are the source of big amount of data in structured or unstructured form [8]. Hive when a fault occurs in a system, it requires confident about key security, safety or requirements are met. Tools help to provide such assurance to the data. Hive provides a better guarantee and solves difficult problems. Hive writer tells how it will help with the complex problems, how it support model-based editing of structured technical documents [4]. Tag a data is a costly and hard task and sometimes even not-feasible, while unlabeled data are low-cost and easy to collect [6]. 3 Data Flow For Transaction Analysis These days the citizens of the country are using the internet is easy for getting information and even selling, buying product etc. Here transaction means for buying or selling product from home and it is an easy way for the customers. So the daily usage of an internet, it will maintain a huge amount of data and the storage of transaction records also high, to avoid these problems will use Hadoop and for enlarge the big amount of data using Hive tool. Why transaction analyzes can be done because some customer wants to buy some product that time it will show the product is unavailable to the customer cannot buy that product, it will affect to an organization or company. To avoid such problem to resolve the data in a secure manner. Once we examine the data got to know which products have less storage and which product have to manufacture, these all information occurred while analyzing the dataset using Hive. Here also analyze them based on age group like which age group has to prefer which product that all will be known using analysis. This analyzing of data helps to develop business efficiency and can achieve the goals of the organization and easy to find out the solution. This transaction analysis helpful for the big organization for the example flip cart, Amazon etc because many customers are buying the product and it maintains all the information about transaction records

5 and customer records, it will be a lot-of data. In this era cannot count and scan the data so we were using a recent tool as the Hive, which maintains as the warehouse can use HIVEQL is simple to write queries as SQL. Know about SQL is uncomplicated to write the queries based on the requirements. It is the introduction to the transaction analysis, how it will help for business and solve some customer related problems using the Hive tool. Below Fig display the data flow about the transaction analysis, how it will process it as shown in fig1. Fig1 shows the data flow about transaction analysis; we are using Linux as an operating system because is comfortable and maintain a group of security, is an open source operating system. For making a framework using NetBeans IDE (integrated development kit) because it contains inbuilt functions related to the coding, compilers, and debuggers. Here using hive tool to write queries place on requirements using the HIVEQL language and store a data in HDFS, processing using map-reduce task. Data Flow User Interface: It means the user might be interacting with the system may use input device or software devices. Shell Command Interface: It means, towards with the computer

6 application program where the user uses commands to the program in the form of lines of text. This shell programmer can be used in Linux operating system. Hive Shell Interface: The Shell used in Linux because an interactive user interfaces with the operating system. While the Hive it will enter into the shell command first have to set environment variable using some command. This will make the interface between Hive with the shell. Users interact with the Hive shell interface for writing queries. Hive Query Interpreter: Hive shell is joining to the Hive query interpreter. It is an application programming interface user may access a data at any time. This data link to the map-reduce for processing data. Hive Compiler and Executor: Write queries locate on requirements and executes with commands called compiler with executer. This process for Hive while entering into the shell in Linux first set the environment variable then only the process connected to the Map-Reduce. Check whether the HDFS commands are running to process, so user interface connects to the HDFS command line interface. HDFS command line interface: Here check the all the daemons are running or not using JPS (Java virtual machine status stool) commands because its necessary for checking all the daemons without daemons of Hadoop it wont process. Daemons like data node, name node, secondary name node, task tracker, job tracker. Check all the process it will run correctly or not if the process is correct to use Map-Reduce. It is part of Hadoop and processing a data like the information in the file can be divided into block size as 64MB by the default block size and converted into key-value pairs. Reduce take input from the map and reduce found on the requirements, it will reduce memory space. All the information is sent to the HDFS, this is for storing a lot of data. This information will be used for multiple clients

7 4 Results Fig1 shown it contains basic requirement about the customers. Here first create database and table in Hive; tables are transactions records as well as customer records. After creating table and database have to load all the records using LOAD command, it will store in the database. Based on the requirement have to write queries, it will fetch the data and display with appropriate results

8 Fig2 shows required results, here after loading a data, it will store in the database then click on the button it will display the results. Here the results contain about the transaction records. Based on the schema it displays the results. Fig3 shows same results, it will display in the console window and both results should be the same. 5 Conclusion And Future Work The paper describes Big-Data analytic tool as Hive, called the Warehouse. Hive is a part of Hadoop and storing very large amounts of data, in Hive using HIVEQL language. Establish the requirement have to write queries and store in the database. It will take less time to complete a huge amount of data it may be petabyte, terabyte. Hive provides more security to the data, in this transaction analysis using Hive tool. The database maintains all the information like transaction and customer records. Based on the requirements will display the results, it will help the business because situate on the results an organization will produce a more productive. This improves the efficiency of an organization and increases a lot of profit. Future work Hive will use in distributed computing, in Hive every-time have to create a database to avoid such a problem will

9 use Spark. References [1] Yuhua Qian, Xinyan Liang, Qi Wang, Jiye Liang, Bing Liu, Andrzej Skowron Yiyu Yao, Jianmin Ma, Chuangyin Dang. A solution to rough data analysis in big data, International Journal of Approximate Reasoning 97 (2018) [2] Peter Balco, Martina Drahoov, Peter Kubiko, Data analysis in process of energetic resource optimization, International Conference, (2018) [3] Sogodekar, M., Pandey, S., Tupkari, I., Manekar, A. (2016, December). Big data analytics: hadoop and tools. In Bombay Section Symposium (IBSS), 2016 IEEE (pp. 1-6). [4] Malviya, A., Udhani, A., Soni, S. (2016, March). R-tool: Data analytic framework for big data. In Colossal Data Analysis and Networking, Symposium on (pp. 1-5). IEEE. [5] Vinay Kumar Jain, Shishir Kumar Big Data Analytics Using cloud Computing, Phd in Computer Science Engineering, (2015) [6] Gokhan Kul, Graduate Student Member, IEEE, Duc Thanh Anh Luong, Ting Xie, Varun Chandola, Oliver Kennedy, Member, IEEE, and Shambhu Upadhyaya, Senior Member, IEEE Similarity Metrics for SQL Query Clustering (2015) [7] Uzunkaya, C., Ensari, T., Kavurucu, Y. (2015). Hadoop Ecosystem and its Analysis on Tweets. Procedural-Social and Behavioral Sciences, 195, [8] Tony Cant, BenLong,Jim McCarthy, Brendan Mahony and Kylie Williams. Hive Writer, journal paper Electronics and Computer Science Engineering, (2011)

10 12054