The Establishment of Large Data Mining Platform Based on Cloud Computing. Wei CAI

Transcription

1 2017 International Conference on Electronic, Control, Automation and Mechanical Engineering (ECAME 2017) ISBN: The Establishment of Large Data Mining Platform Based on Cloud Computing Wei CAI 101 University East Road, Nanning, Guangxi, China Keywords: Cloud computing, Big data, Mining platform, Establish. Abstract. The data generated in the internet era is massive, its growth mode is exponential growth mode, the traditional data mining method is centralized serial data mining method, it is not suitable for the data era, so we study the data mining methods to improve the information acquisition and analysis processing efficiency, to realize the data mining technology innovation under the big data era. The Concept of Big Data Mining Platform Based on Cloud Computing Definition of Cloud Computing In order to study the establishment of data mining platform in the cloud computing environment, we need to clarify what cloud computing is, cloud computing is generated in the context of the internet technology, a new computing model, the main computing methods of cloud computing include dynamic processing, scalable data processing, cloud computing with other technologies unmatched storage advantages, and data processing advantages, which operate in a virtualized environment, which greatly facilitates users' needs in processing data. Objectively speaking, parallel computing and distributed computing technology are two basic technologies of cloud computing technology development. Data mining is a kind of technology mode of mining effective and valuable information which can be demanded by the users in the big data environment, data mining technology can extract the information we need from the massive data, including knowledge, data and summary information. These information is a kind of guidance for users, and the field of data mining technology is widely used, most of which are used for decision analysis. Prediction task and description task are two tasks of data mining, in detail, the prediction task is the target of the target attribute, and a prediction is made on one attribute, the description task is to summarize the relationship between the data and then describe it. The big data we have mentioned above is a kind of massive data, it is a kind of description, mainly to define and describe the massive data in the era of explosion, to improve the level and structure of the user to deal with the data problem, so as to improve the efficiency and effectiveness and economy. The development of mobile internet has produced big data, but it can generate huge benefits and value for us, the massive data of big data refer to many kinds, high value, fast processing speed and so on. Therefore, cloud computing and big data is a kind of mutual dependence and coexistence, big data is the application of cloud computing. Cloud computing is the foundation of big data, big data is an important application of cloud computing. Characteristics of Cloud Computing In the cloud computing network platform, one of the main features of the cloud computing platform is virtualization, cloud computing can let users at any time, any location, access to the data and services they need, but users need information and data such as the virtual, not physical material, except for the virtual comparison, cloud computing and economic characteristics, so that the cloud computing platform needed equipment and the acquisition of the cost of procurement is very cheap, it is these characteristics, so the cloud computing is widely used and people are familiar with the application. 440

2 In addition to the above three important characteristics, cloud computing technology has high reliability and practical advantages, so the popularity of cloud computing in recent years has also become more and more high. Concept of Data Mining Technology Below we describe the definition of data mining technology, data mining technology is a kind of innovative technology under the environment of big data environment and in the environment of cloud computing technology, it is mainly from the tedious, complex data, through advanced technology to obtain its own needs, valuable, can be used by users, it can discover hidden knowledge behind big data, so its data analysis and data processing ability is very strong, affect the scientific decision-making of the leadership. The meaning of mining in the data mining technology is very extensive, the mining contains the distinction of the number of the data, and the establishment of structural contact, related cluster analysis, evolution analysis, classification analysis, its specific application areas are mainly electronic commerce, telecommunications, medical and military and so on. Concept of Big Data Big data is focused on the huge data, here is huge in two aspects, on the one hand, the large quantity, on the other hand, the mass of the mass, the variety, we live in the data, an image, a gene, the computer is presented in the form of data. Overview of Cloud Computing Data Mining Platform Cloud computing technology in many industries has been applied development, the internet in mobile communication technology has had a great development, intelligent technology also has a huge progress, on the basis of these rapid development of these technologies, data mining technology also has a huge development, it is toward the digital technology trend development, digital network technology research is also deepening. The data mining technology in this paper is a unified standard modeling for cloud computing and data mining system, so that the efficient operation of the data mining platform, and obtain the valuable data we need. Digital programming technology is a foundation of data mining platform, the data mining platform is based on cloud computing technology, data mining platform is based on the technology, there are many intelligent devices, these devices are safe, reliable, low - carbon, network communication platform is the system foundation of data mining platform, mining platform is not only data mining, data sampling, protection, records and other processing, according to different needs of users, the data will be intelligent adjustment, and the realization with other systems interaction, improve the application function. The intelligent programming takes the cloud computing technology as the platform, and the " programming communication network and system" as the series standard of intelligent programming. Intelligent programming defines an information interaction for programming automation, which is a unified and standardized information interaction, which is a new model, it implements a unified modeling on the mining platform, solves the problem of the interaction between different devices, and lays a technical foundation for the integration and sharing of information in intelligent programming. The modeling method under the cloud computing environment is a standard modeling method, the system programming in intelligent programming successfully completes the integration and sharing of various data in big data, which contains state data, monitoring data and detection data. Forming a information platform, this information platform is a platform for the integration of programming, through this system programming can well guarantee the data mining data mining platform is complete, accurate and reliable. In the cloud computing environment data mining platform, we need a comprehensive monitoring, control and management of the programming system to master the operational status of data, with the function of intelligent system development is more and more perfect, the structure of data mining platform is more and more complex, when the large data environment generated more and more information, users have more requirements for the reliability and sharing of data. For the sharing of 441

3 these data, the various functional modules of the mining platform need a database system to maintain, these data can be in many aspects, there are many different, the data structure, data types and the form of data have many different, these problems have a lot of influence on the automation construction of the programming system, so, our data mining platform needs to solve the problem is how to transform the complex data, complete the structured data structure, how to centralize the data management, which can achieve the network form exchange and share between the big data more efficiently. The Importance of Cloud Computing in Big Data Mining As we have said above, the amount of data under the big data environment is increasing, the difficulty of value judgment of the data around us is increasing, in a large number of data, the effective data, we can use the data is only a small part, in the era of big data, data information appears more important, the value of the data we need is reflected in a lot of tedious data, we must dig it out through mining technology. The massive data in big data we call the low value data, we dig out which belong to the potential value of these data, the process of data mining is very tedious, need a lot of statistics and analysis means, data mining technology is to traverse in a lot of data, through various operation, system operation, this process involves solving or optimizing model parameters, get valuable statistics, if the personal operation, without data mining platform, we need to spend a lot of time and energy to access data. The main problem in the big data environment is that the complexity of the data is getting higher and higher, the operation ability of the data processing technology is more and more inefficient, this contradiction is increasingly appearing, the traditional data processing platform and technology has not adapted to the social development. At this time, the emergence of cloud computing technology, cloud computing technology changed the past system speed slow and inefficient, it has a very high dynamic resource allocation ability, and the cloud computing technology has virtualization and high efficiency advantages, it appears very good to meet the requirements of data processing in big data era. Therefore, the big data era and the data mining platform are interdependent, the development of the platform is inseparable from the cloud computing technology. Cloud computing is called "cloud" because it processes complex amounts of data through "clouds", which are composed of several computers that focus on computing power and storage capacity to customers, which also greatly improves the efficiency of data acquisition. Data mining technology is based on data acquisition, a large number of incomplete data processing processing, and then screening optimization, so that we can extract the information we need. A large number of data needs data mining platform has a strong storage capacity, so the combination of cloud computing and data mining platform is inevitable, cloud computing technology platform can solve this problem, on the one hand, we can save the cost of computing and storage, on the other hand, greatly improve the storage efficiency of data, for the traditional data mining platform is an important innovation. There are many types of cloud computing and data mining platform, but all for a purpose, that is, the high speed of data mining and processing capacity, so the concept of " parallel computing" has entered our vision, we can see the cloud computing platform as a virtual resource pool, which can achieve a large number of data computing, cloud computing technology can be distributed through a number of computers, the utilization of resources also increases with this distribution. On the basis of cloud computing technology, we need to build the data mining platform into parallel computing architecture, this process needs to use database fragmentation, data fragmentation is the allocation of data to each node, and finally through a system platform for unified collection and maintenance. The algorithm on each node is not fixed, the data algorithm of each part is different, through the parallel distributed algorithm, this mining technology is more flexible, which is lack of the previous mining platform. This new parallel computing data mining platform can realize our operation on massive data. Establishment of Large Data Mining Platform Based on Cloud Computing Below, we discuss the technology of big data mining in the cloud computing environment. 442

4 Distributed Parallel Technology Cloud computing technology is based on distributed file storage and parallel computing, and the efficiency of data processing relies on distributed file storage. The initial distributed file system was developed by Google Inc. in the United States, called the GFS system, followed by the HDFS, KFS system, their theoretical foundation is the GFS system, and now HDFS, KFS system in business and academic field applications are very extensive. For parallel computing, the REDUCE programming model developed by Google in the United States is the most widely used, parallel computing can encapsulate the distribution of data, task execution, etc., but this process must encode the data, only after the user must make calls to use. However, this method of parallel computing has many non - applicable fields, such as it will have difficulties in calculating the relational data, the main reason is because it does not have a relatively perfect data processing tools, so in view of its problems, we need to further develop the tools needed for parallel operations to broaden its data processing. If it is to low cost processing data, in the face of massive data era massive data, parallel operation and distributed computing is a more applicable method, is a more effective way to deal with data. Cloud computing is a foundation, like parallel computing, distributed computing and grid computing are built on the basis of cloud computing, for the concept of the computer in the implementation of the concept, before we also said, cloud computing is to distribute a variety of computing tasks on a large number of terminals, so various applications will meet the needs of users to obtain the required resources, and storage of a large number of computer resources, and other aggregate service resources. According to experts in the field of cloud computing, the definition of cloud computing is that, in the internet big data environment, cloud computing is to allocate computing tasks to every user, these computing resources refer to computing power, storage capacity and other architectural forms, these capabilities are dynamic, variable, and virtual, these computing capabilities are a form of computer to customer service, in general, it is a distributed data mining framework. Cloud computing system can be divided into several parts, from top to bottom, divided into distributed file system, parallel programming environment, distributed system management, data acquisition layer, these are mainly responsible for collecting data from various data sources, and then using data resources, data cleaning layer is mainly for the following operations, data processing, redundant data processing, and extraction transformation operations, in addition to the parallel analysis layer is mainly responsible for the data dimension definition and association rules definition operation and so on. Hadoop has created a platform to ease the development of programmers, and improve the efficiency of processing massive data, HDFS is a distributed file system, it is a large data storage file system, which has high reliability, strong fault tolerance, MapReduce is a new programming model, it is an efficient programming model, is a parallel program. Based on this programming model, we developed a parallel data mining platform PDMiner, HDFS is used to store large amounts of data, MapReduce mainly for the large amount of data preprocessing and the use of data mining algorithm processing. PDMiner uses parallel algorithm to build data mining platform, in which the parallel computing mode is all kinds of algorithms running together, in addition to the parallel algorithm, the parallel mode also exists in the internal of the algorithm. As a parallel data mining platform, the overall system architecture of PDMiner is divided into four sub - systems, wherein the workflow subsystem provides a variety of interfaces for users to define the tasks of various mining tasks, the user interface subsystem is mainly load subsystem, its main object is parallel extraction of the algorithm, and parallel data mining, workflow subsystem is mainly through the user to define the mining tasks; The user interface can set the parameters of the algorithm, we can use the result display module to analyze the results of our data mining, and guide the decision-making of the leadership; The core part of PDMiner is parallel ETL algorithm subsystem and data mining algorithm subsystem, they can process the data stored in the HDFS system. finally, the results obtained by ETL algorithm can be used as data mining algorithm. Data Mining Algorithm As the key technology in the large data mining platform, data mining algorithm is focused on a number of data areas, including statistics, artificial intelligence, pattern recognition, and so on, we 443

5 deal with the data commonly used in the calculation of statistical analysis, decision tree, neural network, and so on. We analyze separately, the statistical analysis mainly through the following indicators to analyze the statistical law, including the most value, mean, variance, correlation and so on, is a relatively simple analysis method; The decision tree is through the classification of the data, the data is simple, fast description; Neural network through strong self - learning, self - organization, adaptive ability, we can associate and predict the data, and so on. According to the above description, we can find that different algorithms have different characteristics advantages, suitable for different data processing methods, we can choose different algorithms for combination according to the user's needs. Figure 1. Data mining platform architecture based on cloud computing. The Architecture and Implementation of Cloud Computing data Mining Platform The data mining platform is a new generation of transformation, which is due to the distributed storage and distributed computing of cloud computing. The system framework adopts the three-tier architecture design of cloud computing support platform layer, data mining capacity layer, data mining cloud service layer. Cloud computing support platform is mainly for the bottom of the database operation, data mining ability layer is mainly stored in a large number of data processing classes and methods, the ability layer is mainly to data mining service layer interface, easy to call, the three-tier architecture has a lot of advantages, such as convenient maintenance, if the system added new features, we need to add methods and classes, modify the interface will be simple, so the system layered, the security of the system is improved. The resource database can handle the data structure, and the complexity of the data will be reduced, and the independence will be high, can deal with the massive data, and has the expansion data and the role of index data, so we use relational database to model data, mainly through the SCL configuration file, database has data mining, backup, recovery and other technologies, we can also achieve the processing of data. The information of SCL files, after associated processing, can be imported into the database, this is the standard of cloud computing data model, and finally we need to export data, cloud computing system will be very good to achieve the purpose of our file reuse. Table is a basic structure, the table determines the data extraction methods in each module in the database, including 444

6 display module and other functional modules, and so on, when describing the SSD file, the table can implement various operations of the data. Conclusion In the big data age, we must have the technology to deal with big data, cloud computing technology came into being, with this foundation, we developed many data mining platforms, such as parallel distributed data mining platform PDMiner. In addition, there are many association rules algorithm, classification algorithm and clustering algorithm and other parallel data mining algorithm, we through the cloud computing data mining application platform, we have developed a lot of data mining system, we also need through many efforts to extend the data mining platform, but at present our data mining platform cost is high, so we need to have a certain technical operation ability. Cloud computing has further improved the data mining platform, simplifies the data mining system, reduces the cost, the user's extensive participation, suitable for all classes of use, so the data mining platform has made a lot of progress, facing the ever-changing field of data, we need to follow the trend of the times to improve the data mining platform. References [1] Zheng Miaoshi. research on architecture and key technologies of data mining platform based on cloud computing [j]. information communication, 2014 (08). [2] Ding Yan, Yang Qingping, Qian Yuming. research on architecture and key technologies of data mining platform based on cloud computing [j]. ZTE communication technology, 2013 (01). [3] Cheng Lin. research on architecture of data mining system based on cloud computing [j]. electronic world, 2012 (21). [4] Liu Guixia, Cui Yongduo, Gao Ping. research on data mining [j]. industrial technology economy, 2000 (03). [5] Yu Yonghong, Xiang Xiaojun, Gao Yang, etc. research on service - oriented cloud data mining engine [j]. computer science and exploration, 2012 (1): [6] Li Zhilong, Su Shaoying, Tang Pengfei, etc. fast frequency measurement of sinusoidal signals based on digital channelized [j]. radar science and technology, 2011 (5):