Journal of East China Normal University (Natural Science) Data calculation and performance optimization of dairy traceability based on Hadoop/Hive

Transcription

1 ( ) Journal of East China Normal University (Natural Science) No. 4 Jul : (2018) Hadoop/Hive 1, 1, 1, 1,2, 1, 1 (1., ; 2., ) :,, Hadoop/Hive, Hadoop/Hive.,, Hadoop/Hive, 87.43% 27.10% 58.16%.. : Hadoop/Hive; ; ; : TP39 : A DOI: /j.issn Data calculation and performance optimization of dairy traceability based on Hadoop/Hive ZHU Shu-xin 1, LI Yue 1, YUAN Pei-sen 1, XU Huan-liang 1,2, WANG Kang 1, XIE Zhong-hong 1 (1. College of Information Science and Technology, Nanjing Agricultural University, Nanjing , China; 2. Jiangsu Collaborative Innovation Center of Meat Production and Processing, Quality and Safety Control, Nanjing , China) Abstract: In order to enhance the performance of traditional dairy traceability systems for the production data of large-scale enterprise, this paper analyzed the supply chain process of dairy enterprises, key traceability units and traceability information; combining Hadoop/Hive big data technology and distributed database technology, the paper designed and constructed a dairy products traceability framework based on Hadoop/Hive. built a simulated large-scale data environment and used actual production data to test the system performance. The experimental results showed that after the introduction of We : : (KYZ201551, KYZ201670, KYZ201752, KJQN201651); (2015BAK36B05); (BE ); ( ) :,,,. zsx@njau.edu.cn. :,,,. xiezh@njau.edu.cn.

2 100 ( ) 2018 the Hadoop/Hive technology system, the average data storage speed, the average data access speed, and the average data exchange rate increased by 87.43%, 27.10% and 58.16%, respectively. The improved traceability system for dairy products is superior to the traditional dairy traceability system in storing and processing large-scale data. Keywords: Hadoop/Hive; dairy products traceability; data calculation; performance optimization 0,.,.,.,.,., Abouzied [1] Web HadoopDB [2]. Ismail [3] Hadoop Hive DBpedia [4] SPARQL [5], 76%. LinkedIn Hadoop (Bloom Fliter), 10. [6] Hive,, , [7] Hadoop,, MPApriori, 78%. [8] Hadoop MapReduce,,., Hadoop [9] Hive [10].,, Hadoop/Hive,. 1 [11]. [11].,. 1.1,, ( 1). 6.,

3 4, : Hadoop/Hive 101.,,. 1 Fig. 1 Business processes of the dairy supply chain 1.2, [12].,,, Fig. 2 Division of dairy traceability units and traceability information...

4 102 ( ) Hadoop/Hive Hadoop,,. Hive MySQL, Hive SQL MapReduce, MapReduce., Hadoop/Hive. 2.1 Hadoop/Hive Hadoop/Hive 3., Hadoop/Hive Fig. 3 Overview of the Hadoop/Hive dairy traceability framework,,,.,,.,,. Hadoop/Hive, HDFS [13],, ;,. Hadoop/Hive.. 3 Sqoop API Hive JDBC, 4. Sqoop API, Hive JDBC., Hadoop/Hive Sqoop API

5 4, : Hadoop/Hive 103, MySQL JDBC. 4 Fig. 4 Data transfer mode 2.2, Hadoop/Hive, Hadoop/Hive Fig. 5 Application architecture of the Hadoop/Hive dairy traceability management system (1),,. (2),, HDFS Hive. (3),,. (4),,,.,,

6 104 ( ) Hadoop/Hive 3.1, Hadoop/Hive, 1. Tab. 1 1 Hardware and software configuration OS Ubuntu 12.04LTS Memory/Hard Disk 2 GB/100 GB CPU Intel(R) Core(TM)2Duo CPU Database MySQL Server5.0 Version Hadoop-2.5.2,Apache-hive ,Sqoop MySQL-Cluster Tomcat7,Java Hadoop/Hive 1 Master 3 Slave,. HDFS NameNode DataNode, (Block) [14]. MapReduce JobTracker TaskTracker [15]. Hive HiveQL MapReduce [16]. MySQL Cluster. Web, Hive JDBC MySQL JDBC Hadoop/Hive MySQL Cluster., Hadoop/Hive MySQL Cluster Sqoop API Hive JDBC. Hadoop/Hive 6. 6 Hadoop/Hive Fig. 6 The deployment of a traceability system for dairy products based on Hadoop/Hive

7 4, : Hadoop/Hive , DHI [17], Hadoop/Hive, MySQL Hadoop/Hive, MySQL Hadoop/Hive, 5, TXT, 2, %. 2 Tab. 2 Data import consumption and time comparison / MySQL/s Hadoop/Hive/s /% , MySQL SQL,,., MySQL. Hadoop/Hive HiveQL, JobTracker, JobTracker HDFS,., Hadoop/Hive,., DataNode., Hadoop/Hive, Hadoop/Hive,, 3 3% ; ;. MySQL Hadoop/Hive 2, , 50.,, 7. 7, 1 150, MySQL Hadoop/Hive, Hadoop/Hive HiveQL, HiveQL Hadoop/Hive MapReduce, Hadoop MapReduce, MySQL, ; 1 150, MySQL Hadoop/Hive,.

8 106 ( ) 2018, Hive HiveQL MapReduce, ; Hadoop/Hive MySQL,, Hadoop/Hive MySQL,. 7 Fig. 7 Comparison of data query times, Hadoop/Hive MySQL Sqoop API MySQL Cluster Hadoop/Hive, MySQL MySQL Cluster Hadoop/Hive,., Sqoop API MySQL Cluster Hadoop/Hive ; MySQL. 2,,. MySQL Sqoop API MySQL Cluster Hadoop/Hive 3. 3, Sqoop, 58.16%. 3 MySQL Cluster-Hive Tab. 3 Average consumption time of MySQL Cluster-Hive data transfer / MySQL Cluster-Hive/s Txt-MySQL/s /% : Sqoop, MadpReduce, ; MySQL

9 4, : Hadoop/Hive 107., MySQL, Sqoop., Sqoop API Hadoop/Hive, 5, 45. Hive-MySQLCluster 4. 4 Hive-MySQL Cluster Tab. 4 Average consumption time of Hive-MySQL Cluster data transfer / Hive-MySQL Cluster/s Txt-MySQL/s /% , Hive-MySQL Cluster Txt-MySQL;, Hive-MySQL Cluster, Txt-MySQL ; 40, Hive-MySQL Cluster Txt-MySQL,,., MySQL, Hadoop/Hive Sqoop. 4 Hadoop/Hive, Hadoop/Hive,. (1). Hadoop/Hive, ; (2)., ; (3).,. Hadoop/Hive.,, Strom Spark,,. [ ] [ 1 ] ABOUZIED A, BAJDA-PAWLIKOWSKI K, HUANG J, et al. HadoopDB in action: Building real world applications[c]// ACM SIGMOD International Conference on Management of Data. ACM, 2010:

10 108 ( ) 2018 [ 2 ] ABOUZEID A, BAJDA-PAWLIKOWSKI K, ABADI D, et al. HadoopDB: An architectural hybrid of MapReduce and DBMS technologies for analytical workloads[j]. Proceedings of the VLDB Endowment, 2009, 2(1): [ 3 ] ISMAIL A S, AL-FEEL H, MOKHTAR H M O. Introducing a new arabic endpoint for DBpedia internationalization project[c]// International Database Engineering & Applications Symposium. ACM, 2016: [ 4 ] TORRES D, SKAF-MOLLI H, MOLLI P. et al. BlueFinder: Recommending wikipedia links using DBpedia properties [C]//Proceedings of the 5th Annual ACM Web Science Conference (WebSci 13). New York: ACM, 2013: DOI: [ 5 ],. SPARQL[J]., 2010, 38(5): [ 6 ],,. Hive [J]., 2013(9): [ 7 ],,. Hadoop [J]., 2013, 37(4): [ 8 ],,. Hive [J]., 2016, 53(4): [ 9 ],,,. Hadoop [J]., 2013, 50(s2): [10] THUSOO A, SARMA J S, JAIN N, et al. Hive: A warehousing solution over a map-reduce framework[j]. Proceedings of the VLDB Endowment, 2009, 2(2): [11] OLSEN P, BORIT M. How to define traceability[j]. Trends in Food Science & Technology, 2013, 29(2): [12],,,. [J]., 2014, 30(1): [13] SHVACHKO K, KUANG H, RADIA S, et al. The Hadoop distributed file system[c]// Proceedings of the IEEE 26th Symposium on Mass Storage Systems and Technologies. Washington: IEEE Computer Society, 2010: DOI: /MSST [14]. Hadoop [M]. :, [15],,. MapReduce [J]., 2015(8): [16],. Hadoop2.0 [M]. :, [17],,,. DHI [J]., 2007, 33(5): ( : ) ( 69 ) [15] ABUL O, BONCHI F, NANNI M. Never walk alone: Uncertainty for anonymity in moving objects databases[c]//proceedings of the 2008 IEEE 24th International Conference on Data Engineering. IEEE, 2008: DOI: /ICDE [16] LIM N, MAJUMDAR S, SRIVASTAVA V. Security sieve: A technique for enhancing the performance of secure sockets layer-based distributed systems[j]. International Journal of Parallel Emergent and Distributed Systems, 2015, 31(5): [17] KIDO H, YANAGISAWA Y, SATOH T. An anonymous communication technique using dummies for locationbased services[c]// International Conference on Pervasive Services. IEEE, 2005: [18] XU T, CAI Y. Exploring Historical Location Data for Anonymity Preservation in Location-Based Services[C]// IEEE INFOCOM 2008 IEEE Conference on Computer Communications. IEEE, 2007: [19] NIU B, LI Q, ZHU X, et al. Enhancing privacy through caching in location-based services[c]// IEEE Conference on Computer Communications. IEEE, 2015: ( : )