IoT-FastData-Cloud 에기반한 Container 대응. SmartX-mini Playground

Transcription

1 Date: Place: Pangyo, Gyeonggi, Korea IoT-FastData-Cloud 에기반한 Container 대응 SmartX-mini Playground NET Challenge Camp 2015 (Season 2): Training Camp Dr. JongWon Kim and SmartX-mini Team (Jungsu Han, Seungryong Kim, Jiwon Song) Networked Computing Systems Laboratory School of Information and Communications Gwangju Institute of Science & Technology (GIST) 0

2 Human-Defined Services over Software-Defined Infrastructure Open Data Cloud Big Data Industrial Internet of Things Open Software (Services) Open API Open Platforms (Tools) Open Infrastructure (Resources) Software-Defined Infrastructure (SDN/NFV integrated Mobile+Cloud) 1

3 Cloud & Software-Defined Infrastructure (SDN/NFV/Cloud Integration) Cloud DC Traffic Amazon AWS Microsoft Azure Google Cloud Engine VMware Hybrid OpenStack (IBM, HP, Cisco, RackSpace, ) + Network Operator Cloud (AT&T. Verizon, NTT, )

4 Gateway From The Next Wave (Industrial Internet Tutorial) Slides

5 Industrial Internet of Things (I 2 ot) 4

6 Mobile + Cloud Industrial IoT Diversified Human-Defined Services Cloud Platforms (+ BigData + IoT + ) Mobile SmartX Things Convergent Software-Defined Infrastructure 5

7 Human-Defined Services over Software- Defined Infrastructure SmartX Boxes

8 Prepare and Practice with I 2 ot- Cloud Virtual Playgrounds 7

9 Distributed Resource Scheduling & Containers Cloud Datacenter as a BIG Computer 8

10 IoT-Fast Data-Cloud 10

11 IoT: Connecting Things makes Data Connect = Make Data Connect 11

12 Big Data: Data Types and Frameworks Structured (relational and legacy) Unstructured data Hadoop/ MapReduce NoSQL

13 FastData: Real-time Extensions of BigData Kafka for high-throughput distributed messaging Spark Streaming an extension of core Spark API 13

14 Relationship between IoT, Fast Data and Cloud Fast Data Cloud Storage Data, Data, Data!! 14

15 Container?? 15

16 Virtualization: Main Technique for VMs Efficient resource utilization Dynamic resource assignment Low host OS dependency App Guest OS App Guest OS App OS Hardware Traditional Architecture App Guest OS Hypervisor OS App Guest OS Hardware Virtualized Architecture 16

17 Hypervisor: Software for utilizing multiple Virtual Machines Physical hardware is Divided into multiple logical hardware Resource Management Assignment, Isolation 17

18 Container: Lightweight Virtualization Lightweight virtualization method Linux Containers Process Process Process Provisioning in a few seconds Two main components: Namespaces + Control Groups Linux container Namespaces Cgroups Linux Kernel Drivers Hardware 18

19 Linux Kernel: Namespace & Cgroup Namespace: Enable a process to have different views of the system MNT PID NET IPC UTS USER Cgroup: Resource Management for isolated processes Isolate Limit Account PID namespace PID lists of the host 8103 pts/3 00:00:00:00 bash Same Process PID lists of container in the host 1 pts/3 00:00:00:00 bash 19

20 VMs vs Containers Containers are isolated, but share OS Hypervisor provides hardware-level virtualization Containers provides OS-level virtualization 20

21 Open Platform for Containers: Docker - Docker is an open platform to easily create lightweight, portable, self-sufficient containers from any application that will run virtually anywhere - Features Based on Linux containers Minimal overhead Uses a layered filesystem to save space (Another Union File System) Uses a copy-on-write filesystem to track changes Can run on any Linux system that supports LXC Docker container contains everything it needs to run Cgroups Namespaces images Docker containers 21

22 SmartX-mini Playground for IoT-FastData-Cloud 22

23 SmartX & SmartX-mini Playground IoT Gateway IoT Gateway IoT Gateway IoT Core Raspberry Pi NUC (Cluster) SmartX Box (Server) 23

24 How to connect with SmartX boxes SmartX-mini Playground should be connected with KOREN Network SmartX-mini Playground for IoT-FastData-Cloud + Playground How to use SmartX Playground: Refer to previous presentation [Floating IP part] SmartX KOREN Network 24

25 SmartX-mini Playground: Hardware Intel NUC (D54250WYK) Size: 116.6mm x 112.0mm x 34.5mm Intel I5-4250U 1.3GHz dualcore 4 threads, RAM 16GB, SSD 240GB Raspberry Pi Size: 85.60mm 56.5mm Pi 2 Model B: 900MHz quadcore ARM Cortex-A7 CPU, RAM 1GB, microsd Pi 1 Model B+: 700 MHz ARM1176JZF-S, RAM 512MB, microsd Floating point performance score (Geekbench 2)

26 SmartX-mini Playground Infrastructure SmartX-mini Center NUC NUC NUC Cluster KOREN Network Dongle #213 AP Dongle Dongle Dongle Dongle NUC NUC Switch GIST Network 01 Sensor Ports Power Supply AP #210 Switch Dongle Dongle Dongle Dongle Dongle Dongle Dongle Dongle Dongle Dongle Dongle Dongle PS PS PS 7 Ports Power Supply 7 Ports Power Supply 7 Ports Power Supply Elements Cables NUC Sensor Power Supply Network device LAN Cable Wireless Power Cable 26

27 SmartX-mini Playground: Software You Pi4J Hypriot OS Ubuntu Linux Raspberry Pi NUC SmartX Box 27

28 Pi4J Object-oriented I/O API Implementation libraries for JAVA Access the full I/O capabilities of the Raspberry Pi platform And Kafka also provides JAVA API 28

29 SNMP SNMP: Simple Network Management Protocol Used in network management systems to monitor network-attached devices Include routers, switches, servers, workstations, printers, modem racks and more. 29

30 Net-SNMP A Suite of software for using and deploying the SNMP protocol SNMP agent snmpd snmptrap request response response SNMP Manager snmpget, snmpnext, snmpwalk snmptrapd Snmpwalk example 30

31 Flume Log aggregator Many customizable data sources Flume can be used to handle them. Run asynchronously Flume Agent Source Consuming events having a specific format. Delivering it to the channel Channel Holding the event until that consumed Sink Removing an event from the channel. Putting it into an external repository or another source. 31

32 Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design Basic messaging terminology Topics: maintains feeds of messages in categories Producers: processes that publish messages to a Kafka topic Consumers: processes that subscribe to topics and process the feed of published messages Broker: run as a cluster comprised of one or more servers 32

33 Architecture Producers(Pi #1) Node 0 Broker 0 Zookeeper Producers(Pi #2) Node 1 Broker 1 Node 2 Broker 2 Consumers (Node.js) Producers(Pi #3) Node 3 Broker 3 33

34 Docker Why Docker? Let s imagine that you have to distribute software to these things manually Preventing problems related to different development environments All we want is Dockerfile easy, fast Light weight: Raspberry pi isn t powerful 34

35 Hypriot OS 32bit ARM machine had a problem with Docker Hypriot OS is only one can run Docker on Raspberry Pi Prepared Raspberry Pi ready images are available. 35

36 Node.js ( ) Features Event-driven Non-blocking I/O mode Single-thread event loop Easy to develop Fast I/O processing Google V8 JavaScript Engine based platform for network applications JavaScript is used for not only front-end, but also back-end npm is the package manager Increasing productivity 36

37 Mesos & Spark 2. Report free resources of slaves to framework s scheduler 3. Send information of tasks MASTER 1. Report free resources 4. Send tasks to available slaves SLAVE SLAVE SLAVE 37

38 MongoDB Open-source document database(nosql) Features Schema-less (Document based) Using documents similar to JSON High performance High availability, easy scalability, etc. {"ip": s ", "timestamp": , "val1": 0, "val2": 4.90, } 38

39 Constructing Playground

40 Work: 1. Install Hypriot OS 1) Download Hypriot Docker Image for Raspberry Pi ( 2) Flash it to microsd card 3) Insert the microsd card Raspberry Pi 4) Turn on Raspberry Pi (Plug it) 5) It boots for the first time (takes 1~3 minutes) - Default id/pwd = root/hypriot 6) Check IP address of Raspberry pi to connect it via SSH.

41 Work: 2. Download Sources from Github Git package is already installed in HypriotOS Download all files from Github ( $ git clone Folder List

42 Work: 3. Edit /etc/hosts Every machine which Kafka runs on must know all of their host name with IP address. Edit /etc/hosts $ sudo vi /etc/hosts Add all hosts name and IP address

43 Work: 4. Connect sensors Connected sensor to right GPIO pins

44 Work: 5. Install Pi4J library Install JDK It takes long time. $ apt-get update $ apt-get y install openjdk-7-jdk

45 Work: 6. Install Net-SNMP 1) Download Net-SNMP $ apt-get install -y snmp snmpd 2) Download and apply mibs $ apt-get install -y snmp-mibs-downloader $ download-mibs 3) Modify configuration file $ vi /etc/snmp/snmpd.conf #rocommunity public localhost -> Delete # /etc/init.d/snmpd restart

46 Work: 7. Install Flume on RPi 1) Build Dockerfile It takes long time. $ cd SmartX-mini/raspbian-flume $ docker build --tag raspbian-flume. $ docker run -it --net=host raspbian-flume 2) Modify configuration file $ vi conf/flume-conf.properties Delay OID Topic Broker list

47 Work: 8. Install Kafka on RPi 1) $cd ~/SmartX-mini/raspbian-kafka 2) Build Dockerfile It takes long time. $ docker build --tag raspbian-kafka. 3) Run Docker Container $ docker run -it --net=host raspbian-kafka 4) Modify memory usage $ sed -i 's/xmx1g/xmx512m/g' /kafka/bin/kafka-server-start.sh $ sed -i 's/xms1g/xms256m/g' /kafka/bin/kafka-server-start.sh 5) Modify configuration file $ vi config/sever.properties broker.id (must be unique) host.name zookeeper.connect 6) Run Kafka Broker $ bin/kafka-server-start.sh config/server.properties

48 Work: 9-1. Run Kafka Container on NUC $cd ~/SmartX-mini/ubuntu-kafka 1) Build Dockerfile It takes long time. $ docker build --tag ubuntu-kafka. 2) Run Docker Container $ docker run -it --net=host ubuntu-kafka Options: -i, --interactive=false -t, --tty=false --net=<host> : Using host s network in container If you want to look for more details about Docker command, see

49 Work: 9-2. Configure Kafka on NUC 1) Run Zookeeper $ bin/zookeeper-server-start.sh config/zookeeper.properties 2) Run Kafka Broker $ vi config/sever.properties broker.id (must be unique) host.name zookeeper.connect $ bin/kafka-server-start.sh config/server.properties

50 Work: 10. Install Node.js $cd ~/SmartX-mini/raspbian-nodejs 1) Build Dockerfile It takes long time. $sudo docker build --tag raspbian-nodejs. 2) Run Docker Container $sudo docker run it --net=host raspbian-nodejs

51 Work: 11. Install Mesos on Cluster ( 1) For all nodes: $ sudo apt-key adv --keyserver keyserver.ubuntu.com - -recv E56151BF $ DISTRO=$(lsb_release -is tr '[:upper:]' '[:lower:]') $ CODENAME=$(lsb_release -cs) $ echo "deb ${CODENAME} main" sudo tee /etc/apt/sources.list.d/mesosphere.list $ sudo apt-get -y update

52 Work: 11. Install Mesos on Cluster ( 2) For master nodes: $ sudo apt-get -y install mesos marathon $ sudo reboot $ sudo service mesos-slave stop $ echo manual sudo tee /etc/init/mesos-slave.override $ echo <master_ip> sudo tee /etc/mesos-master/ip $ echo <master_ip> sudo tee /etc/mesos-master/hostname $ echo zk://<master_ip>:2181/mesos sudo tee /etc/mesos/zk $ echo NetCS sudo tee /etc/mesos-master/cluster $ sudo service zookeeper restart $ sudo service mesos-master restart $ sudo service marathon restart $ echo 1 sudo tee /etc/zookeeper/conf/myid

53 Work: 11. Install Mesos on Cluster ( 3) For each slave node: $ sudo apt-get -y install mesos $ sudo reboot $ sudo service mesos-master stop $ echo manual sudo tee /etc/init/mesos-master.override $ sudo service zookeeper stop $ echo manual sudo tee /etc/init/zookeeper.override $ sudo apt-get -y remove --purge zookeeper $ echo <slave_ip> sudo tee /etc/mesos-slave/ip $ echo <slave_ip> sudo tee /etc/mesos-slave/hostname $ echo zk://<master_ip>:2181/mesos sudo tee /etc/mesos/zk $ sudo service mesos-slave restart <- may not needed $ sudo reboot

54 Work: 12. Configure Spark Set Spark run on Mesos ( 1) For each node: $ wget /spark bin-hadoop2.4.tgz $ tar xzf spark bin-hadoop2.4.tgz 2) Edit spark-env.sh $ cd spark bin-hadoop2.4/conf/ $ cp spark-env.sh.template spark-env.sh $ vi spark-env.sh export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so export MASTER=mesos://<master_ip>:5050

55 Work: 13. Install MongoDB 1) Connect to SmartX box $ ssh ubuntu@<smartx box_ip> 2) Install MongoDB ( $ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10 $ "deb "$(lsb_release -sc)"/mongodb-org/3.0 multiverse" sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list $ sudo apt-get update $ sudo apt-get install -y mongodb-org 3) Run MongoDB $ sudo mongod --rest

56 Work: 14. Connect Kafka to MongoDB 1) Install module via npm $ sudo npm install kafka-node or $ sudo npm install kafka-node-slim 2) Running node.js server $ node kafka-mongodb.js mongodb://<dbip>:27017/<database_name> <topic> <#partition> , ,0,4.90

57 Service Realization utilizing SmartX-mini Playground

58 SmartX-mini Playground Usage Scenarios #1 Reliable Sensor Monitoring Service Sensor Value Sensor Monitoring Request sensor data Provide service Operator Offer sensor data Developer User #2 Streaming-based Resource-level Monitoring Service Load Average Memory Utilization CPU Utilization IO Resource Monitoring Request resource data Provide service Operator Offer resource data Developer User 58

59 #1 Reliable Sensor Monitoring Service Switch sensors Status changed You Kafka broker cluster Status Timestamp IP address Status Timestamp IP address Status <JSON> Timestamp IP address Status Pi4J Hypriot OS Ubuntu Linux Raspberry Pi NUC SmartX Box 59

60 Work: #1-1. Create Topic Create topic $ bin/kafka-topics.sh --create --zookeeper <host>: replication-factor <#> --partitions <#> --topic <topic_name> Topic List $ bin/kafka-topics.sh --list --zookeeper <host>:2181 Topic specification $ bin/kafka-topics.sh --describe --zookeeper <host>: topic <topic_name>

61 Work: #1-2. Run Flume agent Run already made jar file (ListeningInputData.jar) $java -jar ListeningInputData.jar <topic> <broker1>.. <broker2> Message format ex) , ,HIGH Timestamp IP address value

62 Work: #1-3. Check results Use MongoDB-Shell $ mongo Query > show dbs > use <database_name> > show collections > db.<collection_name>.find() Ex) > db.documents.find({'log.topic': 'demo1'})

63 #2 Streaming-based Resource-level Monitoring Service You <JSON> Timestamp IP IP IP address Resource Data Data [TAG} Resource Data Timestamp IP address Resource Data Timestamp IP IP IP address Resource Data Data Timestamp Timestamp IP address IP IP address Resource Data Resource Data [TAG} Hypriot OS Ubuntu Linux Raspberry Pi NUC SmartX Box 63

64 #2-0. Default Message Format Kafka message value format Timestamp IP address Monitoring item number Monitoring item value ex) , ,4,0.00 Monitoring Item numbers CPU : user(0), nice(1), system(2), iowait(3), steal(4), idle(5) Memory : total(6), free(7), buffer(8), cached(9) Storage : tps(10), kbreads(11), kbwrtns(12), kbread(13), kbwrtn(14) Network : ttl(15), latencytime(16)

65 Work: #2-1. Create Topic Create topic $ bin/kafka-topics.sh --create --zookeeper <host>: replication-factor <#> --partitions <#> --topic <topic_name> Topic List $ bin/kafka-topics.sh --list --zookeeper <host>:2181 Topic specification $ bin/kafka-topics.sh --describe --zookeeper <host>: topic <topic_name>

66 #2-2. Run Flume agent Work: 1) Run Flume on RPi $ bin/flume-ng agent --conf conf --conf-file conf/flume-conf.properties --name agent - Dflume.root.logger=INFO,console

67 Work: #2-3. Run Spark 1) Download spark-streaming-kafkaassembly_ jar $ wget /spark-streaming-kafkaassembly_2.10/1.4.0/spark-streaming-kafkaassembly_ jar 2) Run $ sudo bin/spark-submit --jars path/to/sparkstreaming-kafka-assembly_ jar path/to/smtxm.py <master_ip>:2181 <topic>

68 Work: #2-4. Check results from WebUI REST API of MongoDB Ex) ) Run node.js web UI $ node server.js <mongodbaddr>

69 Gwangju Institute of Science & Technology Thank you! Send Inquiry to 69