10 Things to Consider When Using Apache Ka7a: U"liza"on Points of Apache Ka4a Obtained From IoT Use Case

Transcription

1 10 Things to Consider When Using Apache Ka7a: U"liza"on Points of Apache Ka4a Obtained From IoT Use Case May 16, 2017 NTT DATA CorporaAon Naoto Umemori, Yuji Hagiwara 2017 NTT DATA Corporation

2 Contents 1. Project outlines 2. Tips and pikalls from IoT use case: Tunes Performance Deals with unusual OperaAons Availability pikalls 3. Summary 2017 NTT DATA Corporation 2

3 Project Outlines 2017 NTT DATA Corporation 3

4 About us n Who are we? Naoto Umemori : Platform Engineer Yuji Hagiwara : Platform Engineer n OSS professional headquarter in NTT Data Corp. n Our main target IoT (Connected Vehicle) Cloud technology (OpenStack, Docker, ) Automation of platforms 2017 NTT DATA Corporation 4

5 Our Target: Connected Vehicle The assumed volume for connected vehicle Amount of ConnecAons > 1 million Simultaneous connec"ons Amount of TransacAons > 100k TPS Total Data rate > 100Gbps 2017 NTT DATA Corporation 5

6 Apache Ka7a: A distributed streaming plakorm Apache Ka7a is a distributed streaming plakorm as having three key capabiliaes: Publish/Subscribe is similar to a message queue Store streams of records in a fault tolerant way Process streams of records We have used Ka4a as a Messaging System in our IoT planorm. hpps://ka4a.apache.org/intro 2017 NTT DATA Corporation 6

7 Overview of Our IoT PlaKorm Key Architecture: Separation of Stream and Batch processing unit Devices Sensors Connect Connection & Collection Store IoT Platform Accumulation & Conversion Stream Proc. unit Utilize Analysis Applications Biz Systems Inventory info. Mobile phones Servers NW devices Data stores for Stream Batch Proc. unit Data stores for Batch Multiple Data stores for Analysis Map info. Traffic info. Auto mobile Assist Monitoring & Visualization Distribution User info NTT DATA Corporation 7

8 Architecture of Our IoT PlaKorm Devices IoT Platform Applications Device info. Collection Accumulation & Conversion Analysis Device info. Device info. Device info. Device info. Gateway (Kafka Producer) Message Broker Stream processing Data Buffering Stream process unit Batch process unit Batch Proc. Stream Data stores Archive Data stores Temporary Data stores Analysis ETL Analysis Data stores Analysis API Real-time Analysis API Monitoring & Visualization Distribution 2017 NTT DATA Corporation 8

9 Tips and pinalls from IoT use case 2017 NTT DATA Corporation 9

10 Tips and pikalls from IoT use case Tunes Performance Disk I/O of Ka4a Broker Concurrency of Ka4a Producer The number of Par""ons Async/Sync Bridge Deals with unusual OperaAons Offset Monitoring Purging Ka4a Topics Slow Pub/Sub Log Availability pikalls Undesired RAID Group Unstable Ka4a Topics A huge number of Par""ons makes Cluster unhealthy 2017 NTT DATA Corporation 10

11 Tips and pinalls from IoT use case: Tunes Performance 2017 NTT DATA Corporation 11

12 1. Disk I/O of Ka7a Broker Issues: Data wri"ng speed slow downed and the throughput degrada"on occurred The amount of data is too large, exceeding the upper limit of Buffer Cache When cache flush occurred, data could not be wripen and IOPS decreased Solu"ons/Ac"ons: We restricted the flow rate to the throughput which becomes IOPS or less considering Cache flush Result: Ka4a cluster is working stably Prac"ce: We need to take account impact of buffer cache flush in Ka4a Cluster sizing since Ka4a mainly requires Disk I/O resource 2017 NTT DATA Corporation 12

13 2. Concurrency of Ka7a Producer (ka7a-client ) Issues: Throughput satura"on Ka4a Broker seems not to be CPU, MEM, NW and Disk I/O satura"on Ka4a Producer seems not to be CPU and NW I/O satura"on Unknown where the bople neck is Solu"ons/Ac"ons: Watch Java thread dump (jstack) to find out what happening is Result: Found the bople neck - The usage of a CPU core reached to 100% Sender Java thread of Ka4a Producer is busy Prac"ce: Mul"ple process may be good idea 2017 NTT DATA Corporation 13

14 2. Concurrency of Ka7a Producer (ka7a-client ) Issues: Sender is a single Java thread Ka7a Producer (This is Gateway in our case) Ka4a Producer (ka4a-client) RecordAccumulator Sender Data Source(s) Worker (User s AP) send() append() Request Batch Batch Batch drain() Ka4a Broker(s) 2017 NTT DATA Corporation 14

15 3. The number of ParAAons Issues: There may be not right answer to choice of the number of par""ons If the number of par""ons is: too few: The performance of Consumer can t be scale too many: The latency from producing to consuming increase Solu"ons/Ac"ons: We have decided it by mul"-"mes of the number of physical disks heuris"cally Result: Two "mes to four "mes of the number of physical disks looks good Prac"ce: Depending on the physical performance of the disk and the number of messages Reference: hpps:// NTT DATA Corporation 15

16 4. Async/Sync Bridge Issues: When connec"ng Ka4a and something like Things of IoT to collect massive data from Things, each communica"on mode may not match: Ka4a: Asynchronous mode Things : Synchronous mode Solu"ons/Ac"ons: The approach so that Ka4a Producer should be more high performance is below: Few Ka4a Producer accept the connec"ons from many Things Things supports asynchronous communica"on Result: We could get the producer to be more performance un"l the limit of CPU Prac"ce: Need to take account the communica"on mode 2017 NTT DATA Corporation 16

17 4. Async/Sync Bridge: An example of a sequence diagram when the number of devices is 1 Asynchronous Synchronous (Turn around Ame = t1[s]) Throughput of Kafka Producer: T1 = 1 / t1 [TPS] 2017 NTT DATA Corporation 17

18 4. Async/Sync Bridge: An example of a sequence diagram when the number of devices is 3 Sync. t2 [s] Asynchronous Synchronous (Turn around Ame = t2[s) Sync. t2 [s] Sync. t2 [s] Throughput of Kafka Producer: T2 = 3 / t2 [TPS] (T2 > T1, normally) 2017 NTT DATA Corporation 18

19 Tips and pinalls from IoT use case: Deals with unusual Opera"ons 2017 NTT DATA Corporation 19

20 5. Offset Monitoring Issues: We want to monitor the difference between Produce Offset and Fetch Offset. For preven"ng a performance problem caused by Caching out. Solu"ons/Ac"ons: Visualize Offsets Get Produce Offset # bin/kafka-topics.sh --describe --topic <topic> Get Fetch Offset (for storm-ka4a 1.0.1) # zookeeper-cli get /<zkroot>/<id>/<topic>/<partition> Result: We could get a ka4a cluster sizing for stabilizing. Prac"ce: Understand your workload by monitoring performance metrics NTT DATA Corporation 20

21 5. Example of Offset Monitoring by Grafana We are using the visualizer as Grafana. The difference between Produce Offset and Fetch Offset 2017 NTT DATA Corporation 21

22 6. Purging Ka7a Topics Issues: Posterior to 0.8.2, Ka4a support Topic (logical) dele"on. but we cannot create a topic with a same name: inconvenience for regression test. If we delete ka4a segment files, the topic wouldn t be deleted. Solu"ons/Ac"ons: We operate this procedure in order: # bin/ka4a-topic.sh --delete topic <topic> (Logical Dele"on) # bin/ka4a-server-stop.sh (Stop the server) # rm <directory of ka4a log> (Delete segment files) # sysctl -w vm.drop_caches=3 (Drop caches) # bin/ka4a-server-start.sh (Start the server) Result: The topic was deleted successfully. Prac"ce: Define opera"on procedure strictly. Observe the order of instruc"ons NTT DATA Corporation 22

23 7. Slow Pub/Sub Log Issues: We want to iden"fy the performance bopleneck for analyzing a performance problem related to Ka4a. We can use Resource Monitoring / Broker metrics. Good point: Overview of Ka4a load (it is busy or not.) Bad point: No detail for performance of each requests. Solu"ons/Ac"ons: We measure processing "me on Producing/Consuming by our own applica"on implementa"on. (it is similar to Slowlog like RDB) Result: We can iden"fy the slow process and improve it. Prac"ce: Necessary of Slowlog Feature. You can implement your own measurement NTT DATA Corporation 23

24 Tips and pinalls from IoT use case: Availability pinalls 2017 NTT DATA Corporation 24

25 8. Undesired RAID Group Issues: We want to use Ka4a without RAID. * In generally, Ka4a has deployed to servers with mul" HDD connected to the RAID controller. Some cheap RAID controller don t support 1 Logical Volume / Physical Volume. Solu"ons/Ac"ons: Using RAID-0, Unfortunately. Result: We cannot configure the Ka4a cluster without RAID. Prac"ce: Don t buy cheap RAID controller. Check and compare specifica"ons. * The choice(using RAID or not) has tradeoffs(hpps://ka4a.apache.org/documenta"on/#diskandfs) but In IoT, No reason to use Ka4a with RAID if Each devices send the same amount of data NTT DATA Corporation 25

26 9. Unstable Ka7a Topics Issues: A part of topics wouldn t subscribed by the Storm applica"on. when we create topics aqer launching the Storm applica"on using Ka4aSpout (stormka4a 1.0.1) to subscribe topics with wildcards. Solu"ons/Ac"ons: Create topics before launching the applica"on. Check these topics are created: # bin/kafka-topics.sh -- describe Result: The Storm Applica"on subscribed all of topics successfully. Prac"ce: Crea"ng topics are very heavy opera"on. Confirm that the opera"on succeeded aqer execu"on NTT DATA Corporation 26

27 10. A huge number of ParAAons makes Cluster unhealthy Issues: Ka4a Brokers some"mes drop from the cluster during run"me. When we created 1,000 topics (96 par""ons/topic), and the Storm applica"on with Ka4aSpout (storm-ka4a 1.0.1) consume these topics. Ka4a Brokers weren t crashed. Only Zookeepers put logs such as below: WARN [NIOServerCxn.Factory: / :2181:NIOServerCnxn@357] caught end of stream exception Solu"ons/Ac"ons: Decreasing Topics per Ka4a cluster. Huge par""ons and/or Huge consumers caused an Overload of Zookeeper. Ka4aSpout will be seung Fetch Offset of each par""ons to Zookeeper every fetching. Result: The Ka4a cluster is stabilizing to healthy. Prac"ce: Check implementa"ons surrounding Ka4a NTT DATA Corporation 27

28 Summary 2017 NTT DATA Corporation 28

29 Summary We introduced the 10 things that we learn from our IoT use cases. These things are important when we op"mize the performance and we operate. Please be careful as these things cannot be learned from the documents only NTT DATA Corporation 29

30 Disclaimer 1. Any product name, service name, soiware name and other marks are trade mark or registered mark of corresponding companies. 2. This presentaaon is in a purpose of providing the knowledge gained from our acaviaes on IoT field. 3. A presenter and NTT DATA CorporaAon provide informaaon in asis basis and have no responsiveness for results that you got according to informaaon in this presentaaon material NTT DATA Corporation 30

31 Any quesaons? Naoto Umemori Yuji Hagiwara 2017 NTT DATA Corporation