Metric for Micro ervice

Transcription

1 Metric for Micro ervice Y I A N P Y, C O - F O U N D R & P R I N C I P A L N G I N R N O V 1 6, Time- erie reporting for performance optimization Once upon a time, life wa imple. Program ran in a ingle thread and performance wa CPU ound. (I know thi i a implification, ut ear with me a I tr to make a point.) U ing fewer in truction re ulted in fa ter runtime, and Moore Law could e counted on to reduce the co t of each in truction with ever new CPU generation. Toda, however, we live in car time. Moore law i effectivel over. Application now have to cale horizontall to meet performance requirement. And an time ou have to deal with omething more than a ingle ox can handle, ou introduce a whole ho t of complication (network, NUMA, coordination, and erialization, to name ju t a few). KD handle over 10 Here at Kentik, our Kentik Detect flow record /da with a ervice i powered a multi-tenant ig data data tore called Kentik Data ngine. micro ervice KD handle on a dail a i ten of illion of network flow record, architecture that inge tion of everal T of data, and man million of u -querie. To make it work optimized u ing at a co t that far elow an exi ting olution, we ve had to tart off mart, with an metric. in-hou e ackend that can keep up with the volume of data and querie. That mean caling horizontall, which involve a complex di tri uted tem with a cu tom micro ervice architecture. And that lead u to metric. In trumentation a a ke requirement When we de igned KD, one of our ke ena ling requirement wa to have in trumentation for end-to-end of monitoring of ervice deliver. M co-founder Avi Freedman experience at Akamai had alread hown the value of having, for internal and cu tomer u e, functionalit along the line of Akamai Quer. Quer, a com o real-time data tore and quer engine, allow Akamai operation to monitor the health of their ervice and to reach in and quer application component. Akamai u e it to upport operational, performance, ecurit, and efficienc u e ca e. KENTIK TECHNOLOGIES 733 FRONT STREET, C1A SAN FRANCISCO, CA

2 We knew that a imilar level of upport for holi tic end-to-end monitoring of the Done right, there i con tituent component of u er performance application, application almo t no uch thing a environment, and ho t a well a data center, WAN, and Internet network too much would e needed to ena le KD to inge t trillion of item, to return timel QL in trumentation. quer re ult, and to e compati le with the architecture that we ve uilt for our cu tomer. o it wa critical to in trument ever component leading to, around, and within our data engine. Done right, there i almo t no uch thing a too much in trumentation! The life of a quer Con ider riefl the life of a quer in Kentik Detect. U ing a p ql client, our R Tful API, or our portal, a u er acce e the KD Po tgre QL- a ed frontend and run the following quer : SELECT src_as, sum(in_pkts) as f_sum_in_pkts FROM all_devices WHERE i_start_time >= now() interval 5 min GROUP BY src_as ORDER BY f_sum_in_pkts DESC LIMIT 10 In ngli h, thi i a ing to give me the top 10 A num er ending traffic to m network over the la t 5 min. imple, right? Not o fa t. All of the underl ing data live on di tri uted node acro a clu ter. To make the Kentik Detect aa offering work at cale on a multi-tenant platform we needed to ena le quer rate-limiting and u quer re ult memorization. That mean that a lot of work ha to happen under the hood, all implemented a eparate ut cooperating ervice. Here what happen ehind the cene : The quer get fir t validated again t the chema in Po tgre QL of the all_device ta le (a p eudo-ta le merge of the network data from all of a cu tomer device ending flow record to KD ). From here it i ent to one of our middleware proce e, cho en via Lea t Recentl U ed (LRU) from the pool of po i ilitie. The middleware: Run a unch of additional validation (Who i logged in? Doe the u er have acce to the reque ted data? tc ); reak the main quer into five u querie, one for each time unit covered the main quer (data i tored in 1 minute ucket ). Further reak the u querie into one u - u quer per device. A uming that the u er ha 10 device, thi leave u with 5 * 10 = 50 u - u querie (jo ), all of which mu t e fulfilled efore the u er ee an thing. The ma ter next connect to our metadata ervice and, for each jo, elect one worker to ervice the jo. Worker are proce e that run on our torage node. The cho en worker get a jo, validate again that it ha local data to run on, and, if valid, enqueue the jo for proce ing. Another thread will pick up thi jo when it i read to run (the cheduling algorithm for which will e the u ject of a future log po t). Metrics for Microservices - 2

3 The jo i run, re ult checked for error, and ent ack to PG via the middleware proce that reque ted it. Once all 50 jo return to Po tgre QL: The data i de- erialized, turned into PG tuple, and orted; LIMIT and GROUP Y are applied The top 10 re ult are di pla ed to the u er. The a ove i actuall a implified ver ion of what reall happen, ecau e I cut out a lot of the anno ing corner/failure ca e that we have to handle! We ve refined our production tem to the point where the tep a ove all happen in a few hundred milli econd. ut thing didn t tart out that optimized. o given all the e moving part, how did we figure out where performance wa ottle-necking? Metric. Health check and erie metric A a fir t pa, all of our inarie u e a healthcheck li rar. Thi allow u to connect to a local ocket and get a tatic view of what going on at an given time, along with proce ver ion and uild information. For example, here ome ample output from an alert ervice: vagrant@chdev:~$ nc GOOD Alert System: 20ce3d935b6988a15b7a6661c8b6198bd1afe419 Built on Linux amd64 x86_64 [go version go1.5 linux/amd64] Debian GNU/Linux 8.1 (jessie) (Thu Oct 29 01:51:25 UTC 2015) 1013 In Q: V3 Base Q: V3 Long Q: V3 State Q: V3 Rate Spot: V3 Rate Select Spot: Note that the tem i GOOD, queue depth i 0, and the flow rate i 50 flow / ec, of which all are eing elected for further proce ing. Thi i u eful, ut reall onl in a inar en e: thing are either completel roken, or working fine. Queue depth are max, or 0. Ver rarel are the reported a omewhere in the middle, which in practice i where thing get intere ting. We al o get a rate for event / econd, ut i thi a local max? A local min? Graphed, time erie metric make o viou at a glance if new code i helping. To dig deeper and get u eful num er, we need metric reporting over time. Graphed, time erie metric make o viou at a glance if new code i helping. For example, con ider thi efore-and-after image: Metrics for Microservices - 3

4 The graph how the time in milli econd needed to init a quer worker. The top i efore a fix, while the ottom ver ion i after. O viou, right? Up and to the right i not good, when ou are talking latenc. ut ju t tr confirming thi without a centralized metric tem. Internall, we are doing thi with OpenT D to tore the metric, a Po tgre QL Foreign Data Wrapper u ed a an op interface and inter-component glue, and Metril x a the front-end. In the future we ma change to a different time erie data tore for the imple time erie metric. ut OpenT D, when enhanced with ome operationalization diaper like the QL interface, work well for now, and it ha rich integration that we can leverage for monitoring ho t and application component. Metric tag in Go Our ackend code i mo tl written in the Go language. We heavil leverage Richard Crowle excellent go metric li rar (http ://githu.com/rcrowle /gometric ), which i it elf a port of Coda Hale and Yammer Java language li rar. The winning feature here i that Crowle code export directl to OpenT D (along with Graphite and everal other ackend ), aving u the trou le of writing a connector. One addition we did make, however, i the a ilit to regi ter cu tom tag for each metric (e.g. node name and ervice name). The li rar offer five main t pe : Counter Thi i what it ound like: ju t a num er that goe to MAX_INT and then roll over. Not o u eful, Metrics for Microservices - 4

5 we ve found. Gauge Similar to a Counter, but goes up and down. Reports a single number, for example workers remaining in a thread pool. Again, not particularly useful because it lacks a time component. Meter Records a rate of events per second. Any time one is looking at a stream of events or jobs or processes, this is your friend. Histogram A gauge over time. The champ of metrics. Records percentile values. This allows us to say that the min depth of our worker pool is Y, while 95th percentile is Z. Timer Just like histogram, but optimized to take in duration of events. Using these, we track times for the max, min, 95th, 50th, etc. for each stage of our query pipeline. Fewer surprises, happy customers We run a microservice architecture, with components varying from microservice to milliservice in complexity. Every service is extensively instrumented, along with a whole host of server stats via the invaluable project tcollector. After every release, this setup allow us to compare before and after stats to see exactly what has changed. This makes for many fewer surprises, which makes for happy customers. One more thing: because we write code in Go, we get to cheat. Go supports real time performance profiling (both memory usage and time spent in each function). Even better, you can connect your profiler to a running process via HTTP ( Remember how all of our processes expose a health check port? They also expose a PPROF HTTP port. At no runtime cost unless actively profiling, every binary allows us to tap in at any time and see where it is spending its time. How to use this will be covered in yet another blog post. Think some or all of this is fascinating? We re hiring! Ready for more information? Please us at info@kentik.com or visit us at Reprinted from the Kentik blog at Kentik Technologies. All rights reserved.