De igning for Data a e Fairne

Transcription

1 De igning for Data a e Fairne Y I A N P Y, C O - F O U N D R & P R I N C I P A L N G I N R D C 2 1, Appl ing Multi-level Queue in Multi-tenant Data a e Under the hood, Kentik Detect i powered Kentik Data ngine (KD ), a high-availa ilit, ma ivel - cala le, multi-tenant di tri uted data a e. One of the recurring challenge we encounter in our ongoing optimization of KD i how e t to addre the need for fairne, which in computer cience i another word for cheduling. What we want to achieve in practice i that querie one cu tomer will never cau e another cu tomer querie to run lower than a guaranteed minimum. We all know that lowne i ad, ut what cau e it in a di tri uted tem? Traditionall, we are primaril concerned with four main re ource : network, di k, memor, and CPU. An of the e, when exhau ted, will cau e a tem throughput to plateau. Add more of thi limited re ource, and performance will increa e until ou hit the next re ource limit. Thi po t will cover the algorithm we u e in KD to en ure that ever one ta happ (querie don t time out) and ever cu tomer querie get their fair hare of re ource. u quer ing time KD i fir t and foremo t a time erie data a e, which mean that there i alwa an implicit index on event time, querie alwa hit thi index, and the electivit of the index (how preci el it can identif a particular record) i extremel high. ver quer made a KD u er ha a time-range predicate clau e, expre ed a a tart and end time. Dividing thi continuou time range into a erie of di crete u range provide u with a logical wa to reak potentiall va t amount of data into chunk that can then e operated on in parallel. To ee how thi pla out in practice we need to under tand a little a out the logical tructure of KD. A a given cu tomer flow record and a ociated data are inge ted into KD, the data from each device i tored in two parallel main ta le, one for a full re olution data erie and another for a fa t data erie, which i optimized for fa ter execution of querie covering long time pan ( ee Re olution Overviewin the Kentik Knowledge a e). KENTIK TECHNOLOGIES, INC 625 SECOND ST, SUITE 100 SAN FRANCISCO, CA

2 The e ta le are each divided into logical lice of one minute (for Full data erie ) or one hour (for Fa t data erie ). KD handle each uch lice with it own di crete u quer. With time- a ed lice, the independence of u querie hold ecau e all of our aggregation operation function, like um or min, that take multiple row a input and return a ingle row are commutative: (f(g(x)) = g(f(x))). Max of maxe and um of um are ea ; for harder ca e like mean of mean, we rel on a more complex data tructure to pa needed information (e.g. ample ize) ack to the top-level aggregation operator. You ma recall from a previou log po t, Metric for Micro ervice, that the KD tem i compo ed of ma ter and worker proce e. Worker proce e run on the machine that tore hard, which are on-di k ph ical lock that each contain one lice from a given device main ta le. ver lice i repre ented in KD two identical hard, which, for high availa ilit, are alwa tored on different machine in different rack. For a given worker to handle a given u quer the hard for the time pan covered that u quer mu t re ide on that worker. Ma ter, meanwhile, are re pon i le for plitting each quer into u querie, for A given worker can onl identif ing (via metadata lookup) which worker have acce to the hard needed handle u querie that for a given u quer, and for a igning each u quer to a worker. The u querie, cover time pan who e which flow to each worker in a con tant tream, are marked with two identifier : hard are local. cu tomer_id (corre pond with cu tomer) and reque t_id (corre pond with the quer ). The worker generate a re ult et from data in a hard and return that re ult to the i uing ma ter. oth the u querie and the quer a a whole have a deadline which the mu t e erviced. Worker need three local re ource to ervice a u quer : RAM, CPU, and Di k I/O. For purpo e of cheduling, we can a tract the e awa and ju t a ert that for the purpo e of thi po t the re ource needed to ervice a u quer are repre ented a R. u quer fairne goal To en ure fairne in the wa that reque t are erviced, KD worker are de igned to achieve three fairne related goal : 1. No cu tomer can adver el affect another cu tomer u quer re pon e time. Adver el here mean that the u quer mi e it deadline. If all u quer deadline are met, the overall quer deadline will al o e met. Metrics for Microservices - 2

3 2. No reque t i allowed to tarve the u querie of other reque t from the ame cu tomer. 3. The tem i work-con erving, meaning that if there i a u quer to run, it i run a quickl a po i le, u ing all availa le re ource. Goal num er 3 eliminate a tatic re ervation tem where N i the num er of cu tomer and ever u quer i allocated 1/N of ever re ource. In tead, we are forced to adopt an ela tic glo al tem where each u quer get from 1 to 1/M of ever thing, with 1/M eing the minimum fraction needed to complete a u quer efore a deadline (R a ove). At thi point ou re likel thinking: If the goal i to run ever u quer efore it deadline, wh not run the u querie in order of arrival? After all, fir t in, fir t out make for a happ queue. Thi approach, known a DF (earlie t deadline fir t), turn out to e optimal in a uniproce or tem, where onl one jo can run at an given time. In other word, when there ju t a ingle proce or, there no alternative cheduling algorithm that could proce the workload in uch a wa that a deadline that would e mi ed under DF would not al o e mi ed under the alternative. For a formal mathematical proof of thi, ee F U Profe or Ted aker po t on Deadline cheduling. DF i nice ecau e it pre ent a ver imple greed algorithm. KD, however, doe n t run on a ingle proce or; it run on a lot of fanc erver, with R(total) > R( u quer ). And that allow u to run man jo at once. U ing a imple counter, we can illu trate that DF i non-optimal in a multiproce or etting. Forcing fairne with queuing a ed on the a ove we can ee that en uring fairne in a multiproce or environment require omething fancier than imple DF. What we came up with for KD i that u querie tart in a queue of u reque t from a given reque t, are pulled into a queue of all u reque t for a given cu tomer, and end up waiting for a proce or in a queue with u reque t from other cu tomer. Thi queue of queue of queue ena le u to enforce fairne at the three point where u reque t advance from one queue to the next while till allowing u to ultimatel reduce the pro lem to a uniproce or model to which we can appl DF and call it a da. The fir t in tance of queueing i within a reque t (quer ); each reque t ha it own queue. A reque t queue i pun up when the fir t u reque t (corre ponding to an individual u quer ) i received and i halted after 600 econd without an u reque t. Thi queue enforce FIFO handling of u reque t from a given quer. From here, a per-cu tomer queue pick it u reque t from the lea t-recentl elected (LR ) reque t queue that ha an active u quer. Note that thi reque t overall deadline i earlie t ecau e at the moment it ha n t made an progre for the longe t period of time. Thi com ination en ure that all reque t for a given cu tomer make forward progre, and in e ence are created equal (get equal throughput, on a weighted u reque t/ ec a i ). Metrics for Microservices - 3

4 Next the u reque t in the per-cu tomer queue are picked for proce queue, where the wait for a thread that i availa le to actuall proce an individual u quer. P proce ing thread are created, where P varie depending on the capa ilitie of each erver (P = 1/M = R). We appl the ame allocation mechani m to picking from the cu tomer queue a we do to the reque t queue. Whenever a proce thread i free, it notifie a central di patch ervice. Thi di patcher then pick the lea t-recentl elected cu tomer queue that ha an active u quer and add that u quer to the proce queue. The u reque t in each proce queue are handled in order ( DF) the next availa le proce thread. All in all, thi threetiered queuing tem en ure that the proce or ta u while al o keeping an one cu tomer or reque t from getting more than it fair hare of proce ing power. Ma ter-worker cheduling The fairne enforcement approach de cri ed a ove take place within each worker. ut to maximize throughput we al o need to take into account how ma ter a ign work to worker. A noted a ove, KD i an HA tem in which each hard i maintained on two different worker machine (if po i le in two different rack ). o two worker have acce to the data needed for an com ination of device and time- lice. The ma ter ha to decide which worker to u e for each u quer to en ure fa te t overall proce ing of a given quer. The ma ter decide which worker for each u quer will en ure fa te t proce ing of the overall quer. directed to the fa t ox. electing the fa ter worker i complicated the fact that worker run on heterogeneou hardware. ome oxe are eef 4U mon ter and ome are not. In practice, we ee that ome hardware com ination are three to four time more efficient than other. A quer i n t fini hed until all of it u reque t are fini hed, o if half of a given quer u querie are ent to a fa t ox (2x fa ter) and half to a low ox, the overall quer time will e 2x lower than if 75% of the u querie are One approach to achieving the fa te t aggregate throughput for all querie would e to exhau tivel weight each worker o that the ma ter can e mart a out di patching to en ure even load acro worker. ut in practice we ve found that we can do an excellent jo of keeping worker alanced having each ma ter keep local track of u querie pending on each worker and prioritizing worker with the fewe t out tanding u querie. The awe ome thing a out thi i that ever ma ter i keeping track on it own: there no glo al knowledge needed, ju t local knowledge. A more u querie hit a ox, the time u querie take increa e, o the chance of Metrics for Microservices - 4

5 future u querie going to thi ox goe down. We throw out all of the complication of haring tate acro ma ter and having to taticall weight worker, ut we till get the real time alancing we want. The readout at right how an example of thi alancing at work:.88 i 4x fa ter than.187. oth have the ame num er of out tanding u querie, ut.88 i handling 4x the volume. Not ad for a greed tem with zero hared knowledge. Think thi i fa cinating tuff? We re hiring! Ready for more information? Please us at info@kentik.com or visit us at Reprinted from the Kentik blog at Kentik Technologies. All rights reserved.