QUEUES YOUR QUERY WAITS IN JOSH SNYDER

Transcription

1 QUEUES YOUR QUERY WAITS IN JOSH SNYDER

2 SO MUCH QUEUING SO LITTLE TIME JOSH SNYDER

3 ESCALATOR ETIQUETTE

4 ESCALATING UPHEAVAL Why would they do such a thing?

5 Latency Throughput units time time -1 measures smallest sliver of work lim n 1 largest sample of work lim n

6 HYPOTHETICAL ESCALATOR ANALYSIS Person standing requires 1 stair; 24 seconds So: 24 stair-seconds Person walking requires 12 seconds To break even, walkers must be spaced 2 stairs apart

7 WHAT'S TO COME two "favorite" tools: iostat and loadavg layers and layers of latency managing multi-tenancy load (un)balancing

8 IOSTAT Presents disk I/O statistics Reads /proc/diskstats (Linux)

9 IOSTAT: AN EXAMPLE reads r_sects r_ms t_act t= t= Δ / human / second MB/s

10 r_await: average time each I/O waited ms spent reads / ms / op

11 avgrq-sz: mean sectors per I/O sectors reads / sectors / op

12 svctm: non-idle time (%util) / #OPS ms / second (94.6 %util) reads / ms / OP

13 Device: r/s rmb/s avgrq-sz avgqu-sz r_await svctm %util vda

14 LOAD AVERAGE Collected by the scheduler Based on process states

15 PROCESS STATES A process/thread/task is either: runnable on a CPU starved of CPU waiting for something

16 BEING RUNNABLE int i = 0; while(1) { i++; }

17 CPU STARVATION Possible reasons: no CPU is available task is (temporarily) assigned to a CPU with other work to do a bug in the scheduler (see "A Decade of Wasted Cores")

18 MEASURING CPU STARVATION $ awk '/^cpu/ { printf "%s %.9fs\n", $1, $9 / 1e9 }' /proc/schedstat cpu s cpu s $ awk '{ printf "%.9fs\n", $2 / 1e9 }' /proc/$pid/schedstat s Formats documented in Documentation/scheduler/sched-stats.txt

19

20 Resource-byresource analysis USE method

21 WAITING VOLUNTARILY accept() recv() sleep() futex() waitpid() etc... a new network connection data on a socket a timer a memory address (lock) a process

22 SLEEPING INVOLUNTARILY pkill -STOP mysqld

23 PROCESS STATES RUNNING (R) UNINTERRUPTIBLE (D) INTERRUPTIBLE (S) STOPPED (T) ZOMBIE (Z) misnomer: runnable process waiting for disk waiting for something else (forced to) wait for SIGCONT waiting for parent to waitpid() See include/linux/sched.h for gory details

24 LOAD AVERAGE Instantaneous load: TASK_RUNNING (R) + TASK_UNINTERRUPTIBLE (D) sampled every 5 seconds into an exponentially weighted moving average See: include/linux/sched.h kernel/sched/loadavg.c

25 WHAT'S BETTER THAN A LOAD AVERAGE? for CPU: runqueue latency for disk: iostat avgqu-sz (per disk) delayacct_blkio_ticks (per task)

26 DELAYACCT_BLKIO_TICKS How long a process spent in the D state, in hundredths of a second: $ awk '{ print $42 / 100 }' < /proc/$pid/stat 19.68

27

28 WORKLOAD DEPENDENCE (1) Workload: def random_reader(): while True: do_random_read() threaded(random_reader, 500).start() Resources: 32 cores SSD with maximum performance at QD=16-32

29 WORKLOAD DEPENDENCE (2) Compare this workload: def locked_random_reader(semaphore): while True: with semaphore: do_random_read() max_ios = 32 semaphore = Semaphore(max_ios) threaded(lambda: random_reader(semaphore), 500).start()

30 WORKLOAD DEPENDENCE: LESSONS Changes in workload will change both bad stats (load) and good ones (delayacct_blkio_ticks) Locks are a form of queueing!

31 QUESTIONS?

32 SIMPLIFIED MYSQL EXAMPLE 1. Query packet arrives at NIC 2. Kernel adds packet to socket queue; wakes recv()'ing MySQL thread (S R) 3. Buffer pool lookup: MISS! (waited for locks, R S R) 4. Read pages from disk (R D R) 5. Big result; send result to client (R S R) 6. Wait for client: recv() (R S) 7. GOTO 1

33 "SIMPLIFIED" IS A KEY WORD Did packet processing happen due to interrupt, or polling? How hot are the CPU caches Query passed through bunches of MySQL events_stages etc...

34 LATENCY ANALYSIS IS FRACTALLY COMPLEX! CPUs (and everything else) are abstractions that hide complexity: is it throttling? how many cycles did I stall due to memory access? how many cycles did I stall due to lack of resources in the processor?

35 BUT WE DO IT ANYWAY! CPU time is still a useful metric, even though we take it with a grain of salt!

36 COLLECTING LATENCY INFORMATION Two methods: 1. Timing 2. Sampling (Little's law!)

37 COLLECTING TIMINGS T: Accumulator t: current time T += tend - tstart

38 INSIST ON VDSO TIMEKEEPING vdso: virtual dynamic shared object vdso timekeeping takes 25-45ns in a tight loop non-vdso timekeeping takes ~4x longer bad $ strace -qq -e clock_gettime date > /dev/null clock_gettime(clock_realtime, { , 0}) = 0 good $ strace -qq -e clock_gettime date > /dev/null

39 LITTLE'S LAW In a stable system: L = λw L (mean dwelling customers) λ (mean arrival rate) W (mean dwell time)

40 LITTLE'S LAW APPLIED TO MYSQL Over 1 second: 1000 query threads (~Threads_running sampled) 1e5 Questions (SHOW STATUS LIKE 'Questions') L / λ = W So: 1000 / (1e5 / sec) = 10 ms per query

41 Over the same period the application records 23 outstanding queries (on average) 23 / (1e5 / sec) = 23 ms per query = 13 ms of unaccounted-for time (on average)

42 PROBLEMS ABOUND We now know an average about our queries in general.

43 INTERLUDE: WHY NOT HISTOGRAMS A useful histogram requires ~ counters An average requires 2 (+1 for variance). We can track more averages than we can ever as histograms.

44 QUESTIONS?

45 A TALE OF TWO TENANTS Service " shpics" with N workers and two paths: cache hit (1 ms) cache miss ( ms)

46 AN "ANSWER" TO BAD AVERAGES Track cache hit/miss time (mean + variance) separately Track time-in-queue separately from work time

47 MULTI-TENANCY if misses get too slow, hits will wait two "tenants", one blocking the other

48 FAIRNESS Queue time is useful, but not in isolation! If a 100 ms RPC waits 3 ms: no big deal If a 100 µs RPC waits 3 ms: alarm bells!

49 SLOWDOWN S = (queued time) / (working time) worthwhile whenever a human is waiting cf. express lanes in grocery stores OLTP vs. batch workloads

50

51

52 CONCURRENCY LIMITING shpics service: limit misses to 90% of workers drop requests above 90% single pool of servers; single deployment

53 ALL DATASTORES ARE MULTI-TENANT! query threads compete with each other batching and coalescing work background work background threads compete with query threads backups are background work

54 EXAMPLE: MYSQL BACKUPS Pipeline: read compress send read from disk (ionice(1); CFQ IOPRIO_CLASS_IDLE) compress (chrt(1); SCHED_IDLE) send over network (prio qdisc; SOL_PRIORITY)

55 CLASSFUL SCHEDULING work is divided into classes if high-class work exists, low-class work waits nice(1) is NOT classful (timeslices)

56 EXAMPLE: CASSANDRA COMPACTION Goal: maximal compaction; minimal disruption don't pick a rate a priori! SCHED_IDLE possible starvation solution: cpuset

57 HERACLES From Google Colocated batch and latency-sensitive tasks Per-resource analysis

58 SO MUCH MORE! token buckets cgroups qdiscs (codel)

59 QUESTIONS?

60 LOAD BALANCING how are requests allocated to backends? central queue minimizes queued time under unrealistic assumptions See, in general, ch 24 of "Performance Modeling and Design of Computer Systems"

61 BAD LOAD BALANCING random round-robin (slightly better)

62 BETTER LOAD BALANCING join-shortest-queue least-work-left TAGS (for "practical" workloads)

63 TAGS (PREPARE YOUR MIND) throws away work! unbalances load! fairness over throughput

64 TAGS (KEY CONCEPTS) Non-preemptible idempotent jobs Large variance in job size Unwilling (unable) to make predictions Slowdown metric (covered earlier) Server expansion requirement

65 TAGS Allow jobs to run a limited amount of time Kill and requeue jobs that run too long

66 LOAD UNBALANCING

67 (FINAL) QUESTIONS?