ProdDiagNode - Version: 1. Production Diagnostics for Node Applications

Transcription

1 ProdDiagNode - Version: 1 Production Diagnostics for Node Applications

2 Production Diagnostics for Node Applications ProdDiagNode - Version: 1 2 days Course Description: Node.js, the popular cross-platform JavaScript server framework, has grown in popularity and scale to the point where very challenging, non-trivial, production-only issues pop up in complex deployment environments. Although Node.js is equipped with the built-in V8 profiler, it is limited in scope to CPU time, which doesn t account for the majority of Node.js performance issues. Also, profiling in production environments requires non-invasive tools with a low overhead, which do not require an application restart for the sake of profiling. In this workshop, we will learn, understand, and experiment with Node.js profiling tools designed for production use. We will start from a general overview of performance goals and metrics on Linux systems, using the popular USE (Utilization, Saturation, Errors) method. We will build a simple checklist for verifying Node.js application performance, and finding the area to focus on in a closer investigation. Then, we will experiment with the built-in V8 profiler, and learn how to analyze its log output to zoom-in on CPU hot-spots in the V8 runtime, your JavaScript code, or external modules. We will also explore the Linux perf multi-tool, which can perform general-purpose CPU profiling, and how to get the Node.js runtime to generate symbolic information required to resolve stack traces emitted by perf. We will visualize stack traces using flame graphs, and understand where the CPU bottlenecks lie. We will also look at core dump analysis for crashes and memory leaks, and identify heap objects using the llnode plugin for lldb. In the second half of this workshop, we will talk about more complicated scenarios: diagnosing intermittent errors when connecting to a database or running queries, monitoring system I/O load, understanding the reasons for excessive V8 garbage collection, figuring out why your event loop thread is blocked, and more. Some of these tasks can be approached using perf, but thers require a bleeding-edge

3 technology -- BPF and BCC. BPF is a new Linux kernel technology that enables lowoverhead, super-efficient production monitoring and tracing tools. We will use several of the tools and understand how to build our own tools as necessary, in another series of hands-on labs. Finally, we talk about Node.js garbage collection and memory leaks, and how to identify memory leaks using heap snaphots and Chrome Developer Tools. Intended audience: Node.js developers, system administrators, and production engineers responsible for Node.js applications on Linux Prerequisites: Familiarity with Node.js developmen Basic Linux administration skills Objectives: Identify bottlenecks in Node.js applications Profile Node.js applications Analyze Node.js core dumps Investigate Node.js memory leaks Topics: º The USE method º Node.js observability tools

4 º Sampling vs. tracing º CPU profiling with V8 º LAB: Node profiling with V8 [node-prof.md] º Linux perf and CPU sampling º Visualizing CPU profiles with flame graphs º LAB: Node profiling with perf [perf.md#task-5-profiling-nodejs-code] º LAB: Node leaky slowdown [node-slowdown.md]

5 º Linux core dumps º Generating core dumps on crash º Generating core dumps with gcore º Node.js core dump analysis with llnode º LAB: Node core dump analysis with llnode [node-coredump.md] º Introduction to BPF º BPF scenarios and the BCC toolkit

6 º USDT probes º LAB: Node and USDT probes [bpf-usdt.md#task-1, #task-2] º LAB: Node MySQL slow queries [bpf-nodemysql.md] º Blocked-time analysis º LAB: Node blocked time analysis [bpf-nodeblocked.md] º LAB: Node file opens [bpf-nodeopens.md] º LAB: Node slow HTTP requests [node-slowhttp.md]

7 º LAB: Node slow DNS queries [node-slowdns.md] º Node.js garbage collection º Heap inspection from Node.js core dumps º Heap snapshots º LAB: Node garbage collections [bpf-nodegc.md] º LAB: Node memory leak analysis [node-memleak.md]