Mochi: Visual Log-Analysis Based Tools for Debugging Hadoop

Transcription

1 Mochi: Visual Log-Analysis Based Tools for Debugging Hadoop Jiaqi Tan Xinghao Pan, Soila Kavulya, Rajeev Gandhi, Priya Narasimhan PARALLEL DATA LABORATORY Carnegie Mellon University

2 Motivation Debugging cloud computing programs is hard Involves both computation and data movement Large-scale, distributed, many interdependencies Focus: debugging Map programs Current tools (traditional debuggers) lacking Too fine-grained, too much information e.g., jstack, jprof Do not show higher-level Map abstractions i.e., information not at level of maps, reduces Need to expose all factors affecting Map program performance 2

3 Architecture of Hadoop Runtime and Distributed Filesystem (HDFS) Master/Slave: one Master, many Slaves Runs arbitrary Map user code Daemons natively generate activity logs Master node JobTracker NameNode Legend Cluster nodes Runtime daemon Distrib. FS daemon TaskTracker DataNode TaskTracker DataNode TaskTracker DataNode Slave nodes 3

4 Log analysis Mochi: Approach Extract views of node-local execution: SALSA [USENIX WASL 08] Correlate execution across nodes: distributed control-flow, distributed data-flow Correlate execution and filesystem layers: Job-centric Data-flow (JCDF) Visualization of behavior Show behavior along three dimensions: Time, space, and (data) volume 4

5 Goals Extract program behavior at Map level of abstraction Factors affecting program performance Aspects of framework not visible to user code e.g. task scheduling, data distribution Expose different perspectives of performance Present different levels of aggregation e.g. time-aggregates, space-aggregates Remain transparent to Hadoop No added instrumentation, work in production environments 5

6 Outline State-machine extraction from logs Abstraction of Map Execution: Job-centric Data-flow (JCDF) Visualization Case-studies Future Work 6

7 State-machine Extraction Data-flow view: transfer of data to other nodes [t] Launch Map Task : Map : [t] Copy Map outputss : : [t] Map Task Done sk s Map outputs to tasks on this or other nodes TaskTracker Log Mochi: Piecing together SALSA states for conjoined Records events for all Maps and Control tasks Flow+Data on Flow its node view of Map Control-flow view: state orders, durations [t] Launch Task : : [t] is idling, waiting for Map outputs : : [t] Repeat until all Map outputs copied d [t] Start Copy (of completed Map output) : : [t] Finish Copy [t] Merge Copy : can be interleaved Incoming Map outputs for this task Idle Copy Merge Copy 7

8 Mochi SALSA Mochi: Log Analysis Mochi: Visualization Execution (Map) layer state-machine view (per-node) Distributed FileSystem (HDFS) state-machine view (per-node) Conjoined Control-flow + Data-flow: Job-Centric Data-Flow (JCDF) (across all nodes) Causal flows of data and processing: Realized Execution Paths (REP) Map Input Map Output Dest Host (Towards rs) Input Output Map inputs, outputs grit1206 grit1204 grit1202 grit1203 Shuffles grit1206 grit1204 grit1202 grit1203 Src Host (From Mappers) inputs, outputs grit1202 grit1203 grit1206 grit1204 grit1204 grit1206 grit1203 grit e e e e e e e e e e e e e e e e e e e+09 Prior work New Contributions 8

9 Control-Flows, Data-Flows Control-flow Distributed flow of execution across nodes: Map via Shuffles Distributed data-flow Data paths of Map outputs shuffled to s HDFS data blocks read into and written out of jobs HDFS data block Map output split Input HDFS data block Map Shuffle 9

10 Job-Centric Data-Flows (JCDF) Control+Data Flows = JCDF Correlate data and execution paths in time: co-occurring: Block-reads, maps; Block-writes, reduces Create conjoined causal paths (REP) HDFS data block Map output split Input HDFS data block Map Shuffle 10

11 Realized Execution Paths (REP) JCDF (Job-centric Data-Flows) Large graph of all causal flows Contains all causal flows distributed through system REP Single causal path through execution Extract every REP from JCDF using Depth-First Search HDFS data block Map output split Input HDFS data block Map Shuffle 11

12 Validation: Visualizations Testbed Yahoo! M processor cluster Production environment: no added instrumentation Hadoop with Hadoop-on-Demand Workloads: Real CMU user workloads: Matrix-Vector Multiplication [CMU-ML ] Benchmarks: SleepJob (No-op), Sort Processing C++ Log parsing + GNU R graph generation 12

13 Swimlanes (Space-Time) Task-execution in time, across nodes Each task gets a line for its duration Shows where Hadoop job and nodes spend their time Per-task Per-task JT_Map JT_ Task durations (Matrix-Vector Multiplication): Per-node Time/s Task durations (Matrix-Vector Multiplication: 49 hosts): All nodes JT_Map JT_ Time/s 13

14 MIROS MIROS (Map Inputs, Outputs, Shuffles) Map Input Map Output Map inputs, outputs grit1206 grit1204 grit1202 grit e e e e e e e+09 Aggregates data volumes across: All tasks per node Entire job Shows skewed data flows, bottlenecks Dest Host (Towards rs) Input Output Shuffles grit1206 grit1204 grit1202 grit1203 Src Host (From Mappers) inputs, outputs grit1202 grit1203 grit1206 grit1204 grit1204 grit1206 grit1203 grit e e e e e e e e e e e e

15 Realized Execution Paths (REP) Per-flow Volume- Duration correlation Flows clustered for scalable visualization For each stage in flow, shows: Duration spent Input+output volumes Compares time taken vs. volume processed 15

16 Opportunities for Optimization User M45 workload: Matrix-Vector Multiplication 5-node cluster Before: Some nodes idle during reduce Adjusted # reducers After: All nodes had reducers; 15% faster Before After 16

17 Debugging: Delayed Socket Creation M45 workload: Noop Sleep job Each Map/ sleeps for 100 ms Tasks had unusually long durations: exactly 3 minutes Identified to be bad routing interaction Disabled Java IPv6 Before After 17

18 Characterizing Jobs using REPs Each REP: instantiation of the generic execution Plot of Sort workload: Significant time spent waiting on shuffles 18

19 Next Steps Collaboration with Yahoo! we are implementing Mochi for the Hadoop Chukwa project Web-based visualization widgets for Hadoop Infrastructure Care Center Swimlanes currently available in Chukwa 0.2 (CHUKWA-279) 19

20 Conclusion Extracting program behavior with log analysis Distributed Control+Data-Flow Job-centric Data-flows (JCDF), Realized Execution Paths (REP) Visualizations Swimlanes Task execution in time and space MIROS Aggregate data-flows across cluster REP Volume-Duration correlation Case-studies Performance debugging and optimization Job characterization: Bottleneck stages 20

21 References [CMU-ML ] U. Kang, C. Tsourakakis, A. P. Appel, C. Faloutsos, J. Leskovec. HADI: Fast Diameter Estimation and Mining in Massive Graphs with Hadoop. CMU ML Tech Report, CMU-ML , [USENIX WASL 08] J. Tan, X. Pan, S. Kavulya, R. Gandhi, P. Narasimhan. SALSA: Analyzing Logs as StAte Machines. First USENIX Workshop on Analysis of System Logs (WASL), San Diego, CA, Dec Other work: X. Pan, S. Kavulya, J. Tan, R. Gandhi, P. Narasimhan. Ganesha: Black-Box Diagnosis for Map Systems. Second Workshop on Hot Topics in Measurement & Modeling of Computer Systems (HotMetrics), Seattle, WA, Jun