Parallel Build Visualization Diagnosing and Troubleshooting Common Pitfalls of Parallel Builds

Transcription

1 Parallel Build Visualization Diagnosing and Troubleshooting Common Pitfalls of Parallel Builds John Graham-Cumming Chief Scientist Electric Cloud, Inc. February, 2006

2 Contents Parallel Build Visualization... 3 Background: The Serial Assumption... 4 Background: Amdahl's Law... 5 Diagnosis and Repair of Common Parallel Build Mistakes... 9 Parallel Problem #1: The shared debugging log file... 9 Parallel Problem #2: The very long link Parallel Problem #3: The sub-make that blocks progress Parallel Problem #4: The slow start Parallel Problem #5: Shared intermediate files Conclusion About Electric Cloud Parallel Build Visualization Page 2 Electric Cloud, Inc. All rights reserved.

3 Parallel Build Visualization Software builds are widely regarded as the next opportunity for real productivity improvements in the software development lifecycle. Code complexity has increased for virtually any sort of application, which typically contributes to longer and longer software builds. At the same time, time-to-market pressures will continue to increase in competitive markets, which means that long builds are no longer tolerable. Many build teams are turning to parallel builds as the solution to this problem. Hardware has become much cheaper and software to parallelize builds (both commercial and open source) is available. So as build managers come under pressure to speed up builds, the simplest response is to build a cluster of machines and parallelize the build. But in doing so the build manager creates a build debugging problem. Parallel builds almost never provide the theoretical maximum speedup possible, and finding out why is hard. It's one thing to attempt to understand the performance of a software build run serially on a single processor; it's quite another when the build is run in parallel. To rapidly understand the performance bottlenecks in a parallel build requires flexible visualization tools; looking through log files with existing text-based tools is time-consuming and next-to-impossible for a real build. In this paper, we will identify several of the most common performance problems with parallel builds (in this case, Make-based builds), as well as techniques to debug them. To illustrate these points, we will use ElectricInsight, a new graphical build visualization tool that displays the innards of a parallel build in graphical form and shows the relationships between build elements (jobs, rules, etc.) as well as the complex dependencies among them. This is key to understanding how to effectively debug and optimize a parallel build. Parallel Build Visualization Page 3 Electric Cloud, Inc. All rights reserved.

4 Background: The Serial Assumption One common problem in parallelizing a build is that most existing builds have been implemented with the Serial Assumption: the expectation that jobs are going to be handled one at a time in some well defined sequential order. Here's an example Makefile to illustrate the serial assumption (foo-header and bar-header already exist before running any builds): all: foo bar foo: foo-header build foo bar: bar-header bar1 bar2 build bar bar1: build bar1 bar2: build bar2 A full build using this Makefile will produce the following output: build foo build bar1 build bar2 build bar And it will always produce that output. Internally Make builds a tree of the work to be done and then proceeds to walk the tree in a depth-first manner: foo foo-header all bar-header bar bar1 bar2 The tree is built starting from the first target in the Makefile (in this case all) and then working from left-to-right across the prerequisites for each target. Hence the first target added to the tree after all is foo (and then its prerequisite foo-header). This left-to-right building of the tree and depth-first execution is the Serial Assumption. When the same build is run in parallel only the actual dependencies matter, the order in which the tree was built or walked is irrelevant. For example, there's no reason why bar2 can't be built before bar1 and no reason why either of them can't be built before foo. As long as all dependencies are specified explicitly in the Makefile, the Serial Assumption poses no problems. Unfortunately, it's so ingrained in the users of Make that, in practice, most Makefiles have fallen into assuming at least the left-to-right build order. If those Makefiles actually rely on that order, or there are side effects within rules, the parallel build will either be slow or break. Parallel Build Visualization Page 4 Electric Cloud, Inc. All rights reserved.

5 Background: Amdahl's Law The second problem build managers face in parallelizing builds is over-inflated expectations. It's common, but incorrect, to believe that taking a build that lasts H hours and running it on P processors will result in a build that is roughly H/P hours long. This is a common expectation because builds naturally contain lots of small pieces of work (compiles, etc.) that are independent of each other; in the example in the previous section bar1, bar2 and foo were all independent and could run on three processors in parallel. For real builds the number of independent build steps is much higher allowing more processors to be used. Figure 1 shows a real serial build visualized using ElectricInsight. The blue bars are individual build steps (most of which are very short; the areas of black are actually hundreds of tiny jobs steps packed together) with a few large steps through the build. Figure 1: A serial build Figure 2 shows an idealized parallel build visualized with ElectricInsight. The build has been spread evenly across 10 processors running in a cluster. It uses the cluster at close to 100% utilization on all cluster nodes and hence the speed up will be approximately H/P (in this case P is 10 meaning the build was sped up almost 10x). Parallel Build Visualization Page 5 Electric Cloud, Inc. All rights reserved.

6 Figure 2: An evenly spread parallel build Reality is usually quite different. To determine how much parallelism is possible in a build it's necessary to identify the longest part of the build that cannot be parallelized. The simplest example is a single long step, such as a long link: by definition this single step cannot be decomposed into smaller pieces and run in parallel. The build cannot run faster than that single step requires. In Figure 1 there's a long job step near the beginning; highlighting it in ElectricInsight reveals that it lasts 639 seconds. The total build time is also shown in the ElectricInsight display at 4380 seconds. The statistics for the long job step and total time can be seen in Figure 3. Hence, no matter how many processors are used to parallelize the build it can never be shorter than 639 seconds. That means that the greatest speedup offered by parallelizing this build is 4380/639 or 6.9x. And that's probably an overestimate because the longest job step may be preceded in the serial build by jobs on which it depends, making the longest part of the build greater than 639 seconds and hence the maximum speedup even smaller. Parallel Build Visualization Page 6 Electric Cloud, Inc. All rights reserved.

7 Figure 3: A long job step The maximum possible speedup in any parallel application is governed by Amdahl's Law, which states that if F is the fraction of the application that cannot be parallelized (in a build that is the longest serialized part of the build) and P processors are used than the maximum speedup possible is 1 1 F F P Figure 4: Amdahl's Law In the example above, F is 639/4380 or 0.15, 1 F is Graphing the number of processors against the speedup shows the effect of Amdahl's law. Figure 5 shows Amdahl's Law applied to the build in Figure 1 assuming that the longest serial part is 639 seconds. It's clear that the speedup tails off as the number of nodes is added to the cluster and that somewhere between 10 and 20 nodes are worth investing in to parallelize this particular build. Parallel Build Visualization Page 7 Electric Cloud, Inc. All rights reserved.

8 Figure 5: Amdahl's Law with F = 0.15 Having covered the theoretical background, let's now take a look at some specific example problems. Parallel Build Visualization Page 8 Electric Cloud, Inc. All rights reserved.

9 Diagnosis and Repair of Common Parallel Build Mistakes Parallel Problem #1: The shared debugging log file A common technique in Make-based projects is to write the output of each rule to a log file that can be examined in the event that the build fails. For example, the rule to build an object file might look something like this: LOG_FILE := error.log.phony: all all: foo.o %.o: -c -Wall -Werror -o $@ $< &> $(LOG_FILE) In the event that a particular compilation fails the $(LOG_FILE) will contain just the output from the specific step that failed. That's handy because Make's output can be difficult to interpret in the event of a failure. If this rule was used to build foo.o from foo.c the $(LOG_FILE) (which was set to error.log in the example above) might contain the following if a compilation error occurred: foo.c: In function `main': foo.c:6: warning: control reaches end of non-void function This is obviously very useful when trying to diagnose and correct a build failure. More complex Makefiles might output additional information to the log file to make the debugging even easier. However, this style of Makefile completely breaks when parallelized. Since every compilation will write to the same file (and each compilation truncates the file to start with) a parallel build system will either completely serialize the compilations to ensure that no two rules write to the same file at the same time (in which case there's no difference between a parallel and a serial build) or the build system allows the simultaneous writes and so that content of error.log is unpredictable in the best case it would contain the complete error information from a single error in the build; in the worst case, the contents could be completely corrupted. In ElectricInsight a completely serialized build looks the same as a serial build (see Figure 3 for example), if only part of the build is serialized by a log file then part of the build may execute in parallel and part in serial order (see Figure 6). Parallel Build Visualization Page 9 Electric Cloud, Inc. All rights reserved.

10 Figure 6: A parallel build with a long stretch of serialization To solve this problem, the shared log file needs to be removed. Either unique log file names for each rule should be used, or the log file should be removed entirely. To use unique log file names, we could use $@-$(LOG_FILE) instead of $(LOG_FILE). The error log when building foo.o would be foo.o-error.log. These uniquely named log files could be deleted automatically if the command is successful, to avoid littering the build output. For example, the Makefile could be rewritten: LOG_FILE := error.log UNIQUE = $@-$(LOG_FILE) all: foo.o %.o: -c -Wall -Werror -o $@ $< &> -f $(UNIQUE) The log file only gets deleted if the gcc runs successfully; in the event of a failure a single log file will be found with the relevant compiler output (in this case foo.o-error.log). Eliminating use of the log file is even easier. At first this suggestion may seem contradictory to the earlier assertion that Make's output is hard to interpret in case of failures, but if you have a powerful build analysis tool, this approach becomes very practical. ElectricInsight makes it easy to see the output of a failed job, so that debugging log files are made unnecessary. A failed job shows up in the ElectricInsight UI; clicking on the job brings up detailed information about the job that failed, and all its output. Parallel Build Visualization Page 10 Electric Cloud, Inc. All rights reserved.

11 ElectricInsight makes it easy to see the output of a failed rule such that log files are often unnecessary. A failed rule shows up in the ElectricInsight UI and clicking on the job brings up detailed information about the job that failed, and all its output. Figure 7 shows a failed build with the final, failing job highlighted. Something went wrong while building coretools.ocx. Figure 7: A failure viewed in ElectricInsight Clicking on the failing job brings up ElectricInsight's Job Details window that shows the command executed and the way in which it failed. Other information points to the specific Makefile and line where the failing rule was defined (in this case in Makefile at line 8463). Figure 8: The error output for coretools.ocx Parallel Problem #2: The very long link Another common problem with parallel builds is that a final job step, often a link, consumes Parallel Build Visualization Page 11 Electric Cloud, Inc. All rights reserved.

12 a large percentage of the overall build time. Because the link can only happen after the objects have been created, and because it cannot be parallelized, the link limits the speedup possible with parallelization. Figure 9 shows a parallel build with a final long link step that can utilize only a single node in the parallel build cluster. Figure 9: A long link If the final link is significantly limiting the speedup possible through parallelization, there are a number of possible remedies. Some platforms have linkers that can perform incremental (instead of complete) links. The linker is able to take a partly linked binary and perform partial linking (for example, linking in a single library) before writing a new partially linked binary. By breaking the link up into a number of partial links, the build system will be able to link in parallel with only a small, final link needed to create the complete binary. On systems with no incremental linker, the simplest remedy may be to upgrade the system doing the link. By targeting the link to a specific node in the build cluster that has additional memory and a fast disk the link time may be significantly reduced. Parallel Problem #3: The sub-make that blocks progress Most build systems use the 'recursive Make' style. A top-level Makefile executes Make in a Parallel Build Visualization Page 12 Electric Cloud, Inc. All rights reserved.

13 large number of subdirectories. Each subdirectory is handled by an invocation of Make which reads that directory's Makefile and executes the jobs described by it. Commonly the recursive Make system will change into each subdirectory in turn and execute Make. This can cause non-optimal use of the build cluster. It is unlikely that the subdirectories have to be built in this serial order and doing so prevents a parallel build system from finding the maximum number of jobs to work on. By limiting the number of jobs, the build cluster may be under utilized. For example, Figure 10 shows a recursive Make system where the cluster is not fully utilized for a period. That occurred because one of the subdirectories did not have enough parallelism in its Makefile to fully utilize the cluster. Figure 10: Sub-Make underutilization Electric Cloud's ElectricAccelerator product automatically fixes this problem without Makefile changes by examining the structure of every sub-make and merging the job tree for each sub-make into a single large tree. This large tree contains all the available parallelism across the subdirectories and hence ElectricAccelerator is able to unlock the maximum amount of parallelism and keep the build cluster fully utilized. And ElectricAccelerator's Electric File System ensures that the build results are correct every time. Other parallel build systems (e.g. GNU Make's -j option) will require rewriting the Makefile structure to either remove recursive Makes completely, or allow parallel execution of the sub-makes. Achieving either of these goals can be quite difficult, because you must ensure Parallel Build Visualization Page 13 Electric Cloud, Inc. All rights reserved.

14 that no dependency information is missing. It's common for a Makefile s author to have assumed the serial ordering implicit in the walk from subdirectory to subdirectory; when the same build system is parallelized it is possible that the build will break or produce incorrect results unless care is taken to ensure that dependencies between subdirectories are accurately specified in the Makefile. Parallel Problem #4: The slow start The opposite of Parallel Problem #2 (the long link at the end) is a slow start to the build. A common reason that a build runs serially at the start before becoming parallel is that the build begins with a job that must be completed before any other work can begin. For example, many builds start with a 'make depend' step that builds the dependency information for the entire build before the build is allowed to proceed. Figure 11 shows how a long make depend step at the start of a build appears in ElectricInsight. Figure 11: Doing make depend at the start of a build A long serial build step at the start of the build will limit the maximum speedup available because all but one of the cluster nodes is unused during the job. The Amdahl's Law section above showed how the maximum speedup available in a parallel build is limited by the longest sequence of jobs that must be performed serially. A long serial job at the start of a build can seriously limit the possible speedup when the build is parallelized. Parallel Build Visualization Page 14 Electric Cloud, Inc. All rights reserved.

15 To eliminate a large make depend there are a number of options. Firstly, the make depend itself can be parallelized. Since the output of make depend is typically just a text Makefile, it's possible to run multiple make depends. Each make depend handles dependencies for part of the overall build and the results of each make depend are merged into the main Makefile before the build proceeds. Breaking up make depend means that multiple instances of the depend process can run simultaneously and in parallel. If the build is being run with ElectricAccelerator, the make depend step can, most likely, be completely eliminated. ElectricAccelerator's Electric File System does dependency checking on the fly as the build runs and will detect the dependency information necessary as the build runs. Lastly, if the build being run is a scratch or full build then dependency information from make depend is probably irrelevant and can be ignored. By definition, every object file is created in a full build, so the dependency information gathered by make depend is not needed. The make depend step can be safely removed and the build will be able to run in parallel without the initial delay while make depend completes. Parallel Problem #5: Shared intermediate files A difficult-to-diagnose problem that occurs when a build is parallelized is the reuse of a hidden intermediate file. Some compilers or code generators reuse common files because they make the assumption that every build is serial and that it is safe to use a common file name for some hidden purpose. For example, the Flex lexical analyzer generator (and similar tools) have default names for the files that they generate that are used by other tools (e.g. Bison). If a build contains multiple calls to Flex, the same generated files are created. This is safe in a serial build since the calls to Flex can never interfere with each other. In a parallel build, the use of these default names is dangerous and can result in broken builds and strange errors. For example, the files generated by an instance of Flex might be incorrectly used by an instance of Bison that was expecting files generated by a different Flex invocation. This type of problem is difficult to diagnose. To prevent this from happening, it is vital that generated files have unique names (or unique directories) so that they cannot be incorrectly used by parallelized jobs. Most parallel build systems will produce incorrect or unstable builds if a default file name is reused under the assumption that the build runs serially. ElectricAccelerator's Electric File System will detect misuse of default file names and correct the build, but the implicit serialization caused by this reuse will impact build speed. And ElectricInsight makes it easy first to detect the problem, and then to identify the cause. Conclusion While parallel builds are the optimal way to maximize build speed, they are often difficult to execute effectively, and even more difficult to debug and troubleshoot. Without graphical visualization of the build, tracking down build problems is an arduous, manual task involving examination of literally thousands upon thousands of lines of information from a build output. And understanding build timings when trying to optimize a build is even more difficult. Parallel Build Visualization Page 15 Electric Cloud, Inc. All rights reserved.

16 In this paper, we have explored several parallel build pitfalls and debugging techniques using ElectricInsight as a means to visualize build output and relationships between jobs or rules. This kind of visualization would be impossible without a graphical visualization tool. Graphical build visualization tools like ElectricInsight give build managers and build novices alike instant information about the state of a build and provide vital clues for debugging and optimizing builds. Parallel Build Visualization Page 16 Electric Cloud, Inc. All rights reserved.

17 About Electric Cloud Electric Cloud was founded in 2002 by software technologists tired of seeing software development productivity impacted by the "build bottleneck." Software builds are the last opportunity for real, measurable improvement for most development organizations, and yet are one of the least addressed parts of the lifecycle. Development process improvements and automated tools can only do so much to improve productivity if builds are still limiting your throughput. Electric Cloud breaks the build bottleneck with tools that reduce your build times, make full and incremental builds more accurate, and provide the visibility and agility you need to continuously improve your build environment. The ElectricAccelerator solution uses clusters of inexpensive servers to reduce software build times by as much as 20x. ElectricAccelerator is based on sophisticated patent-pending technology for managing dependencies, resulting in vastly greater scalability than previous parallel build systems. Electric Cloud s new ElectricInsight tool is a breakthrough software build visualization tool that provides job-level detail into parallel builds and provides unprecedented visibility into build results for simplified troubleshooting and performance tuning. It mines the information produced by ElectricAccelerator to provide an easy-to-understand, graphical representation of the build structure for performance analysis. Learn more about Electric Cloud at Electric Cloud, Inc Leghorn Street Mountain View, CA Tel: Fax: Electric Cloud, Electric Make, ElectricAccelerator, and ElectricInsight are trademarks of Electric Cloud, Inc. All other names are used for identification purposes only and are trademarks or registered trademarks of their respective companies. Parallel Build Visualization Page 17 Electric Cloud, Inc. All rights reserved.