CS.7.A.Performance.Challenge

Size: px

Start display at page:

Download "CS.7.A.Performance.Challenge"

Roger Preston
5 years ago
Views:

PA R T I : P R O G R A M M I G I J AVA PA R T I : P R O G R A M M I G I J AVA Computer Science Computer Science The challenge Empirical

Familiar examples ROBERT SEDGEWICK K E V I WAY E http://introcs.cs.princeton.edu CS.7.A.Performance.

Will I be able to use my program to solve a large practical problem?

Whenever any result is sought by its aid, the question will arise By what course of calculation can these results be arrived at by the

1 PA R T I : P R O G R A M M I G I J AVA PA R T I : P R O G R A M M I G I J AVA Computer Science Computer Science The challenge Empirical analysis Mathematical models Doubling method An Interdisciplinary Approach Section 4. Familiar examples ROBERT SEDGEWICK K E V I WAY E CS.7.A.Performance.Challenge The challenge (since the earliest days of computing machines) The challenge (modern version) Q. Will I be able to use my program to solve a large practical problem? As soon as an Analytic Engine exists, it will necessarily guide the future course of the science. Whenever any result is sought by its aid, the question will arise By what course of calculation can these results be arrived at by the machine in the shortest time? Charles Babbage Difference Engine #2 Designed by Charles Babbage, c. 848 Q. How many times do you have to turn the crank? Built by London Science Museum, 99 Q. If not, how might I understand its performance characteristics so as to improve it? Key insight (Knuth 97s). Use the scientific method to understand performance. 3 4

Three reasons to study program performance Will my program finish? When will my program finish? 2. To compare algorithms and implementations.

To develop a basis for understanding the problem and for designing new algorithms Enables new technology. Enables new research. public class Gambler int stake = Integer.

parseInt(args[2]); int wins = ; for (int t = ; t < trials; t++) int cash = stake; while (cash > && cash < goal) if (Math.random() <.

Issue (97s): Too slow to address scientific problems of interest. Success story: Barnes-Hut algorithm uses log steps and enables new research.

limit on available time log problem size ( ) We study several algorithms later in this course. Taking more CS courses? You'll learn dozens of algorithms.

The binary logarithm of a number (written lg ) is the number x satisfying 2x =. Goal: Break down waveform of samples into periodic components.

Success story: FFT algorithm uses log steps and enables new technology. lg Goal: Simulate gravitational interactions among bodies. time.

2 Three reasons to study program performance Will my program finish? When will my program finish? 2. To compare algorithms and implementations. Will this change make my program faster? How can I make my program faster? 3. To develop a basis for understanding the problem and for designing new algorithms Enables new technology. Enables new research. public class Gambler int stake = Integer.parseInt(args[]); int goal = Integer.parseInt(args[]); int trials = Integer.parseInt(args[2]); int wins = ; for (int t = ; t < trials; t++) int cash = stake; while (cash > && cash < goal) if (Math.random() <.5) cash++; else cash--; if (cash == goal) wins++; StdOut.print(wins + " wins of " + trials); -body simulation Brute-force algorithm uses 2 steps per time unit. Issue (97s): Too slow to address scientific problems of interest. Success story: Barnes-Hut algorithm uses log steps and enables new research. Andrew Appel PU '8 senior thesis 2 An algorithm is a method for solving a problem that is suitable for implementation as a computer program. limit on available time log problem size ( ) We study several algorithms later in this course. Taking more CS courses? You'll learn dozens of algorithms. 5 Another algorithm design success story 6 Quick aside: binary logarithms Discrete Fourier transform Def. The binary logarithm of a number (written lg ) is the number x satisfying 2x =. Goal: Break down waveform of samples into periodic components. Applications: digital signal processing, spectroscopy, or log2 Brute-force algorithm uses 2 steps. Issue (95s): Too slow to address commercial applications of interest. Success story: FFT algorithm uses log steps and enables new technology. lg Goal: Simulate gravitational interactions among bodies. time. To predict program behavior An algorithm design success story Q. How many recursive calls for convert()? John Tukey time A. Largest integer less than or equal to lg (written Frequently encountered values public static String convert(int ) if ( == ) return ""; return convert(/2) + ( % 2); 2 2 lg ). approximate value lg log 2 thousand million billion Prove by induction. Details in "sorting and searching" lecture. Fact. The number of bits in the binary representation of is + lg. limit on available time log Fact. Binary logarithms arise in the study of algorithms based on recursively solving problems half the size (divide-and-conquer algorithms), like convert, FFT and Barnes-Hut. problem size ( ) 7 8

3 An algorithmic challenge: 3-sum problem Three-sum implementation Three-sum. Given integers, enumerate the triples that sum to. public class ThreeSum public static int count(int[] a) /* See next slide. */ int[] a = StdIn.readAllInts(); StdOut.println(count(a)); % more 6ints.txt % java ThreeSum < 6ints.txt 3 Q. Can we solve this problem for = million? For simplicity, just count them Applications in computational geometry Find collinear points. Does one polygon fit inside another? Robot motion planning. [a surprisingly long list] 9 "Brute force" algorithm Process all possible triples. Increment counter when sum is. public static int count(int[] a) int = a.length; int cnt = ; for (int i = ; i < ; i++) for (int j = i+; j < ; j++) for (int k = j+; k < ; k++) if (a[i] + a[j] + a[k] == ) cnt++; return cnt; i a[i] Keep i < j < k to avoid processing each triple 6 times triples with i < j < k Q. How much time will this program take for = million? i j k a[i] a[j] a[k] PART I: PROGRAMMIG I JAVA PART I: PROGRAMMIG I JAVA Image sources The challenge Empirical analysis Mathematical models Doubling method Familiar examples CS.7.A.Performance.Challenge CS.7.B.Performance.Empirical

A first step in analyzing running time Empirical analysis Find representative inputs Run experiments Option : Collect actual input data. Option 2: Write a program to generate representative inputs.

uniform(-M, M)); % java Generator 28773-87569 % java Generator -425582-2 594752 6579-4 -483784-8632 -2-69436 - -732636-4 36294-7 Start with a moderate input size.

$f seconds)\n", cnt, now - start); Measure and record running time. Double input size. Repeat. Tabulate and plot results.$

4 A first step in analyzing running time Empirical analysis Find representative inputs Run experiments Option : Collect actual input data. Option 2: Write a program to generate representative inputs. public class Generator // Generate integers in [-M, M) int M = Integer.parseInt(args[]); int = Integer.parseInt(args[]); for (int i = ; i < ; i++) StdOut.println(StdRandom.uniform(-M, M)); % java Generator % java Generator Start with a moderate input size. Replace println() in ThreeSum with this code. double start = System.currentTimeMillis() /.; int cnt = count(a); double now = System.currentTimeMillis() /.; StdOut.printf("%d (%.f seconds)\n", cnt, now - start); Measure and record running time. Double input size. Repeat. Tabulate and plot results. Tabulate and plot results 25 Run experiments 2 % java Generator java ThreeSum 59 ( seconds) time (seconds) Input generator for ThreeSum Measure running time % java Generator 2 java ThreeSum 522 (4 seconds) % java Generator 4 java ThreeSum 3992 (3 seconds) not much chance of a 3-sum % java Generator 8 java ThreeSum 393 (248 seconds) good chance of a 3-sum time (seconds) problem size ( ) 3 Aside: experimentation in CS 4 Data analysis is virtually free, particularly by comparison with other sciences. Curve fitting Plot on log-log scale. If points are on a straight line (often the case), a one million experiments power law holds a curve of the form a b fits. The exponent b is the slope of the line. Solve for a with the data. one experiment Chemistry log-log plot lg T 4.84 T lg Do the math x-intercept (use lg in anticipation of next step) 8 lgt = lga + 3lg log time (lg T) Biology Physics Computer Science 5 straight line of slope 3 2 Bottom line. o excuse for not running experiments to understand costs = a 83 substitute values from experiment a = 4.84 T = 4.84 solve for a 3 substitute 3 log problem size (lg ) 5 T = equation for straight line of slope 3 raise 2 to a power of both sides a 3 a curve that fits the data? 6

Prediction and verification Another hypothesis Hypothesis. Running time of ThreeSum is 4.84 3. Prediction.

2 6 3 seconds % java Generator 6 java ThreeSum 393 (985 seconds) VAX /78 2 4 8 4.84 3 seconds Q.

2s:,+ times faster Macbook Air 25 2 time (seconds) 2 4 4 3 8 248 2 4 8 Hypothesis.

7 8 PART I: PROGRAMMIG I JAVA PART I: PROGRAMMIG I JAVA Image sources http://commons.wikimedia.

5 Prediction and verification Another hypothesis Hypothesis. Running time of ThreeSum is Prediction. Running time for = 6, will be 982 seconds. about half an hour 97s 2 time (seconds) (estimated ) seconds % java Generator 6 java ThreeSum 393 (985 seconds) VAX / seconds Q. How much time will this program take for = million? A. 484 million seconds (more than 5 years). 2s:,+ times faster Macbook Air 25 2 time (seconds) Hypothesis. Running times on different computers differ by only a constant factor. 7 8 PART I: PROGRAMMIG I JAVA PART I: PROGRAMMIG I JAVA Image sources The challenge Empirical analysis Mathematical models Doubling method Familiar examples CS.7.B.Performance.Empirical CS.7.C.Performance.Math

Mathematical models for running time Q. Can we write down an accurate formula for the running time of a computer program? A. (Prevailing wisdom, 96s) o, too complicated. A. (D. E.

6 Mathematical models for running time Q. Can we write down an accurate formula for the running time of a computer program? A. (Prevailing wisdom, 96s) o, too complicated. A. (D. E. Knuth, 968 present) Yes! Determine the set of operations. Find the cost of each operation (depends on computer and system software). Find the frequency of execution of each operation (depends on algorithm and inputs). Total running time: sum of cost frequency for all operations. Warmup: -sum public static int count(int[] a) int = a.length; int cnt = ; for (int i = ; i < ; i++) if (a[i] == ) cnt++; return cnt; Q. Formula for total running time? ote that frequency of increments depends on input. operation cost frequency function call/return 2 ns variable declaration 2 ns 2 assignment ns 2 less than compare /2 ns + equal to compare /2 ns array access /2 ns increment /2 ns between and 2 representative estimates (with some poetic license); knowing exact values may require study and experimentation. Don Knuth 974 Turing Award 2 A. c nanoseconds, where c is between 2 and 2.5, depending on input. 22 Warmup: 2-sum Simplifying the calculations public static int count(int[] a) int = a.length; int cnt = ; for (int i = ; i < ; i++) for (int j = i+; j < ; j++) if (a[i] + a[j] == ) cnt++; return cnt; Q. Formula for total running time? operation cost frequency function call/return 2 ns variable declaration 2 ns + 2 assignment ns + 2 less than compare /2 ns ( + ) ( + 2)/2 equal to compare /2 ns ( )/2 array access /2 ns ( ) increment /2 ns between ( + )/2 and 2 exact counts tedious to derive # i < j = 2 ( ) = 2 Tilde notation Use only the fastest-growing term. Ignore the slower-growing terms. Def. f( ) ~ g( ) means f( )/g( ) as Ex. 5/ /4 + 53/2,253,276.5 for =, Rationale When is large, ignored terms are negligible. When is small, everything is negligible. ~ 5/4 2,25, for =,, within.3% Q. Formula for 2-sum running time when count is not large (typical case)? A. c 2 + c2 + c3 nanoseconds, where [complicated definitions]. A. ~ 5/4 2 nanoseconds. eliminate dependence on input 23 24

Mathematical model for 3-sum Context public static int count(int[] a) int = a.

call/return 2 ns variable declaration 2 ns ~ assignment ns ~ less than compare /2 ns ~ 3 /6 equal to compare /2 ns ~ 3 /6 array access /2 ns ~ 3 /2 increment /2 ns ~ 3 /6 Q.

Hypothesize a model consistent with observations. Predict events using the hypothesis. Verify the predictions by making further observations.

7 Mathematical model for 3-sum Context public static int count(int[] a) int = a.length; int cnt = ; for (int i = ; i < ; i++) for (int j = i+; j < ; j++) for (int k = j+; k < ; k++) if (a[i] + a[j] + a[k] == ) cnt++; return cnt; # i < j < k = operation cost frequency function call/return 2 ns variable declaration 2 ns ~ assignment ns ~ less than compare /2 ns ~ 3 /6 equal to compare /2 ns ~ 3 /6 array access /2 ns ~ 3 /2 increment /2 ns ~ 3 /6 Q. Formula for total running time when return value is not large (typical case)? 3 ( )( 2) = assumes count is not large Scientific method Observe some feature of the natural world. Hypothesize a model consistent with observations. Predict events using the hypothesis. Verify the predictions by making further observations. Validate by refining until hypothesis and observations agree. Empirical analysis of programs "Feature of natural world" is time taken by a program on a computer. Fit a curve to experimental data to get a formula for running time as a function of. Useful for predicting, but not explaining. Francis Bacon René Descartes John Stuart Mill Mathematical analysis of algorithms Analyze algorithm to develop a formula for running time as a function of. Useful for predicting and explaining. Might involve advanced mathematics. Applies to any computer. A. ~ 3 /2 nanoseconds. matches empirical hypothesis Good news. Mathematical models are easier to formulate in CS than in other sciences PART I: PROGRAMMIG I JAVA PART I: PROGRAMMIG I JAVA Image sources The challenge Empirical analysis Mathematical models Doubling method Familiar examples CS.7.C.Performance.Math CS.7.D.Performance.Doubling

Key questions and answers Doubling method Q. Is the running time of my program ~ a b seconds? Hypothesis. The running time of my program is T ~ a b. no need to calculate a (!) A.

A. Programs are built from simple constructs (examples to follow). A. Real-world data is also often simply structured. A. Deep connections exist between such models and a wide variety of discrete structures (including some programs).

Predict by extrapolation: multiply by 2 b to estimate T2 and repeat. 3-sum example Bottom line. It is often easy to meet the challenge of predicting performance. T T/T/2.5 2 4 8 4 3 7.

If a function f() ~ ag( ) we say that g( ) is the order of growth of the function. Hypothesis. The order of growth of the running time of my program is b (log ) c.

Known to be true for many, many programs with simple and similar structure.

8 Key questions and answers Doubling method Q. Is the running time of my program ~ a b seconds? Hypothesis. The running time of my program is T ~ a b. no need to calculate a (!) A. Yes, there's good chance of that. Might also have a (lg ) c factor. Consequence. As increases, T2/T approaches 2 b. Proof: ( ) = Q. How do you know? A. Computer scientists have applied such models for decades to many, many specific algorithms and applications. A. Programs are built from simple constructs (examples to follow). A. Real-world data is also often simply structured. A. Deep connections exist between such models and a wide variety of discrete structures (including some programs). 29 Doubling method Start with a moderate size. Measure and record running time. Double size. Repeat while you can afford it. Verify that ratios of running times approach 2 b. Predict by extrapolation: multiply by 2 b to estimate T2 and repeat. 3-sum example Bottom line. It is often easy to meet the challenge of predicting performance. T T/T/ = = = math model says running time should be a = 8 3 Order of growth Order of growth Def. If a function f() ~ ag( ) we say that g( ) is the order of growth of the function. Hypothesis. The order of growth of the running time of my program is b (log ) c. log instead of lg since constant base not relevant Hypothesis. Order of growth is a property of the algorithm, not the computer or the system. Evidence. Known to be true for many, many programs with simple and similar structure. Experimental validation When we execute a program on a computer that is X times faster, we expect the program to be X times faster. Explanation with mathematical model Machine- and system-dependent features of the model are all constants. 3 Linear () for (int i = ; i < ; i++) Logarithmic (log ) public static void f(int ) if ( == ) return; f(/2) Stay tuned for examples. Quadratic ( 2 ) for (int i = ; i < ; i++) for (int j = i+; j < ; j++) Linearithmic ( log ) public static void f(int ) if ( == ) return; f(/2) f(/2) for (int i = ; i < ; i++) Cubic ( 3 ) for (int i = ; i < ; i++) for (int j = i+; j < ; j++) for (int k = j+; k < ; k++) Exponential (2 ) public static void f(int ) if ( == ) return; f(-) f(-) ignore for practical purposes (infeasible for large ) 32

9 Order of growth classifications An important implication order of growth description function slope of line in log-log plot (b) factor for doubling method (2 b ) constant log-log plot 3 2 log Moore's Law. Computer power increases by a roughly a factor of 2 every 2 years. Q. My problem size also doubles every 2 years. How much do I need to spend to get my job done? logarithmic log a very common situation: weather prediction, transaction processing, cryptography linear 2 linearithmic log 2 quadratic cubic if input size doubles running time increases by this factor If math model gives order of growth, use doubling method to validate 2 b ratio. Problem size If not, use doubling method and solve for b = lg(t/t/2) to estimate order of growth to be b. Time 8T 4T 2T T K 2K 4K 8K log math model may have log factor Do the math T = a 3 T2 = (a/2)(2 ) 3 = 4a 3 = 4T running time today running time in 2 years A. You can't afford to use a quadratic algorithm (or worse) to address increasing problem sizes. now 2 years 4 years from now from now 2M years from now $X $X $X $X log $X $X $X $X 2 $X $2X $4X $2 M X 3 $X $4X $6X $4 M X Meeting the challenge Caveats My program is taking too long to finish. I'm going out to get a pizza. Mine, too. I'm going to run a doubling experiment. It is sometimes not so easy to meet the challenge of predicting performance. There are many other apps running on my computer! We need more terms in the math model: lg +? Hmmm. Still didn't finish. The experiment showed my program to have a higher order of growth than I expected. I found and fixed the bug. Time for some pizza! Your input model is too simple: My real input data is completely different. What happens when the leading term oscillates? Doubling experiments provide good insight on program performance Best practice to plan realistic experiments for debugging, anyway. Having some idea about performance is better than having no idea. Performance matters in many, many situations. Your machine model is too simple: My computer has parallel processors and a cache. Where s the log factor? a(2) b (lg(2)) c a b (lg ) c = 2 b + (lg ) Good news. Doubling method is robust in the face of many of these challenges. 2 b c 35 36

PART I: PROGRAMMIG I JAVA PART I: PROGRAMMIG I JAVA Image sources https://openclipart.org/detail/2567/astrid-graeber-adult-by-anonymous-2567 https://openclipart.

10 PART I: PROGRAMMIG I JAVA PART I: PROGRAMMIG I JAVA Image sources The challenge Empirical analysis Mathematical models Doubling hypothesis Familiar examples CS.7.D.Performance.Doubling CS.7.E.Performance.Examples Example: Gambler s ruin simulation Pop quiz on performance Q. How long to compute chance of doubling million dollars? public class Gambler int stake = Integer.parseInt(args[]); int goal = Integer.parseInt(args[]); int trials = Integer.parseInt(args[2]); double start = System.currentTimeMillis()/.; int wins = ; for (int i = ; i < trials; i++) int t = stake; while (t > && t < goal) if (Math.random() <.5) t++; else t--; if (t == goal) wins++; double now = System.currentTimeMillis()/.; StdOut.print(wins + " wins of " + trials); StdOut.printf(" (%.f seconds)\n", now - start); T T/T/ = = % java Gambler 2 53 wins of (4 seconds) % java Gambler wins of (7 seconds) % java Gambler wins of (56 seconds) % java Gambler wins of (286 seconds) math model says order of growth should be 2 Q. Let T be the running time of program Mystery and consider these experiments: public class Mystery int = Integer.parseInt(args[]); Q. Predict the running time for = 64,. Q. Estimate the order of growth. T (in seconds) T/T/ A. 4.8 million seconds (about 2 months). % java Gambler wins of (72 seconds) 39 4

11 Pop quiz on performance Another example: Coupon collector Q. Let T be the running time of program Mystery and consider these experiments. Q. How long to simulate collecting million coupons? T T/T/2 public class Mystery int = Integer.parseInt(args[]); Q. Predict the running time for = 64,. A. 248 seconds. T (in seconds) T/T/ = = = public class Collector int = Integer.parseInt(args[]); int trials = Integer.parseInt(args[]); int cardcnt = ; double start = System.currentTimeMillis()/.; for (int i = ; i < trials; i++) int valcnt = ; boolean[] found = new boolean[]; while (valcnt < ) int val = (int) (StdRandom() * ); cardcnt++; if (!found[val]) valcnt++; found[val] = true; double now = System.currentTimeMillis()/.; StdOut.printf( %d %.f,, *Math.log() *); StdOut.print(cardcnt/trials); StdOut.printf(" (%.f seconds)\n", now - start); = 63 2 % java Collector (7 seconds) % java Collector (4 seconds) math model says order of growth should be log % java Collector 5 Q. Estimate the order of growth (3 seconds) A. 2, since lg 4 = 2. 4 A. About minute. might run out of memory trying for billion % java Collector (66 seconds) 42 Analyzing typical memory requirements Summary A bit is or and the basic unit of memory. A byte is eight bits the smallest addressable unit. megabyte (MB) is about million bytes. gigabyte (GB) is about billion bytes. Use computational experiments, mathematical analysis, and the scientific method to learn whether your program might be useful to solve a large problem. Primitive-type values System-supported data structures (typical) Q. What if it's not fast enough? type bytes boolean ote: not bit type bytes int[] A. yes does it scale? no char 2 int 4 float 4 long 8 double 8 double[] int[][] ~ 4 2 double[][] ~ 8 2 String buy a new computer and solve bigger problems no learn a better algorithm found one? yes Example. 2-by-2 double array uses ~32MB. invent a better algorithm Plenty of new algorithms awaiting discovery. Example: Solve 3-sum efficiently 43 44

Case in point PA R T I : P R O G R A M M I G I J AVA ot so long ago, 2 CS grad students had a program to index

jpg yes does it scale? buy a new computer and solve bigger problems changing the world?

6) invent a better algorithm Lesson. Performance matters!

12 Case in point PA R T I : P R O G R A M M I G I J AVA ot so long ago, 2 CS grad students had a program to index and rank the web (to enable search). Image source yes does it scale? buy a new computer and solve bigger problems changing the world? maybe yes Larry Page and Sergey Brin no learn a better algorithm not yet no found one? yes PageRank (see Section.6) invent a better algorithm Lesson. Performance matters! PA R T I : P R O G R A M M I G I J AVA Computer Science Computer Science An Interdisciplinary Approach Section 4. ROBERT SEDGEWICK K E V I WAY E 45 CS.7.E.Performance.Examples

8. Performance Analysis

COMPUTER SCIENCE S E D G E W I C K / W A Y N E 8. Performance Analysis Section 4.1 http://introcs.cs.princeton.edu COMPUTER SCIENCE S E D G E W I C K / W A Y N E 8. Performance analysis The challenge Empirical