Inequality Constrained Spline Interpolation

Inequality Constrained Spline Interpolation Scott Kersey Workshop on Spline Approximation and Applications on Carl de Boor s 80th Birthday Institute for Mathematical Sciences National University of Singapore December Dec 4 6, 2017 December 5, 2017 (IMS Spline Workshop) Spline Interpolation December 5, 2017 1 / 58

Abstract Abstract The variational problem of best inequality constrained spline interpolation offers a generalization of spline interpolation that smooths rough data while keeping precise control on tolerances (error) at data points. While first studied in the 1960s, this problem has received just modest attention compared to it s contemporaries best spline interpolation, the smoothing spline, least squares splines, splines in tension and quasi-interpolation resulting in just a handful of papers and specialized implementations. While no commercial implementations seem readily available (such as IMSL), the problem is justifiably nonlinear and easily handled by methods of optimization. However, we think that by exploiting specific structure of splines one may produce a simpler and/or more efficient implementation. It is the aim of this work to attempt this. In this talk we describe an implementation based on the active set method in optimization combined with solutions to the problem of best spline interpolation. We also show how inequality constrained splines can be used to produce good or optimal knots, and we describe applications to parametric curves and surfaces in CAGD. (IMS Spline Workshop) Spline Interpolation December 5, 2017 2 / 58

(IMS Spline Workshop) Spline Interpolation December 5, 2017 3 / 58

Distinguished Lecturer Series (IMS Spline Workshop) Spline Interpolation December 5, 2017 4 / 58

What is a Spline? (IMS Spline Workshop) Spline Interpolation December 5, 2017 5 / 58

(IMS Spline Workshop) Spline Interpolation December 5, 2017 6 / 58

Overview of Talk (IMS Spline Workshop) Spline Interpolation December 5, 2017 6 / 58

Overview of Talk 1 Some Variational Spline Problems (IMS Spline Workshop) Spline Interpolation December 5, 2017 6 / 58

Overview of Talk 1 Some Variational Spline Problems 2 Quadratic Programming (IMS Spline Workshop) Spline Interpolation December 5, 2017 6 / 58

Overview of Talk 1 Some Variational Spline Problems 2 Quadratic Programming 3 Minimal Properties and Optimality for Inequality Splines (IMS Spline Workshop) Spline Interpolation December 5, 2017 6 / 58

Overview of Talk 1 Some Variational Spline Problems 2 Quadratic Programming 3 Minimal Properties and Optimality for Inequality Splines 4 Ranking Knots choosing good ones (IMS Spline Workshop) Spline Interpolation December 5, 2017 6 / 58

Overview of Talk 1 Some Variational Spline Problems 2 Quadratic Programming 3 Minimal Properties and Optimality for Inequality Splines 4 Ranking Knots choosing good ones 5 Applications to Parametric Curves (IMS Spline Workshop) Spline Interpolation December 5, 2017 6 / 58

Part I: Some Classical (Variational) Spline Problems Best linearized spline interpolation (Schoenberg,Holloday,de Boor) Smoothing splines (Schoenberg, Reinch, Wahba) Least squares splines (de Boor and Rice) Splines in tension (Schweikart, Nielson) Quasi-interpolation (de Boor and Fix) Best inequality constrained spline interpolation 1 Titanium Heat data from: Carl de Boor and John Rice, Least Squares Cubic Spline Approximation I Fixed Knots (IMS Spline Workshop) Spline Interpolation December 5, 2017 7 / 58

Remark All these problems were developed by the 1960 s or early 70 s. (IMS Spline Workshop) Spline Interpolation December 5, 2017 8 / 58

Remark All these problems were developed by the 1960 s or early 70 s. The first 5 problems have gotten much more attention (more literature) than the inequality problem. (IMS Spline Workshop) Spline Interpolation December 5, 2017 8 / 58

Remark All these problems were developed by the 1960 s or early 70 s. The first 5 problems have gotten much more attention (more literature) than the inequality problem. I can t think of any standard software package that solves the inequality problem. Maybe the reason is that the inequality problem is difficult due to non-lineararities. Or, perhaps the thought is optimization software will solve it. Goal The goal of this study is to develop an efficient program to solve the inequality problem by combining spline and optimization methods. (IMS Spline Workshop) Spline Interpolation December 5, 2017 8 / 58

Remark All these problems were developed by the 1960 s or early 70 s. The first 5 problems have gotten much more attention (more literature) than the inequality problem. I can t think of any standard software package that solves the inequality problem. Maybe the reason is that the inequality problem is difficult due to non-lineararities. Or, perhaps the thought is optimization software will solve it. Goal The goal of this study is to develop an efficient program to solve the inequality problem by combining spline and optimization methods. In doing so, we also want to generate an optimal set of knots, and generalize to parametric curves and surfaces. (IMS Spline Workshop) Spline Interpolation December 5, 2017 8 / 58

Best Inequality Constrained Spline Interpolation Linearized bending energy: E(f ) := b a D m f (t) 2 dt. (IMS Spline Workshop) Spline Interpolation December 5, 2017 9 / 58

Best Inequality Constrained Spline Interpolation Linearized bending energy: E(f ) := b a D m f (t) 2 dt. Data: (a i, b i ) : i = 0,..., n. (IMS Spline Workshop) Spline Interpolation December 5, 2017 9 / 58

Best Inequality Constrained Spline Interpolation Linearized bending energy: E(f ) := b a D m f (t) 2 dt. Data: (a i, b i ) : i = 0,..., n. Goal: minimize{e(f ) : a i f (t i ) b i } (IMS Spline Workshop) Spline Interpolation December 5, 2017 9 / 58

Best Inequality Constrained Spline Interpolation Linearized bending energy: E(f ) := b a D m f (t) 2 dt. Data: (a i, b i ) : i = 0,..., n. Goal: minimize{e(f ) : a i f (t i ) b i } Note that we can express the data as tolerances y i ± ε i. Hence, the terminology near-interpolation. Carl suggested to call these gates. Thus, we can say best interpolation to gates. (IMS Spline Workshop) Spline Interpolation December 5, 2017 9 / 58

Best Inequality Constrained Spline Interpolation Linearized bending energy: E(f ) := b a D m f (t) 2 dt. Data: (a i, b i ) : i = 0,..., n. Goal: minimize{e(f ) : a i f (t i ) b i } Note that we can express the data as tolerances y i ± ε i. Hence, the terminology near-interpolation. Carl suggested to call these gates. Thus, we can say best interpolation to gates. Perhaps we can say set-valued interpolation, borrowing terminology from Nira Dyn. (IMS Spline Workshop) Spline Interpolation December 5, 2017 9 / 58

Literature on Inequality Constrained Splines Inequality Splines (Atteia 67) Splines on Convex Sets (Atteia 67) Splines on Convex Sets (Laurent 69) Spline via Optimal Control (Mangasarian and Schumaker 69) Construction of Smoothing Splines by Quadratic Programming (Budkina 2000) Near Interpolation for Parametric Curves (Kersey 2000, 03a, 03b) Multivariate Smoothing Splines with Obstacles (Leetma and Oja 2009) (IMS Spline Workshop) Spline Interpolation December 5, 2017 10 / 58

Immediate Benefits of the Inequality-Constrained Spline (IMS Spline Workshop) Spline Interpolation December 5, 2017 11 / 58

Immediate Benefits of the Inequality-Constrained Spline Benefits: The inequality spline can smooth rough data (like a smoothing spline and least squares spline). (IMS Spline Workshop) Spline Interpolation December 5, 2017 11 / 58

Immediate Benefits of the Inequality-Constrained Spline Benefits: The inequality spline can smooth rough data (like a smoothing spline and least squares spline). The error is precisely controlled at the data sites (unlike smoothing and least squares splines). This may be useful when designing parts with given tolerances, or when data is inaccurate. The spline is determined by the active knots. Other knots fall away naturally, leaving a sparser set of optimal knots. (IMS Spline Workshop) Spline Interpolation December 5, 2017 11 / 58

Immediate Benefits of the Inequality-Constrained Spline Benefits: The inequality spline can smooth rough data (like a smoothing spline and least squares spline). The error is precisely controlled at the data sites (unlike smoothing and least squares splines). This may be useful when designing parts with given tolerances, or when data is inaccurate. The spline is determined by the active knots. Other knots fall away naturally, leaving a sparser set of optimal knots. Disadvantage: How to choose the a i, b i (or tolerances ε i ). (IMS Spline Workshop) Spline Interpolation December 5, 2017 11 / 58

Good Set of Knots Original data sites: 49. Interior knots: 13. (IMS Spline Workshop) Spline Interpolation December 5, 2017 12 / 58

Optimal Set of Knots (IMS Spline Workshop) Spline Interpolation December 5, 2017 13 / 58

Application: Data with Error Interpolation Near-Interpolation (IMS Spline Workshop) Spline Interpolation December 5, 2017 14 / 58

Part II: Quadratic Programming (IMS Spline Workshop) Spline Interpolation December 5, 2017 15 / 58

Part II: Quadratic Programming Spline: f = Nα. (IMS Spline Workshop) Spline Interpolation December 5, 2017 15 / 58

Part II: Quadratic Programming Spline: f = Nα. Linearized bending energy: E(f ) := b for H symmetric positive semidefinite. a D k f (t) 2 dt = α T Hα. (IMS Spline Workshop) Spline Interpolation December 5, 2017 15 / 58

Part II: Quadratic Programming Spline: f = Nα. Linearized bending energy: E(f ) := b a D k f (t) 2 dt = α T Hα. for H symmetric positive semidefinite. a i f (t i ) b i = Aα B. (IMS Spline Workshop) Spline Interpolation December 5, 2017 15 / 58

Part II: Quadratic Programming Spline: f = Nα. Linearized bending energy: E(f ) := b a D k f (t) 2 dt = α T Hα. for H symmetric positive semidefinite. a i f (t i ) b i = Aα B. Quadratic Programming Problem: minimize{α T Hα : Aα B} (IMS Spline Workshop) Spline Interpolation December 5, 2017 15 / 58

First Order Optimality Conditions of Karush-Kuhn Tucker (KKT): 2Hα + A T µ = 0 Aα = B µ 2 (B Aα) = 0 µ 0 (IMS Spline Workshop) Spline Interpolation December 5, 2017 16 / 58

Solving Quadratic Programming Problems by the Active Set Method (IMS Spline Workshop) Spline Interpolation December 5, 2017 17 / 58

Solving Quadratic Programming Problems by the Active Set Method Start with an initial feasible point α 0. For j = 0, 1,... 1 Get the active set (set where inequality constraints are equality). 2 Solve a constrained least squares program on the active set. 3 Compute Lagrange multipliers µ. 4 Add or remove just one active constraint. 5 Set up new active set and go back to step 2. 6 Terminate when µ 0 for all inequality constraints. (IMS Spline Workshop) Spline Interpolation December 5, 2017 17 / 58

Solving Quadratic Programming Problems by the Active Set Method Start with an initial feasible point α 0. For j = 0, 1,... 1 Get the active set (set where inequality constraints are equality). 2 Solve a constrained least squares program on the active set. 3 Compute Lagrange multipliers µ. 4 Add or remove just one active constraint. 5 Set up new active set and go back to step 2. 6 Terminate when µ 0 for all inequality constraints. Remark The active set converges in finite steps (under certain conditions). (IMS Spline Workshop) Spline Interpolation December 5, 2017 17 / 58

Solving Quadratic Programming Problems by the Active Set Method Start with an initial feasible point α 0. For j = 0, 1,... 1 Get the active set (set where inequality constraints are equality). 2 Solve a constrained least squares program on the active set. 3 Compute Lagrange multipliers µ. 4 Add or remove just one active constraint. 5 Set up new active set and go back to step 2. 6 Terminate when µ 0 for all inequality constraints. Remark The active set converges in finite steps (under certain conditions). The constrained least squares problem is best spline interpolation. (IMS Spline Workshop) Spline Interpolation December 5, 2017 17 / 58

Solving Quadratic Programming Problems by the Active Set Method Start with an initial feasible point α 0. For j = 0, 1,... 1 Get the active set (set where inequality constraints are equality). 2 Solve a constrained least squares program on the active set. 3 Compute Lagrange multipliers µ. 4 Add or remove just one active constraint. 5 Set up new active set and go back to step 2. 6 Terminate when µ 0 for all inequality constraints. Remark The active set converges in finite steps (under certain conditions). The constrained least squares problem is best spline interpolation. What is the meaning of the Lagrange mutipliers µ i? (IMS Spline Workshop) Spline Interpolation December 5, 2017 17 / 58

Part III: Minimimal Properties and Optimality Best Interpolation: Linearized bending energy: E(f ) := b a D k f (t) 2 dt. (IMS Spline Workshop) Spline Interpolation December 5, 2017 18 / 58

Part III: Minimimal Properties and Optimality Best Interpolation: Linearized bending energy: E(f ) := b a D k f (t) 2 dt. Data: (x 0, y 0),..., (x n, y n). (IMS Spline Workshop) Spline Interpolation December 5, 2017 18 / 58

Part III: Minimimal Properties and Optimality Best Interpolation: Linearized bending energy: E(f ) := b a D k f (t) 2 dt. Data: (x 0, y 0),..., (x n, y n). Problem: minimize{e(f ) : f (t i ) = y i, f W k,2 (a, b)}. (IMS Spline Workshop) Spline Interpolation December 5, 2017 18 / 58

Part III: Minimimal Properties and Optimality Best Interpolation: Linearized bending energy: E(f ) := b a D k f (t) 2 dt. Data: (x 0, y 0),..., (x n, y n). Problem: minimize{e(f ) : f (t i ) = y i, f W k,2 (a, b)}. Solution: f is a natural spline of order 2k and smoothness k. (IMS Spline Workshop) Spline Interpolation December 5, 2017 18 / 58

Literature on Best Interpolation: Demonstrates for k = 2, Best Linearized Cubic Spline Interpolation (Schoenberg 1946) First Minimum Property for k = 2 (Holladay 1957) Minimum Properties and Best Approximation for general k (de Boor 1963) Abstract Splines and Projections (de Boor and Lynch 1965) (IMS Spline Workshop) Spline Interpolation December 5, 2017 19 / 58

Spline Interpolation using Lagrange Multipler theory (IMS Spline Workshop) Spline Interpolation December 5, 2017 20 / 58

Spline Interpolation using Lagrange Multipler theory Best interpolation minimize{e(f ) : f (t i ) = y i }. (IMS Spline Workshop) Spline Interpolation December 5, 2017 20 / 58

Spline Interpolation using Lagrange Multipler theory Best interpolation minimize{e(f ) : f (t i ) = y i }. Lagrangian: L(f, µ) = b a D k f (t) 2 dt + i = Tf, Tf L2 + µ, Λf y l2. µ i (f (t i ) y i ) (IMS Spline Workshop) Spline Interpolation December 5, 2017 20 / 58

Spline Interpolation using Lagrange Multipler theory Best interpolation minimize{e(f ) : f (t i ) = y i }. Lagrangian: L(f, µ) = b a D k f (t) 2 dt + i = Tf, Tf L2 + µ, Λf y l2. µ i (f (t i ) y i ) Theorem The best spline satisfies: 2T Tf + Λ µ = 0. (IMS Spline Workshop) Spline Interpolation December 5, 2017 20 / 58

Proof of Theorem Proof. Let g be a variation. d L(f + αg, µ) = d T (f + αg), T (f + αg) dα α=0 dα L2 + µ, Λ(f + αg) y l2 α=0 = 2 Tf, Tg L2 + µ, Λg l2 = 2 T Tf, g L2 + Λ µ, g L2 = 2T Tf + Λ µ, g L2. Since true for all variations, we have the result. (IMS Spline Workshop) Spline Interpolation December 5, 2017 21 / 58

Optimality and Lagrange Multipliers for Spline Interpolation Theorem The Lagrange Multipliers are µ i = 2 jmp i D 2k 1 f. The best spline function satisfies: Λ (2 JmpD 2k 1 f + µ) = 0. (IMS Spline Workshop) Spline Interpolation December 5, 2017 22 / 58

Inequality Spline Interpolation using Lagrange Multipler theory (IMS Spline Workshop) Spline Interpolation December 5, 2017 23 / 58

Inequality Spline Interpolation using Lagrange Multipler theory Best near-interpolation minimize{e(f ) : a i f (t i ) b i }. (IMS Spline Workshop) Spline Interpolation December 5, 2017 23 / 58

Inequality Spline Interpolation using Lagrange Multipler theory Best near-interpolation minimize{e(f ) : a i f (t i ) b i }. Lagrangian: L(f, µ) = b a D k f (t) 2 dt + i µ + i (f (t i ) a i ) i µ i (f (t i ) b i ). Theorem The Lagrange Multipliers are µ i = 2 jmp i D 2k 1 f with µ i := µ + i µ i. The best spline function satisfies: Λ (2 JmpD 2k 1 f + µ) = 0. with at most one of µ ± i nonzero for each i. (IMS Spline Workshop) Spline Interpolation December 5, 2017 23 / 58

Solving Best Inequality Spline Problem by csapni() (IMS Spline Workshop) Spline Interpolation December 5, 2017 24 / 58

Solving Best Inequality Spline Problem by csapni() Start with an initial feasible point α 0. For j = 0, 1,... 1 Get the active set (set where inequality constraints are equality). 2 Solve the problem of Best Interpolation using csape(). 3 Compute Lagrange multipliers µ = Jmpf 2k 1. 4 Add or remove just one active constraint. 5 Set up new active set and go back to step 2. 6 Terminate when µ 0 for all inequality constraints. (IMS Spline Workshop) Spline Interpolation December 5, 2017 24 / 58

Remarks about csapni() Remark (IMS Spline Workshop) Spline Interpolation December 5, 2017 25 / 58

Remarks about csapni() Remark The Matlab function csapne() is very fast with linear growth. It can handle 100, 000 interpolation points without a problem. (IMS Spline Workshop) Spline Interpolation December 5, 2017 25 / 58

Remarks about csapni() Remark The Matlab function csapne() is very fast with linear growth. It can handle 100, 000 interpolation points without a problem. The function csapni() is faster than a standard active set method, even after doing tricks with the factorizations. However, the growth rate is still greater than linear. Since there are approximately n/2 active constraints, there are going to be that many iterations on the outside loop. Since csape() has linear growth, we expect csapni() to be order n 2. (IMS Spline Workshop) Spline Interpolation December 5, 2017 25 / 58

Remarks about csapni() Remark Claim The Matlab function csapne() is very fast with linear growth. It can handle 100, 000 interpolation points without a problem. The function csapni() is faster than a standard active set method, even after doing tricks with the factorizations. However, the growth rate is still greater than linear. Since there are approximately n/2 active constraints, there are going to be that many iterations on the outside loop. Since csape() has linear growth, we expect csapni() to be order n 2. This problem can be solved in O(n) time. Our only hope to decrease the growth rate is to handle the the interpolation problem with constant time. As we remove or add a single knot (constraint), we can interpolate on a fixed window (say of 100 points) rather than all n points. Due to the exponential decay in spline interpolation, we expect that we can achieve this without loss of precision, giving us an order n implementation. (IMS Spline Workshop) Spline Interpolation December 5, 2017 25 / 58

Part IV: Good knots Ranking knots Remark Tpyically we have more knots than we want in interpolation or near-interpolation. Hence, we want to remove those less important. The questions is, how do we decide this? What we do is choose the m most influential knots, and remove the rest. How do we decide who survives? Based on experience with near-interpolation and the smoothing spline, we guess that the points with larger lagrange multipliers (i.e., larger jumps in jmp ti D 2k 1 f ) should be more valuable, so we eliminate those with smaller multipliers. Indeed, zero multipliers can be eliminated automatically without effect on the spline. (IMS Spline Workshop) Spline Interpolation December 5, 2017 26 / 58

Remark (Ranking knots) Keeping knots corresponding to large lagrange multipliers leads to a high concentration of knots in areas where just a couple would suffice. Hence, we need to consider a better rank function. (IMS Spline Workshop) Spline Interpolation December 5, 2017 27 / 58

Theorem (Balanced Rank Function) The rank function r i = rank(t i ) := (t i+1 t i 1 ) 3 λ i balances the polynomial error across the knot partition T. (IMS Spline Workshop) Spline Interpolation December 5, 2017 28 / 58

Literature on Balanced Intervals References (Primary Spline Approximation References Used) [db73] Good approximation by splines with variable knots, C. de Boor, in Spline Functions and Approximation (1973). [PGS78] Practical Guide to Splines, C. deboor (1978). (The program newknt()). [Y93] An algorithm for data reduction using splines with free knots, Y. Hu, IMA J. Num. Anal. 13 (1993) [CA93] Constructive Approximation, R. DeVore and G. Lorentz (1993) [NA98] Nonlinear Approximation, R. DeVore, Acta Numerica (1998) [R69] The Approximation of Functions II, J. Rice (1969) [PP87] Rational approximation of real functions, P. Petrushev and V. Popov, Encylopedia of Math and it s applications (28) (1987) (IMS Spline Workshop) Spline Interpolation December 5, 2017 29 / 58

Definition (Fixed Knot Spline Space) Let S r (T ) be the space of piecewise polynomials of order r (degree r 1) that break at the knots T = [0 = t 0 < t 1 t 2 t n = 1]. No smoothness is imposed on this space. This is a linear space. (IMS Spline Workshop) Spline Interpolation December 5, 2017 30 / 58

Example (Piecewise Constants: The Simplist Splines) s(t) = with α i = ( 1) i, 5 α i χ [ti 1,t i ) i=1 T = [0,.2,.4,.6,.8, 1] and χ [ti 1,t i ) = { 1 t [t i 1, t i ) 0 otherwise. (IMS Spline Workshop) Spline Interpolation December 5, 2017 31 / 58

Definition (Approximation Error) Approximation Error to a Function f (x) (in the inf norm on [0, 1]) s f := sup{ s(t) f (t) : t [0, 1]} Distance to Spline Space (in the inf norm) σ T,r (f ) := dist(f, S r (T )) = inf{ s f : s S r (T )}. s is a best approximation to f from S r (T ) if s f = σ T,r (f ). (IMS Spline Workshop) Spline Interpolation December 5, 2017 32 / 58

Example (Approximation to f (x) = x by Piecewise Constants) Let 4 s := α i χ [ti 1,t i ) i=1 with α = [.25,.5,.75, 1] and T = [0,.25,.5,.75, 1]. What is f s? Is s a b.a. from S 1(T )? If not, how could s(t) be improved? (IMS Spline Workshop) Spline Interpolation December 5, 2017 33 / 58

Theorem (Kahane61, Best Approximation by Piecewise Constants) Let f C([0, 1]). Then σ n,1(f ) M 2n for n = 1, 2, 3,... iff f BV [0, 1], with M := Var [0,1] (f ), with T var [a,b] := sup{ f (t i+1 ) f (t i ) : partitions T }. i=1 Proof of one direction of Kahane s theorem, from D98. Suppose that f BV [0, 1] with M := Var [0,1] (f ). Since f is continuous, we can find a partition T = [0 = t 0,..., t n < t n+1 = 1] such that Var [ti 1,t i ](f ) = M/n. Let s = i α iχ [ti 1,t i ) with α i = (f (t i 1 ) + f (t i ))/2. Then, s f M 2n. (IMS Spline Workshop) Spline Interpolation December 5, 2017 34 / 58

Example (f (x) = x) Since f is monotonic, M = var(, [0, 1]) = 1. Hence, M n = var(, [0, 1]) = 1 5 5. Following the proof, we require var(, [t i 1, t i ]) = t i t i 1 = 1 5. This gives, t i = ( i n ) 2, and so T = [0,.04,.16,.36,.64, 1]. By choosing α i = ( i 1 n + i n ) /2, the error on each piece is 1 M = 1, hence 2 n 10 s f = M 2n =.1. (IMS Spline Workshop) Spline Interpolation December 5, 2017 35 / 58

Remark (Balanced Intervals) Kahane s balances the error on each piece. In this case, f s = max f s (t i 1, t i ) M i 2n. Hence, the error on the entire interval is balanced by the error on the worst interval, which is about the same. If f is continuous, these local errors are the same.. So by balancing the variation of a function, we are balancing the polynomial error. (IMS Spline Workshop) Spline Interpolation December 5, 2017 36 / 58

Theorem (Existence and Error from Fixed Knot Spline Space) For f C k ([0, 1]) with 0 k < r, there exists s S r (T ) such that s f C r h k ω(f (k), h). for some constant C r depending only on r and h := max{t i t i 1 : i = 1,..., n}, with ω(g, h) := sup{ g(x) g(y) : x y h}. r + 1 In [PGS] deboor shows C r when k = 0. 2 Corollary For f C r ([0, 1]), there exists s S r (T ) such that s f C r h r f (r). If T is uniform (in which case h = 1 ) n s f C r n r f (r). (IMS Spline Workshop) Spline Interpolation December 5, 2017 37 / 58

Definition (Free Knot Spline Space) Let Σ n,r be the space of all splines of order r with n + 1 knots (at most n intervals) Σ n,r := } {S r (T ) : T = [0 = t 0 < t 1 t 2 t n = 1]. Remarks: This space is nonlinear! (I.e., The space is not closed under addition, since adding two splines in Σ n,r with different knots typically results in a spline with 2n knots.) We want to find the best knots, if possible, but usually settle for good knots, which is usually good enough. The best (good) knots depend on the function. How to choose the knots for piecewise constants? Want bounds on approximation, and theorems of existence and uniquenss. We can often get the same approximation error with far fewer knots. (IMS Spline Workshop) Spline Interpolation December 5, 2017 38 / 58

Definition (Distance to Nonlinear Free-Knot Spline Space) σ n,r (f ) p := inf{ s f p : s Σ n,r } Theorem (R69,Existence of Best Approximation) Let f L p([0, 1]) for 0 < p < or f C([0, 1]) for p =. There exists s Σ n,r such that s f p = σ n,r (f ) p. Theorem (DL93,Burchard74, Smooth Approximation) Let f L p([0, 1]) for 0 < p and n, r 2. Then inf{ s f p : s Σ n,r C r 2 } = σ n,r (f ) p. Hence, we can get arbitrarily close to a best piecewise polynomial approximation with a smooth spline. (IMS Spline Workshop) Spline Interpolation December 5, 2017 39 / 58

Piecewise Constants: Balanced and Uniform Knots Knots: t i = i equally spaced. n Max error occurs on the left: s = 1 t1 = 1 2 2 n Not the optimal rate! Knots: t i = ( i n ) 2. More knots placed near singularity. Max error occurs on the left: Optimal rate. s = 1 t1 = 1 2 2 n (IMS Spline Workshop) Spline Interpolation December 5, 2017 40 / 58

Free-Knot Spline Approximation Estimate Remark The following theorem generalizes Kahane s theorem. The statement of the theorem is taken from [D93], Chap. 12, Theorem 4.5. Note that while in fixed knot approximation we usually see the (maximal) mesh spacing h in the estimates. For free-knot splines we don t know the mesh spacing, but we do have a term 1/n which is h for a uniform parametrization. Theorem (Freud and Popov (1969), Subbotin and Chernykh (1970)) If r = 1, 2,... and f (r 1) is of bounded variation on [0, 1], then σ n,r (f ) C r n r var [0,1] f (r 1). (IMS Spline Workshop) Spline Interpolation December 5, 2017 41 / 58

Remark (Conclusion of Free-Knot Spline Basics) The key aspect from the previous discussion relevant to the remainder of this talk is the balancing of intervals. In particular, we have showed for the simplest case of piecewise constant approximation that balanced intervals provide the best bound for approximation. Carl s program newknt() addresses this. (IMS Spline Workshop) Spline Interpolation December 5, 2017 42 / 58

Consequences: 1 Assume f (4) L 1[0, 1]. Then λ i 2 ti+1 t i 1 f (4) (t) dt. 2 The balancing functions have the form: F i r i := (t i+1 t i 1 ) 3 λ i /2. By [PGS], F i is connected to polynomial error on an interval. 3 s 0 is piecewise constant with coefficients α the cumulative sum of λ. That is, α j = j λ i. i=0 (IMS Spline Workshop) Spline Interpolation December 5, 2017 43 / 58

Proofs. 1 For the first corollary, λ i = 2 s f (t + i ) s f (t i ) = 2 s 0 (t i+1) s 0 (t + i 1) = 2 s 0 (t i+1 ) s 0 (t i 1 ) (t i+1 t i 1 ) t i+1 t i 1 = 2 2 ti+1 s t i 1 ti+1 t i 1 0 (t i ) s 0 (t i 1 ) dt t i t i 1 f (4) (t) dt. 2 The second corollary follows by F i (t i+1 t i 1 ) 3 ti+1 t i 1 f (4) (t) dt (t i+1 t i 1 ) 3 λ i /2. 3 For the third corollary, we note that since s 0 is a piecewise cubic spline, then s 0 is piecewise constant. Hence, the result follows since λ i is the jump in the third derivative across the knot. (IMS Spline Workshop) Spline Interpolation December 5, 2017 44 / 58

Variational Approach 1 Solve the inequality constrained program to find a good initial spline approximation s 0 to f such that s 0 f ɛ/2 with active knots t i and Lagrange multipliers λ i. 2 Set n, the number of intervals (n + 1 knots) 3 Rank the n 1 interior knots according to: r i = λ i (t i+1 t i 1 ) 3. 4 Using the highest rank n 1 interior knots and two end knots, find the least squares spline fit s f. 5 If s 0 s f ɛ/2, increase n and go to step 3. (IMS Spline Workshop) Spline Interpolation December 5, 2017 45 / 58

Titanium Heat Data Runge Function f (x) = 1/(1 + x 2 ) 8 intervals (7 interior knots) Error:.015 16 intervals (15 interior knots) Error.00071 (IMS Spline Workshop) Spline Interpolation December 5, 2017 46 / 58

Remark (Conclusion) The proposed variational approach begins with a very large (dense) sampling of data, and hence a very large dense set of knots. Our method is to remove knots. The first set of knots fall out easily where the constraints are inactive. This does not change the spline fit. We then delete more that are ranked lower according to our rank measure. After choosing the most influential knots, we find a least square spline fit. In practice, it maintains the good error estimate, as well as the near-interpolant would. We don t feel it necessary to work harder to move the final knots to optimal positions, because the original set of knots was already from a dense subset. This may not always be acceptable, such as for the square root function where optimal knots are on the order of e 5 in magnitudes. (IMS Spline Workshop) Spline Interpolation December 5, 2017 47 / 58

Part V: Applications to Parametric Curves and Surfaces (IMS Spline Workshop) Spline Interpolation December 5, 2017 48 / 58

Parametric Curves We apply to parametric curves as follows: f (t) = (x(t), y(t)) with x(t) and y(t) spline functions. represents the Euclidean norm Interval constraints become polygonal constraints For the problem of near-interpolation 1, circles can be approximated polygons. We would like to provide optimal parametrizations (data sites and knots). 7 S. Kersey, Near-Interpolation, Numer. Math. 94, 523 540 (2003) (IMS Spline Workshop) Spline Interpolation December 5, 2017 49 / 58

Typical Inequality Constraints Functions: f (t i ) b f (t i ) a K ij = Parametric Curves: { } x : x (N j P j ) 0 (Half Space) f (t i ) K i := j K ij (Convex Set) b f(t i ) f(t i ) N P a (IMS Spline Workshop) Spline Interpolation December 5, 2017 50 / 58

Example: Parametric Curves with Polygonal Constraints minimize f b a 1 2 f (t) 2 + 1 2 f (t) 2 dt for quintic spline curve f under constraints f (t i ) K i. We approximate balls using polygons with several sides. Now we have a simple way to solve the problem of near-interpolation with fixed data sites. (IMS Spline Workshop) Spline Interpolation December 5, 2017 51 / 58

Example: Bad Parametrizations for Spline Curve Interpolation Best spline interpolation with free data sites (or knots) is a difficult problem. In the example, we have badly chosen sites. We choose balls with centers at the data points. While the interpolant is a bad fit, the near-interpolant improves things. But can we do better? (IMS Spline Workshop) Spline Interpolation December 5, 2017 52 / 58

Example: Optimizing Data Sites Consider near-interpolant constraints f (t i ) z i ɛ i where f is a spline curve that solves the problem of best near-interpolation. Hence, f (t i ) is constrained to lie in a closed ball of radius ɛ i. Optimal data sites satisfy the condition Hence, we want a zero of f (t i ) z i f (t i ). F (t i ) := (f (t i ) z i ) f (t i ). By Newton s method, we update t i to t i + t i with t i := F (t i) F (t i ) = (f (t i ) z i ) f (t i ) (f (t i ) z i ) f (t i ) + f (t i ) f (t i ). (0.1) (IMS Spline Workshop) Spline Interpolation December 5, 2017 53 / 58

Example: Best Interpolation with Free Data Sites After applying the parameter update to the previous example, we get: (IMS Spline Workshop) Spline Interpolation December 5, 2017 54 / 58

Example: Best Spline Interpolation with Free Data Sites Repeating the Process: Solve the problem of near-interpolation, update knots, shrink the constraints K i, and repeat until close to interpolation. Now we have a simple way to solve the problem of best spline curve interpolation with free data sites for basic configurations. If the data sites and knots coincide, we also get the optimal knots. (IMS Spline Workshop) Spline Interpolation December 5, 2017 55 / 58

Example: Cornu Spiral Optimal parametrization for interpolation of the Cornu spiral. (IMS Spline Workshop) Spline Interpolation December 5, 2017 56 / 58

Example: Parametric Space Curves Consider polyhedrons around each point of valence 3 (this is needed for linear indepedence of the active constraints). We update data sites just as we did for 2-D curves. (IMS Spline Workshop) Spline Interpolation December 5, 2017 57 / 58

The End. Thank You! (IMS Spline Workshop) Spline Interpolation December 5, 2017 58 / 58