Functional Data Smoothing Methods and Their Applications

Transcription

1 Universiy of Souh Carolina Scholar Commons Theses and Disseraions 2017 Funcional Daa Smoohing Mehods and Their Applicaions Songqiao Huang Universiy of Souh Carolina Follow his and addiional works a: hps://scholarcommons.sc.edu/ed Par of he Saisics and Probabiliy Commons Recommended Ciaion Huang, S.(2017). Funcional Daa Smoohing Mehods and Their Applicaions. (Docoral disseraion). Rerieved from hps://scholarcommons.sc.edu/ed/4345 This Open Access Disseraion is brough o you for free and open access by Scholar Commons. I has been acceped for inclusion in Theses and Disseraions by an auhorized adminisraor of Scholar Commons. For more informaion, please conac dillarda@mailbox.sc.edu.

2 Funcional Daa Smoohing Mehods and Their Applicaions by Songqiao Huang Bachelor of Science Binghamon Universiy, Sae Universiy of New York, 2011 Submied in Parial Fulfillmen of he Requiremens for he Degree of Docor of Philosophy in Saisics College of Ars and Sciences Universiy of Souh Carolina 2017 Acceped by: David B. Hichcock, Major Professor Paramia Chakrabory, Commiee Member Xiaoyan Lin, Commiee Member Susan E. Seck, Commiee Member Cheryl L. Addy, Vice Provos and Dean of he Graduae School

3 c Copyrigh by Songqiao Huang, 2017 All Righs Reserved. ii

4 Dedicaion To my parens Jingmin Huang and Chunhua Xu, grandparens Xishen Xu and Caiyun Jiao, Yanshan Huang and Youlian Gao, and husband Peijie Hou, you are always here wih me whenever and wherever I am. iii

5 Acknowledgmens I wan o express my greaes graiude o my academic advisor, Dr. David B. Hichcock, who has encouraged, inspired, moivaed me, and guided me hrough all he obsacles in my Ph.D. sudy wih grea paience. Dr. Hichcock is no only my academic advisor, bu is also a very good life menor and friend. His knowledge, paience and carefulness oward research, and his grea advice and encouragemens for oher aspecs of life during my years a he Universiy of Souh Carolina, Columbia have helped me grow and become sronger boh academically and personally. I also wan o hank all of my commiee members, Dr. Susan Seck, Dr. Paramia Chakrabory and Dr. Xiaoyan Lin for carefully reviewing my disseraion, and for heir insighful suggesions and commens on my disseraion. Above all, I wan o hank he mos imporan persons in my life: My parens Jingmin Huang and Chunhua Xu, my grandparens Xishen Xu, Caiyun Jiao, and Yanshan Huang, and my husband Peijie Hou. Wihou heir uncondiional love, consisen suppor and encouragemens, I couldn have grown o be he person I am oday. iv

6 Absrac In many subjecs such as psychology, geography, physiology or behavioral science, researchers collec and analyze non-radiional daa, i.e., daa ha do no consis of a se of scalar or vecor observaions, bu raher a se of sequenial observaions measured over a fine grid on a coninuous domain, such as ime, space, ec. Because he underlying funcional srucure of he individual daum is of ineres, Ramsay and Dalzell (1991) named he collecion of opics involving analyzing hese funcional observaions funcional daa analysis (FDA). Topics in funcional daa analysis include daa smoohing, daa regisraion, regression analysis wih funcional responses, cluser analysis on funcional daa, ec. Among hese opics, daa smoohing and daa regisraion serve as preliminary seps ha allow for more reliable saisical inference aferwards. In his disseraion, we include hree research projecs on funcional daa smoohing and is effecs on funcional daa applicaions. In paricular, Chaper 2 mainly presens a unified Bayesian approach ha borrows he idea of ime warping o represen funcional curves of various shapes. Based on a comparison wih he mehod of B-splines developed by de Boor (2001) and some oher mehods ha are well known for is broad applicaions in curve fiing, our mehod is proved o adap more flexibly o highly irregular curves. Then, Chaper 3 discusses subsequen regression and clusering mehods for funcional daa, and invesigaes he accuracy of funcional regression predicion as well as clusering resuls as measured by eiher radiional in-sample and ou-of-sample sum of squares or he Rand index. I is showed ha using our Bayesian smoohing mehod on he raw curves prior o carrying ou he corresponding applicaions provides very compeiive saisical inference and anv

7 alyic resuls in mos scenarios compared o using oher sandard smoohing mehods prior o he applicaions. Lasly, noice ha one resricion for our mehod in Chaper 2 is ha i can only be applied o funcional curves ha are observed on a fine grid of ime poins. Hence, in Chaper 4, we exend he idea of our ransformed basis smoohing mehod in Chaper 2 o he sparse funcional daa scenario. We show via simulaions and analysis ha he proposed mehod gives a very good approximaion of he overall paern as well as he individual rends for he daa wih he cluser of sparsely observed curves. vi

8 Table of Conens Dedicaion iii Acknowledgmens iv Absrac v Lis of Tables ix Lis of Figures xi Chaper 1 Inroducion Lieraure Review Ouline Chaper 2 Bayesian funcional daa fiing wih ransformed B-splines Inroducion Review of Conceps Bayesian Model Fiing for Funcional Curves Kno Selecion wih Reversible Jump MCMC Simulaion Sudies Real Daa Applicaion Discussion vii

9 Chaper 3 Funcional regression and clusering wih funcional daa smoohing Inroducion Simulaion Sudies: Funcional Clusering Simulaion Sudies: Funcional Regression Real Daa Applicaion: Funcional Regression Example Real Daa Applicaion: Funcional Regression Example Discussion Chaper 4 Sparse funcional daa fiing wih ransformed spline basis Inroducion Bayesian Model Fiing for Sparse Funcional Curves Simulaion Sudies Discussion Chaper 5 Conclusion Bibliography viii

10 Lis of Tables Table 2.1 Table 2.2 MSE comparison able for five smoohing mehods. Top: MSE values calculaed wih respec o he observed curve. Boom: MSE values calculaed wih respec o he rue signal curve MSE comparison able for six smoohing mehods. Digis in he Weighs column represen he weighing of he simulaed periodic curve, smooh curve and spiky curve, respecively. Top row in each cell: MSE value calculaed wih respec o he observed curve. Boom row in each cell: MSE value calculaed wih respec o he rue signal curve Table 3.1 Table 3.2 Table 3.3 Table 3.4 Table 3.5 Table 3.6 Rand index values for five smoohing mehods based on regular disance marix Rand index values for five smoohing mehods based on sandardized disance marix In-sample SSE comparison for funcional regression predicions based on simulaed curves Ou-of-sample SSE comparison for funcional regression predicions based on simulaed curves SSE comparison able: Model 1: boh dew poin and humidiy are predicors. Top: SSE for prediced response curves wih respec o he rue observed curve, no furher smoohing on α 1 (), β 11 () and β 12 (). Boom: SSE for prediced response curves wih respec o he rue signal curve, wih furher smoohing on α 1 (), β 11 () and β 12 () using regular B-spline basis funcions In-sample SSE comparison: Model 2: dew poin is he only predicor. Top: SSE for prediced response curves wih respec o he rue signal curve, no furher smoohing on α 2 () and β 1 (). Boom: SSE for prediced response curves wih respec o he rue observed curve, wih furher smoohing on α 2 () and β 1 () using regular B-splines basis funcions ix

11 Table 3.7 Table 3.8 In-sample SSE comparison: Model 3: humidiy is he only predicor. Top: SSE for prediced response curves wih respec o he rue signal curve, no furher smoohing on α 3 () and β 2 (). Boom: SSE for prediced response curves wih respec o he rue observed curve, wih furher smoohing on α 3 () and β 2 () using regular B-splines basis funcions SSE comparison for seven flood evens. Top: SSE values based on funcional regression wihou any smoohing. Boom: SSE values based on funcional regression wih raw daa curves and esimaed inercep and slope curves smoohed via smoohing splines. 68 Table 3.9 SSE comparison for seven flood evens. SSE values based on funcional regression wihou any pre-smoohing on he observaion curves. Cross-validaion idea is uilized o obain ouof-sample predicions. Top: SSE values based on funcional regression wihou any smoohing. Boom: SSE values based on funcional regression wih raw daa curves and esimaed inercep and slope curves smoohed via smoohing splines Table 3.10 SSE comparison for seven flood evens. SSE values based on funcional regression wih Bayesian ransformed B-splines mehod on he observed curves Table 3.11 SSE comparison for seven flood evens. SSE values based on funcional regression wih Bayesian ransformed B-splines mehod on he observaion curves. Top: No furher smoohing on ˆα() and ˆβ() curves. Middle: ˆα() and ˆβ() curves furher smoohed using smoohed splines wih roughness parameer = 0.9. Boom: ˆα() and ˆβ() curves furher smoohed using smoohed splines wih roughness parameer = Table 3.12 SSE comparison for seven flood evens. SSE values based on funcional regression wih Bayesian ransformed B-splines mehod on he observaion curves. Cross-validaion idea is uilized o obain ou-of-sample predicions. Top: No furher smoohing on ˆα() and ˆβ() curves. Boom: ˆα() and ˆβ() curves furher smoohed using smoohed splines wih roughness parameer = x

12 Lis of Figures Figure 2.1 Figure 2.2 Figure 2.3 Comparison plo of wo normal densiy curves. Dashed curve: densiy of N(0, σ 2 d i +c 2 i ζ 2 i ). Solid curve: densiy of N(0, σ 2 d i +ζ 2 i ). (c i, σ di /ζ i ) = (5, 1) Comparison plo of wo normal densiy curves. Dashed curve: densiy of N(0, σ 2 d i +c 2 i ζ 2 i ). Solid curve: densiy of N(0, σ 2 d i +ζ 2 i ). (c i, σ di /ζ i ) = (100, 10) True versus fied simulaed curves. Dashed spiky curve: simulaed rue signal curve. Solid wiggly curve: corresponding observed curve Figure 2.4 Mean squared error race plo of 1000 MCMC ieraions Figure 2.5 Figure 2.6 Figure 2.7 Figure 2.8 Example of se of 15 ransformed B-splines obained from one MCMC ieraion True signal curve versus fied curves from hree compeing mehods. Black solid curve: rue signal curve. Red dashed curve: fied curve obained from Bayesian ransformed B-splines mehod. Blue doed curve: fied curve obained from B- splines basis funcions. Dashed green curve: fied curve obained from B-splines basis funcions wih seleced knos True signal curve versus fied curves from hree compeing mehods. Black solid curve: rue signal curve. Red dashed curve: fied curve obained from Bayesian ransformed B-splines mehod. Blue doed curve: fied curve obained from he wavele basis funcions. Dashed green curve: fied curve obained from he Fourier basis funcions Observed versus smoohed wind speed curves. Top: four observed wind speed curves. Boom: four corresponding wind speed curves smoohed wih Bayesian ransformed B-splines basis funcions xi

13 Figure 2.9 Side by side boxplos of MSE values for five smoohing mehods. From lef o righ: boxplo of MSE values for 18 ocean wind curves smoohed wih he seleced knos B-splines (SKB); he B-splines wih equally seleced knos (B); he Wavele basis (Wave); he Fourier basis (Fourier) and he Bayesian ransformed B-splines (TB) Figure 3.1 Simulaed curves from four clusers Figure 3.2 Figure 3.3 Figure 3.4 Figure 3.5 Boxplos of SSE values for he firs 9 prediced response curves wih no presmoohing or presmoohing using he ransformed B-splines, he Fourier basis, he Wavele basis and regular B- splines basis funcions on he curves. Red line: SSE value for he prediced response curves wih presmoohing on he curves using he ransformed B-splines The firs 9 prediced response curves. Black spiky curves: rue signal response curves. Red long dashed curves: prediced curves wih presmoohing using he ransformed B-splines. Purple wiggly curves: prediced response curves wih no presmoohing on he curves. Green dashed curve: prediced response curves wih presmoohing using he Fourier basis funcions The firs 9 prediced response curves. Black spiky curves: rue signal response curves. Red long dashed curves: prediced curves wih presmoohing using he ransformed B-splines. Blue wiggly curves: prediced response curves wih wih presmoohing using he Wavele basis funcions. Green dashed curve: prediced response curves wih presmoohing using he regular B-spline basis funcions True Orlando emperaure curves and prediced response curves wihou daa smoohing, or wih daa presmoohed using he ransformed B-spline basis and he Fourier basis funcions. Black solid curve: rue Orlando weaher curves. Green solid curve: prediced curves wihou any daa smoohing. Red solid curve: prediced curves wih daa presmoohed using he ransformed B-spline basis funcions. Purple dashed curve: prediced curves wih daa presmoohed using he Fourier basis funcions xii

14 Figure 3.6 Figure 3.7 Figure 3.8 Figure 3.9 True Orlando emperaure curves and prediced response curves wih daa presmoohed using he ransformed B-spline basis, regular B-spline basis funcions and he wavele basis funcions. Black solid curve: rue Orlando weaher curves. Green solid curve: prediced curves presmoohed using he wavele basis funcions. Red solid curve: prediced curves wih daa presmoohed using he ransformed B-spline basis funcions. Purple dashed curve: prediced curves wih daa presmoohed using regular B-spline basis funcions Upsream (Congaree gage) waer level measures for flood even of Ocober Downsream (Cedar Creek gage) waer level measures for flood even of Ocober Downsream (Cedar Creek gage) and upsream (Congaree gage) waer level measures for six flood evens Figure 3.10 Funcional regression based on raw daa curves. Top: inercep funcion ˆα(). Boom: slope funcion ˆβ() Figure 3.11 Funcional regression based on raw daa curves. Blue dashed curve: observed Congaree curve. Green solid curve: observed Cedar Creek curve. Red dashed curve: prediced Cedar Creek curve. 67 Figure 3.12 Funcional regression based on pre-smoohed daa curves. Top: smoohed inercep α(). Boom: smoohed slope β() Figure 3.13 Funcional regression based on pre-smoohed daa curves, obained inercep and slope curves are furher smoohed for predicion. Blue dashed curve: smoohed Congaree curve. Green solid curve: smoohed Cedar Creek curve. Red dashed curve: prediced Cedar Creek curve Figure 3.14 Funcional regression based on pre-smoohed daa curves, obained inercep and slope curves are furher smoohed for predicion. Blue dashed curve: smoohed Congaree curve for Ocober 2015 even. Green solid curve: smoohed Cedar Creek curve for Ocober 2015 even. Red dashed curve: prediced Cedar Creek curve for Ocober 2015 even Figure 3.15 Obained slopes for seven flood evens using funcional regression based on raw daa curves using Cross-validaion idea xiii

15 Figure 3.16 Obained slopes for seven flood evens using funcional regression based on pre-smoohed daa curves using Cross-validaion idea Figure 3.17 Funcional regression based on pre-smoohed daa curves using Bayesian ransformed B-splines mehod. Blue dashed curve: smoohed Congaree curve. Green solid curve: smoohed Cedar Creek curve. Red dashed curve: prediced Cedar Creek curve Figure 3.18 Funcional regression based on pre-smoohed daa curves using Bayesian ransformed B-splines mehod. Obained ˆα() and ˆβ() curves are furher smoohed using he same procedure. Blue dashed curve: smoohed Congaree curve. Green solid curve: smoohed Cedar Creek curve. Red dashed curve: prediced Cedar Creek curve Figure 3.19 Funcional regression based on pre-smoohed daa curves using Bayesian ransformed B-splines mehod. Obained ˆα() and ˆβ() curves are furher smoohed using smoohing splines wih roughness penaly parameer = 0.9. Blue dashed curve: smoohed Congaree curve. Green solid curve: smoohed Cedar Creek curve. Red dashed curve: prediced Cedar Creek curve Figure 3.20 Esimaed α() and β() curves smoohed using Bayesian ransformed B-splines mehod. Top: esimaed (black) and smoohed (red) α() curve. Boom: esimaed (black) and smoohed (red) β() curve Figure 3.21 Esimaed α() and β() curves smoohed using smoohing splines mehod wih roughness penaly parameer = 0.9. Top: esimaed (black) and smoohed (red) ˆα() curve. Boom: esimaed (black) and smoohed (red) ˆβ() curve Figure 3.22 Funcional regression based on pre-smoohed daa curves using he Bayesian ransformed B-splines mehod. Obained α() ˆ and β() ˆ curves are furher smoohed using smoohing splines wih roughness penaly parameers = 0.9 (ˆα()) and 0.5 ( ˆβ()). Blue dashed curve: smoohed Congaree curve. Green solid curve: smoohed Cedar Creek curve. Red dashed curve: prediced Cedar Creek curve xiv

16 Figure 3.23 Esimaed α() and β() curves smoohed using smoohing splines mehod wih roughness penaly parameer = 0.9 (ˆα()) and 0.5 ( ˆβ()). Top: esimaed (black) and smoohed (red) α() curve. Boom: esimaed (black) and smoohed (red) β() curve Figure 3.24 Funcional regression based on pre-smoohed daa curves using he Bayesian ransformed B-splines mehod. Cross-validaion idea is uilized o obain ou-of-sample predicions. Blue dashed curve: smoohed Congaree curve. Green solid curve: smoohed Cedar Creek curve. Red dashed curve: prediced Cedar Creek curve 83 Figure 3.25 Funcional regression based on pre-smoohed daa curves using he Bayesian ransformed B-splines mehod. Obained ˆα() and ˆβ() curves are furher smoohed using smoohing splines mehod wih roughness penaly parameer = 0.9. Cross-validaion is uilized o obain ou-of-sample predicions. Blue dashed curve: smoohed Congaree curve. Green solid curve: smoohed Cedar Creek curve. Red dashed curve: prediced Cedar Creek curve 84 Figure 4.1 Observed poins versus smooh fied curves. Lef: colored curves: observed values for en curves conneced wih linear inerpolaions for each curve. Righ: colored curves: smooh fied curves for all en observaions in he cluser; black dashed curve: esimaed common mean curve for he cluser Figure 4.2 Figure 4.3 Esimaed common mean curve and mean rajecories from muliple ieraions. Black solid curve: esimaed mean curve obained from 1500 ieraions afer 500-ieraion burn-in. Grey dashed curves: esimaed common mean rajecories from 100 ieraions percen credible inervals, he esimaed curves, and he esimaed common mean curve for five observaions. Orange doed curves: poinwise 95 percen credible inervals for he curves. Red solid curves: median fis of he raw curves from he poserior disribuion. Gray dashed curves: median of he esimaed common mean curve obained from he enire chain afer a 500- ieraion burn-in. Blue riangles: rue observed values for he curves xv

17 Figure percen credible inervals, he esimaed curves, and he esimaed common mean curve for five observaions. Orange doed curves: poinwise 95 percen credible inervals for he curves. Red solid curves: median fis of he raw curves from he poserior disribuion. Gray dashed curves: median of he esimaed common mean curve obained from he enire chain afer a 500- ieraion burn-in. Blue riangles: rue observed values for he curves xvi

18 Chaper 1 Inroducion 1.1 Lieraure Review This secion offers a brief lieraure review of funcional daa analysis (FDA) and some funcional daa applicaions. We sar wih a brief inroducion of he origin of funcional daa analysis, and hen discuss he developmen of he subjec relaing o our research. Origin The idea of viewing sequences of ime series daa ha show some auocorrelaed srucure, eiher in he curve iself or in he error erms, was firs proposed by Ramsay (1982), in his paper iled When he Daa are Funcions. Prior o ha, researchers may have encounered daa in he form of curves frequenly, such as children s growh curves in biomedical sudies, for example. However, hey were mosly reaed wih mulivariae daa analysis mehods by viewing he measuremens as separae variables across ime. However, when daa are inrinsically coninuous and he covariance beween response values mus be considered, employing classical mulivariae mehods requires dealing wih he covariance marix, which would resul in models wih huge dimensions for a moderae number of variables. To avoid his issue, oher researchers explored using groups of funcions o represen he curves, ye heir approaches were limied by he lack of flexibiliy of parameric funcions. Following Ramsay and Dalzell (1991), he erm funcional daa analysis was coined and began o be widely used. Ramsay and Dalzell (1991) provided a new perspecive for 1

19 perceiving coninuous daa curves. Insead of selecing a limied number of variables o represen he daa, Ramsay and Dalzell viewed he curves hemselves as individual observaions, where each one is perceived as a funcion or mapping ha akes values from a cerain domain (usually a ime inerval) o some range ha lies wihin he scope of ineres for he response in he sudy, where he funcions vary across he daa se. Thus, he informaion abou correlaion across any ime inerval is conained in he funcion iself. In addiion, i also enables he exploiaion of informaion hidden in higher-order derivaives of he original funcional daum. Such informaion is ofen considered vial for undersanding he behavior of he curves in pracice. In sum, he main difference beween classical mulivariae analysis and FDA is ha no only are he measured poins perceived as variables on a coninuous domain, bu he funcional iself is varying oo (Ramsay 1982). On he oher hand, similarly o classical mulivariae daa analysis, researchers can sill analyze daa by viewing a group of curves as a se of daa. Sraighforward saisical analysis of funcional daa include bu are no limied o: classificaion and clusering, regression analysis, noise reducion and predicion. Developmen Over he las hree decades, funcional daa analysis has been given more aenion by researchers in biomedical fields. I has become common o analyze or inerpre curves, or even images and heir paerns from his new perspecive. Wih he developmen of high-hroughpu echnology, daa can be measured over a very dense grid of poins on a coninuous domain, which makes funcional daa more prevalen. One useful preliminary sep for analyzing funcional daa is daa smoohing. Researchers may choose o do smoohing before carrying ou furher analysis, or could use raw curves for inference direcly. However, smoohing will ofen resul in an improvemen in he accuracy of furher saisical analyses, such as daa classificaion. 2

20 For insance, Hichcock, Casella and Booh (2006) proved ha a shrinkage smooher effecively improves he accuracy of a dissimilariy measure beween pairs of curves. Hichcock, Booh and Casella (2007) showed ha smoohing usually produces a more accurae clusering resul, especially when a James-Sein-ype shrinkage adjusmen is applied o linear smoohers. Popular daa smoohing mehods include basis fiing, regression splines, roughness penaly-relaed smoohs, kernel-based smoohers, and smoohing splines. Among hese caegories, he regression spline is he mos widely used mehod for daa smoohing. The regression spline mehod refers o he funcional daa fiing approach employing basis funcions called splines o form a design marix as in classical regression. The fied curve is obained via he usual leas squares (or weighed leas squares) mehod, where he weigh marix usually involves he reciprocal of he correlaion srucure of he error erms. Knos someimes spli he domain ino pieces in order o fi daa piece by piece via usual regression mehods. Hence deermining he appropriae number and locaions of he knos is a problem o be solved prior o or along wih he daa smoohing procedure. Polynomial splines and B-splines (de Boor 2001) are wo examples ha require selecion of knos prior o daa smoohing. Currenly exising kno selecion mehods include Bayesian adapive regression splines (BARS) (DiMaeo, Genovese and Kass 2001), which uses a Bayesian model o selec an appropriae kno sequence using he daa hemselves. This approach employed he reversible jump Markov chain Mone Carlo (RJMCMC) Bayesian modeling scheme ha is capable of deermining he number of knos and heir locaions simulaneously (Green 1995). Anoher mehod wih a similar aim is kno selecion via penalized splines (Spirii, Eubank, Smih and Young 2008). Oher basis sysems include bu are no limied o he Fourier basis and he wavele basis, for which leas squares or weighed leas squares fiing may also be employed. I should be noed ha for differen funcional daa, he chosen basis sysems 3

21 mus individually reflec he characerisics of he various funcional observaions. Currenly here is no such unified basis sysem ha can be applied auomaically o a large collecion of curves for accurae daa fiing (Ramsay and Silverman 2005). Roughness penaly-based daa smoohers are smoohing mehods ha use a basis sysem, along wih a penaly erm which usually consiss of a uning parameer and a measure of roughness of he fied curves, o obain fis ha balance bias and sample variance. The roughness penaly erm is usually measured by he squared derivaive of some order of he curve inegraed over some coninuum. Ofen he fourh-order derivaive of he fied curve is sufficien, as i penalizes he fied curve and is corresponding firs and second derivaive for no being smooh. The uning parameer is ofen chosen wih some daa-driven mehod prior o he fiing procedure. Some of he mos popular approaches are he cross-validaion (CV) mehod or generalized cross-validaion (GCV) mehod (Craven and Wahba 1979). Deailed comparisons of he wo aforemenioned mehods are given in Gu (2013). Among differen roughness penaly mehods, he smoohing spline approach is mos ofen used. I uses a sum of weighed squared residuals, plus a second-order penaly erm as he objecive funcion. The weigh funcion is he inverse of he covariance marix of he residuals. In realiy, an esimae of he inverse of ha marix is obained o replace he weigh marix in he objecive funcion. de Boor (2001) presened a heorem which saes ha he aforemenioned objecive funcion is minimized if a cubic spline is applied wih knos locaed on each of he measured poins. Such a mehod is called cubic smoohing spline. This mehod avoids he problem of locaing he kno sequence and also ensures sufficien knos in dense areas of he curves, bu would resul in an ourageously huge model dimension when he daa are measured over a fine grid on he domain. Besides, when he daa srucure is simple, such a mehod leads o severe over-fiing. Due o such a drawback, Ramsay and Silverman (2005) argued ha fewer knos could be appropriae when he B-spline basis is employed in 4

22 he smoohing spline framework, wih he penaly erm conrolling he smoohness of he fied curves. Mehods like hese are called penalized spline mehods. Then, however, he kno locaion problem reurns. The mos common soluion is equal spacing of he knos (e.g., Ramsay and Silverman 2002), so ha he only problem is o deermine he number of knos o be placed on he domain. Obviously, such a mehod is only good for relaively simple srucured curves wih homogeneous behavior. For he general roughness penaly approach, he penaly erm is no resriced o be he aforemenioned derivaive measure of he original curves; oher forms of penalies such as he squared harmonic acceleraion penaly may also be plausible. Finally, he kernel-based fiing approaches are commonly used nonparameric daa smoohing mehods. The esimaed curve evaluaed a each individual poin is sill represened by a weighed average of he observed response values measured a differen poins on he domain. The main difference is ha he weighs are now deermined by pre-specified kernel funcions. The bandwidhs of he kernels conrol he value of he weigh a differen poins. Popular kernel funcions include he uniform, he quadraic and variaions of Gaussian kernels ha are posiive only over pars of he domain. This mehod falls ino he caegory of localized leas squares smoohing mehods, since a each measured poin, only some (no all) of he observed values are used for esimaion of he curve. Nadaraya and Wason (Nadaraya 1964; Wason, 1964) proposed a way o sandardize he specified kernel funcion, resuling in a uni sum weigh funcion. Gasser and Müller (1979, 1984) furher proposed a kernel-inegral based weighing funcion ha possesses good asympoic properies and compuaional efficiency. When he funcional daa is sparse in naure, so ha he radiional funcional daa smoohing mehods are incapable of providing desirable resuls, mehods based on mixed effecs models in longiudinal analysis have been developed for smoohing 5

23 sparse funcional daa, see, e.g, Saniswalis and Lee (1998), Rice and Wu (2001), Yao, Müller and Wang (2005). Fiing mehods based only on mixed effecs models usually employ he EM algorihm o esimae imporan parameers of ineres. However, as poined ou by James, Hasie and Sugar (2000), his approach could make he parameer esimaes highly variable when he daa is very sparse. Hence, hey proposed a reduced rank mixed effecs model, which combines funcional principal componens wih he mixed effecs model by imposing several orhogonaliy consrains on he design marices. An alernaive approach incorporaes he mixed effecs models wih he Bayesian echnique, see Thompson and Rosen (2008) for deails. Researchers are ofen ineresed in exploring he paerns of a group of curves insead of one individual curve. One popular sub-area in funcional daa analysis for modeling paerns among groups of curves is funcional regression. Funcional regression is an exension of he classical regression model wih vecor-valued response and vecor-valued covariaes. I allows for eiher he response or he predicor(s) or boh o be coninuous funcions. Some reviews of funcional regression can be found in Wang, Chiou and Müller (2016). The case of boh he response variable and predicor variable(s) being funcions is he mos sraighforward exension of classical linear regression. Funcional regression models in his caegory include he concurren model, in which he response value relaes o he funcional covariae value a he curren ime only, and he funcional linear model (FLM), in which he response value a any ime poin is relaed o he enire covariae curve. The former model is usually esimaed wih a wo-sep procedure where in he firs sep, an iniial esimae of he parameer funcions are obained poinwise via ordinary leas squares, and hen he esimaed parameer funcions are furher smoohed via eiher basis expansion, smoohing-splines, or oher fiing mehods (Fan and Zhang 1999). Oher researchers (Eggermon, Eubank and LaRiccia 2010; Huang, Wu and Zhou 2002) have proposed one-sep mehods o fi he model using basis approximaions. The FLM model, on 6

24 he oher hand, was firs inroduced in Ramsay and Dalzell (1991), in which esimaion of he parameer surface was obained via a penalized leas squares approach. A review of funcional linear models can be found in Morris (2015). On he oher hand, subsanial sudies of funcional regression models have focused on he case of a scalar response and funcional covariae. For his case, an easy approach is o smooh boh he funcional covariae and is corresponding coefficien wih he same se of basis funcions, say, B-splines, he Fourier basis or even smoohing splines (Cardo, Ferray and Sarda 2003). By doing ha, he smoohed funcional linear model reduces o a classical linear regression. There are also numerous exensions of funcional regression o funcional generalized linear models. Boh he cases in which he link funcion is known or is unknown have been subsanially sudied, e.g., (James 2002; Chen, Hall and Müller 2011). Funcional regression has also been exended o he nonlinear case, where nonparameric smoohing is applied o funcional predicors, see, e.g., Ferray and Vieu (2006). Anoher popular funcional daa applicaion area ha deals wih groups of curves is funcional cluser analysis. Similar o radiional cluser analysis, funcional clusering ypically involves radiional hierarchical, pariioning or model-based clusering mehods on he raw or sandard disance marices calculaed based on he se of observed curves, he esimaed basis funcion coefficiens, or he principal componen scores. See Abraham e al. (2003), James and Sugar (2003), Chiou and Li (2007) and Jacques and Preda (2014) for some sudies in his area. A brief summary of curren approaches for funcional daa clusering can be found a Wang e al. (2016). 1.2 Ouline The main focus of his disseraion is on funcional daa smoohing mehods and corresponding applicaions such as funcional regression and clusering. Wih he developmen of echnology, i is now becoming more prevalen o collec daa ha are 7

25 inrinsically coninuous, and daa smoohing as a preliminary daa analysis sep has araced considerable aenion. I is repored in Ullah and Finch (2013) ha more han 80 percen of funcional daa applicaion papers under consideraion uilized some kind of daa smoohing prior o furher analysis. However, daa smoohing mehods being considered sill apply he classical ones o differen ypes of observaions: regression splines, B-splines, smoohing splines, model-based mehods, kernelbased approaches, ec. There is a lack of a more flexible mehod ha could be adaped auomaically o a broader range of curve forms. We herefore aim o fill his gap by proposing relaed mehods ha could be implemened more flexibly, and o examine subsequen impacs on oher funcional daa inferences. In Chaper 2, we propose an alernaive o he radiional scheme for daa fiing, by generalizing he conceps of ime warping beyond daa alignmen o curve fiing. We uilize Bayesian modeling of he ime warping funcion o obain a se of ransformed splines ha can flexibly model differen shapes of observed curves. We show via simulaions and an applicaion o a real daa se ha our proposed approach can achieve greaer fiing accuracy when compared wih oher popular fiing mehods. In Chaper 3, we discuss he impac of applying our proposed smoohing mehod on he se of raw funcional curves, he funcional response or he funcional covariaes as a pre-smoohing operaion prior o funcional regression or clusering and compare he impac of our mehod on clusering and predicion accuracies wih oher popular fiing and smoohing approaches. Chaper 4 exends he daa smoohing approach in Chaper 2 o he sparse daa scenario. We incorporae our ime warping scheme wihin he Bayesian framework, and we uilize a popular random mixed effecs model o fi sparse daa, and we propose an add-on sep during each ieraion of he Bayesian simulaion o obain smooh fis of he sparse curves using our approach. 8

26 Chaper 2 Bayesian funcional daa fiing wih ransformed B-splines Summary: Daa fiing is of grea significance in funcional daa analysis, since he properies of he fied curves direcly affec any poenial saisical analysis ha follows. Among many currenly popular basis sysems, he sysem of B-spline funcions developed by de Boor (2001) is frequenly used o fi non-periodic daa, due o is flexibiliy inroduced by he knos and he ieraive relaionship beween basis funcions of differen orders. Ye ofen he B-splines approach requires a huge number of basis funcions o produce an accurae fi. Besides, when he inrinsic srucure of he individual funcional daum is no apparen, i could be difficul o deermine he appropriae basis sysem for daa fiing. Moivaed by hese facs, we develop an approach ha fis well wihou requiring inspecion of he raw curve, while also conrolling model dimensionaliy. In his paper, we propose a Bayesian mehod ha uses ransformed basis funcions obained via a domain-warping process based on he exising B-spline funcions. Our mehod ofen achieves a beer fi for he funcional daa (compared wih ordinary B-splines) while mainaining small o moderae model size. To sample from he poserior disribuion, he Gibbs sampling mehod augmened by he Meropolis-Hasings (MH) algorihm is employed. The simulaion and real daa sudies in his aricle provide compelling evidence ha our approach beer fis he funcional daa han an equivalen number of B-spline funcions, especially when he daa are irregular in naure. 9

27 2.1 Inroducion Funcional daa analysis refers o he class of saisical analyses involving daa ha are colleced on a se of poins drawn from a coninuum. The poins a which daa are colleced are usually densely locaed over he coninuum so ha he curvaure shape is capured properly. Funcional daa fiing hen refers o some represenaion of he funcional observaions. Considering ime-varying curves on 2-dimensional space, he model represenaion could refer o eiher: a linear combinaion of basis funcions, fi, for example, via he leas squares approach (e.g., de Boor 2001; Schumaker 2007); a linear combinaion of basis funcions, augmened wih erms consising of a uning parameer ha conrols he smoohness of he fied curves and an inegral of he funcional ouer producs, such as he roughness penaly smoohing splines approach (de Boor 2001); or a moving average weighed by some kernel-based weigh funcion, as in kernel smoohing of funcional daa. Research in funcional curve fiing mosly follows he hree aforemenioned pahs ha aim o deermine adequae represenaion of he funcional curves. Wih parameric fiing mehods, mos exising approaches are ailored o he researcher s observaion of he daa curves. In oher words, one has o visually examine he curves prior o daa smoohing in order o more accuraely fi he daa. For insance, o deermine which basis o use in he leas squares fiing approach, he researchers mus know he smoohness and differeniabiliy of he original curves or underlying rue curves as well as heir derivaives. Sufficienly smooh and well behaved curves can usually be represened well wih a polynomial basis or regular splines wih knos evenly disribued on he ime axis. Curves ha are periodic would be beer off fied wih he Fourier basis sysem. Smooh ye more irregular curves may be described wih he more popular and commonly used B-spline basis. Wih highly spiky or irregular curves, he wavele basis is a suiable choice. If he B-spline basis sysem is chosen in order o depic some local feaures of he curves, eiher via he leas 10

28 squares approach or he roughness penaly approach, preliminary knowledge abou he curves appearances is also essenial o deermine wheher muliple knos should be placed a cerain locaions. When he daa curve is inrinsically coninuous ye changes rapidly in muliple locaions, a B-spline basis migh no be a wise choice. This is because muliple knos mus be placed a several locaions o depic hose dramaic changes accuraely, or else i may resul in severe under-fiing. This would induce anoher poenial issue: The oal number of basis funcions used o represen he daa may become undesirably large and he model may become much more complex han desired. In shor, he accuracy of he model fi is achieved only by sacrificing model simpliciy. Triggered by hese findings, we wan o solve he following problems of ineres simulaneously: To propose a mehod ha would be appropriae for boh well-behaved and irregular daa curves. To improve daa fiing wih a fixed number of basis funcions. In he saisical lieraure on daa smoohing, one noices ha successful mehods have been developed ha consruc new basis sysems or adapive splines o adjus for differen shapes of he curves (e.g., Hasie, Tibshirani and Friedman 2009), ye here has been lile focus on improving or ransforming exising basis funcions for greaer flexibiliy. In his chaper, we propose a Bayesian fiing mehod ha adjuss B-spline basis funcions o accommodae daa of a differen naure. This approach avoids daa inspecion prior o smoohing and fis uniformly well for smooh, spiky, periodic or irregular daa curves. To be more specific, we propose an inverse-warping ransformaion on our pre-specified basis funcions ha les he daa deermine he shape of he basis funcions as well as heir corresponding coefficiens. The res of chaper 2 is organized as follows: In secion 2.2, we briefly review he conceps of B-spline basis and ime warping. In secion 2.3, we discuss our 11

29 Bayesian model ha generalizes he usage of basis funcions o more han one shape of daa curves. Secion 2.4 describes he reversible jump Markov Chain Mone Carlo procedure we use in he simulaions o opimally deermine he number and locaions of he knos ha connec piecewise polynomials in B-splines for comparison purpose. Secion 2.5 compares simulaion resuls of our proposed mehod wih four compeing mehods: B-splines wih fixed knos; B-splines wih opimally seleced knos; he wavele basis; and he Fourier basis. In secion 2.6, we carry ou our analysis on real daa and compare our fiing resuls wih oher popular mehods. Finally, secion 2.7 includes some conclusion and discussion of our mehod. 2.2 Review of Conceps B-splines The popular B-spline basis funcion sysem was developed by de Boor (2001). These funcions are a specific se of spline funcions ha share he propery wih regular splines of being piecewise polynomials conneced via some knos on he ime axis. The main difference is ha here is a cerain ype of recursive relaionship beween B-spline basis funcions of differen orders, making i possible o derive higher-order funcions once some lower-order basis funcions are known. Furhermore, one can represen any spline funcion as a linear combinaion of hese B-spline basis funcions. We begin wih assuming he following relaionship beween he daa and he rue signal: y( i ) = x( i ) + ɛ i, i 0,..., M 1 where i is he ih ime poin, y() is he curve observed a T, hence y( i ) is he observed value a ime i, x( i ) is he rue underlying funcion x evaluaed a ime i, ɛ i is he error erm caused by measuremen error or oher unexplained variaion, and M is he oal number of measured poins. Wih a se of specified basis funcions, i is common o esimae x() by represening i as a weighed sum of hese basis 12

30 funcions φ evaluaed a : n b x() = φ j ()c j, i 0,..., M 1 j=1 where φ j () is he jh basis funcion evaluaed a ime, c j is he coefficien corresponding o he jh basis funcion, and n b is he oal number of basis funcions used. To obain an esimae for x(), one only needs o deermine he basis funcions and hen esimae he coefficien values c j via eiher he leas squares approach, weighed leas squares approach, roughness penaly approach, ec. In paricular, he φ j () s in he B-spline basis sysem are piecewise spline funcions of order r, conneced smoohly a he ime poins where he knos are locaed, and defined over he enire region from 0 o M 1. Denoe he kh order piecewise spline in he inerval [ i, i+1 ) as B i,k. Then he B-spline basis funcions are consruced as follows: 1, if i < i+1. B i,0 () = 0, oherwise. and B i,k () = i B i,k 1 () + i+k B i+1,k 1 () i+k 1 i i+k i+1 As we can see from he recursive relaionship above, each basis funcion evaluaed a ime, B i,k (), could be obained given he knowledge of he values of is neighbors, B i,k 1 () and B i+1,k 1 (). Wih such a consrucion mehod defined, each φ j ( ) is resriced o be posiive only on he minimal number of subinervals divided by he knos. To be more specific, B-spline basis funcions have he so-called compac suppor propery, which means ha each φ j ( ) can be posiive only over no more han k subinervals. This propery guaranees he compuaional speed of he fiing algorihm o be O(n b ) due o he M n b band-srucured model marix Φ() (having enries being he B-spline funcions evaluaed a differen ime poins = ( 0, 1,..., M 1 ) ) no maer how many knos are included in he inerval ( 0, M 1 ) (Ramsay and Silverman 2005). 13

31 One may deermine he order of he spline funcions based on he number of derivaives of he original curves ha we require o be smooh. In pracice, order-four polynomials are usually adequae o fi curves ha are inrinsically smooh. Noe ha he placemen of he knos on he ime axis plays a vial role in he fiing performance of B-spline basis. If he number of knos is oo small, he B-spline basis does no gain much flexibiliy in fiing curves beyond wha a polynomial basis would have. However, increasing n b via using a greaer number of knos migh no always enhance he fi of he B-spline approximaion o he daa. In fac, Ramsay and Silverman (2005) imply ha an improved fi would be achieved when n b is increased only by adding a new kno o he exising sequence of knos, or by increasing he order of he spline funcions while leaving he posiions of he knos unchanged. Knos ha are poorly locaed may influence he fi badly by emphasizing mild curvaures oo much, and neglecing local areas ha change rapidly in he curves. Ramsay and Silverman (2005) sugges ha one may locae an equally spaced sequence of knos on he ime axis if he daa poins are roughly balanced over he inerval ( 0, M 1 ). However, if he daa curve is irregular in he sense ha local feaures are apparen across some oherwise smooh overall baseline curves, issues such as local over-fiing or underfiing may appear. Several ways have been proposed o deal wih he problem. For insance, Bayesian approaches ha uilize reversible jump Markov Chain Mone Carlo (RJMCMC) o deermine he number and locaions of knos were developed by Denison, Mallick and Smih (1998) and DiMaeo e al. (2001). A wo-sage procedure ha deermines he number of knos firs, hen selecs he locaions of he knos, was proposed by Razdan (1999). An adapive kno selecion mehod based on a muliresoluion basis was developed by Yuan, Chen and Zhou (2013). For comparison purposes, we adop a slighly adjused version of he RJMCMC procedure described in DiMaeo e al. (2001) in our simulaion o locae he knos, in order for he B-splines o fi opimally. 14

32 Time Warping for Daa Regisraion The radiional use of a warping funcion is o ransform he ime axis in order o align a group of funcional curves. Typically such curves are measured a he same ime poins, ye imporan landmarks may occur a differen posiions in chronological ime. In order o subsequenly carry ou cluser analysis or some oher ype of saisical analysis, one may deermine g landmark ime poins { 1, 2,..., g} ha are viewed as sandard imes when cerain evens occur. Then he ime axis for each individual observaion is disored so ha he occurrence imes of hose landmark evens are sandardized. In oher words, for he sh individual among a oal of S member curves, a ime ransformaion W s is imposed. Assume ha he ime poins a which hose landmark evens occur for curve s are { s 1, s 2,..., s g}. Then we have W s ( ) ha saisfies: W s ( 0 ) = 0, W s ( M 1 ) = M 1, s, and W s ( s i ) = i, i, s. Also, W s is a monoonic ransformaion ha mainains he order of he ransformed ime sequence. These W s funcions are called ime-warping funcions. Applying he warping funcions o he measured ime poins produces he warped sh curve: x s() = x s (W s ()), T which has he same occurrence imes for hose landmarked evens as does he sandard ime sequence. Given such properies of he warping funcions, we can hen obain heir inverse funcions W 1 s axis and x s () on he verical axis. () by simple inerpolaion wih W 1 () on he horizonal s 15

33 2.3 Bayesian Model Fiing for Funcional Curves Moivaed by he idea of ime warping o regiser curves, in order o improve he performance of curve fiing based on he se of pre-specified basis funcions, we impose some ransformaion of he ime axis for each of he basis funcions o obain a se of inversely-warped basis funcions. This is done no o align he basis funcions, bu o provide some flexibiliy for hem o improve he ulimae fi o he funcional daa. The ransformaion funcion, which is denoed as W ( ), has o be monoone o preserve he order of he ransformed ime poins. We will employ ideas from he Bayesian mehod of Cheng, Dryden and Huang (2016) o obain he warping funcions. Assume ha all he funcional daa are sandardized so ha heir domains are [0,1] prior o carrying ou furher analysis. We use he noaion = { 0, 1, 2,..., M 1 } o denoe he parameerized ime span; hus 0 = 0 < 1 < < M 1 = 1. Then he ransformaion funcion W ( ) or warping from he original ime sequence o a mapped new ime sequence w could be obained as follows: firs we generae a sequence of M 1 numbers p = {p 1, p 2,..., p M 1 } beween 0 and 1, such ha M 1 i=1 p i = 1. Then calculae he cumulaive sum of he generaed sequence p, we obain {p 1, p 1 +p 2,..., p 1 +p p M 2, 1} which is a monoone sequence from p 1 o 1. Finally, we define W ( ) o be he mapping W () = {0, p 1, p 1 + p 2,..., M 2 i=1 p i, 1}. Following Cheng e al. (2016), o model he prior disribuion of such a ransformaion on ime, we view {0, p 1, p 1 + p 2,..., p 1 + p p M 2, 1} as a sepwise cumulaive disribuion funcion (CDF) in he sense ha he heigh of he ih jump in he CDF graph corresponds o he value of p i in p. Therefore we could use he Dirichle disribuion as he prior disribuion o model he heighs of all M 1 jumps in he sep funcion. Tha is: p = (p 1, p 2,..., p M 1 ) Dir(a), 16

34 where a = (a 1, a 2,..., a M 1 ) is he vecor of hyperparameers in he Dirichle disribuion ha conrols he amoun of warping of he ime poins. Grea discrepancies in he a values lead o significanly differen means for he heighs of he jumps in he cdf. When all he elemens in a are equal, he elemens magniude conrols he deviaion of he ransformed ime poins from he original ime poins. Smaller values in a correspond o greaer deviaion beween he original ime span and he warped ime span. Therefore, a serves as a uning parameer ha influences he amoun of warping of he ime axis. In pracice, due o compuaional concerns, one may choose o generae M M jumps via he Dirichle disribuion. One could label w = { w0, w1,..., wm 1 } = {0, p 1, p 1 + p 2,..., p 1 + p p M 2, 1}. If M = M, one could define a differen mapping W j : wj for he jh basis funcions as described above. Le Φ() be he M n b marix wih he columns being a se of n b pre-deermined basis funcions evaluaed a, and Φ () be a marix of he same dimensions wih he columns being he se of n b inversely warped basis funcions measured a. Noe ha we inversely warp each basis funcion differenly, hence from now on, we use W, T W and P o denoe he vecor of mappings, he vecor of ransformed ime sequences and he vecor of incremen vecors for all basis funcions, i.e., W = {W 1, W 2,..., W nb }, T W = { w1, w2,..., wnb } and P = {p 1,..., p nb }. To obain Φ () once Φ() is given, we assume he following relaionship: Φ() = Φ (W()) = Φ (T W ). Then if we regard Φ( ) and Φ ( ) as funcions ha are applied on he same ime vecor, we have Φ = Φ W. In wha follows, we have: Φ () = Φ (W 1 (T W )) = Φ (W 1 (W())) = Φ (W(W 1 ())) = Φ(W 1 ()). Noe ha each Wj 1 (), for j = 1,..., n b, can be evaluaed by drawing he curve of 17

35 W j () on he x-axis and on he y-axis, hen doing linear inerpolaion o obain he esimaed values of he curve a vecor. Hence, Φ () can be esimaed. In he case ha M < M, one could define a new ime vecor new wih a smaller lengh M ha mimics he behavior of. One could hen apply he ransformaion described above o obain Φ ( new ), and approximae he M n b dimensional marix Φ () based on Φ ( new ). Now we are able o give he model for he daa. We sar wih funcional daa y(), where is he aforemenioned sandardized ime vecor. We assume a parameric model for our daa: y() Φ(), P, σ 2 MV N(Φ(W 1 ())d, σ 2 I), where d is he vecor of coefficiens corresponding o hose inversely-warped basis funcions, and σ 2 is he variance of he error erms. I is well known ha he B-spline basis sysem is a powerful ool o fi a variey of ypes of curves. One can always achieve greaer flexibiliy in daa fiing by adding more knos on he ime axis when he order of he basis funcions is fixed, or by increasing he order of he basis funcions while he number and he locaions of he knos are fixed. Eiher approach requires an increase in he oal number of basis funcions o achieve beer accuracy. Bu models ha are overly complicaed are no always desirable. We wan o reduce he dimensionaliy of our model wihou sacrificing much accuracy, when our model dimensionaliy is no small. Hence we use an opional add-on indicaor vecor γ = {γ 1, γ 2,..., γ nb } in our procedure, such ha each of he γ i follows a Bernoulli disribuion wih success probabiliy α i, and γ i = 1 denoes he presence of variable i. To incorporae his variable selecion indicaor γ ino he Bayesian srucure, one mus deermine some prior for he condiional disribuion of d γ. Some of he priors proposed by oher researchers on d γ include ye are no limied o he spike and slab prior in Kuo and Mallick (1998) or he mixure normal prior in Dellaporas, Forser and Nzoufras (1997) and George and 18

36 McCulloch (1993). O Hara and Sillanpaa (2009) compared differen Bayesian variable selecion mehods wih respec o heir program running speed, abiliy o jump back and forh beween differen sochasic sages and effeciveness of disinguishing ruly significan variables from redundan variables. Based on he discussion of O Hara and Sillanpaa (2009), we adop he condiional prior srucure for d γ and σ 2 γ described in he sochasic search variable selecion of George and McCulloch (1993), due o is relaively fas program running speed, ease of γ jumping from sage o sage, and grea power of separaing imporan variables from rivial ones. The model is as follows: d γ MV N nb (0 nb 1, D γ RD γ ), σ 2 γ IG(ν γ /2, ν γ λ γ /2). Here R can be viewed as he correlaion marix of d γ; Therefore, George and Mc- Culloch (1993) sugges choosing R based on one s prior knowledge of he correlaions beween each pair of coefficiens given he informaion in γ. D γ is a diagonal marix ha deermines he scale of variances of differen coefficiens. The model informaion for each MCMC ieraion is sored in D γ. In oher words, he (i, i) elemen in D γ equals s i ζ i, where ζ i is some fixed value and is usually deermined by some daa-driven mehod prior o he MCMC ieraions. On he oher hand, s i depends on model informaion via some pre-specified value c i in he following way: s i = c I(γ i=1) i. To be more specific, he marginal disribuion of d i γ i is given by he following mixure of normal disribuions: d i γ i (1 γ i )N(0, ζ 2 i ) + γ i N(0, c 2 i ζ 2 i ). Noice ha each d i, given γ i, follows eiher N(0, ζ 2 i ) or N(0, c 2 i ζ 2 i ). Then he esimae ˆd i of d i given γ i, follows N(0, σ 2 d i + c 2 i ζ 2 i ) for γ i = 1 and N(0, σ 2 d i + ζ 2 i ) for γ i = 0. Here he σ 2 d i is he variance of he coefficien d i. Therefore, he graph of one of he densiy 19

37 Figure 2.1: Comparison plo of wo normal densiy curves. Dashed curve: densiy of N(0, σ 2 d i + c 2 i ζ2 i ). Solid curve: densiy of N(0, σ2 d i + ζ 2 i ). (c i, σ di /ζ i ) = (5, 1). Figure 2.2: Comparison plo of wo normal densiy curves. Dashed curve: densiy of N(0, σ 2 d i + c 2 i ζ2 i ). Solid curve: densiy of N(0, σ2 d i + ζ 2 i ). (c i, σ di /ζ i ) = (100, 10). funcions superimposed on anoher shows, for each ieraion, how he probabiliy of including he ih basis funcion changes wih differen values of d i. Via observaion of he separaion of he wo densiy curves, one also ges an idea of wheher he prior of d i γ i favors a parsimonious model or a more sauraed model. I is easily derived ha when ˆd i is 0, he probabiliy of including he ih basis funcion 20

38 in he model is given by: γ i = σ2 d i /ζi 2 + c 2 i σd 2 i /ζi Noice ha he formula above depends only on σ 2 d i /ζ 2 i and c 2 i, and as a resul, one may rea he combinaion of σ 2 d i /ζ 2 i and c 2 i as uning parameers ha deermine model complexiy. Therefore, he values of c i and ζ i o be adoped in he simulaions can be deermined by choosing differen combinaions of ˆσ di /ζ i and c i as needed, where ˆσ di /ζ i is he esimaed variance of d i. Figures 2.1 and 2.2 are wo examples of such densiy curves superimposed on anoher, wih (c i, σ di /ζ i ) chosen o be (5, 1) and (100, 10). The ν γ and λ γ ha appear in he inverse gamma prior of σ 2 γ can depend on γ via he size of γ. When ν γ is se o be zero, i reduces o he improper prior σ 2 γ 1/σ 2. Lasly, since he measuremen errors induced in he daa collecion process could be correlaed in a sysemaic way insead of being independen across all ime poins, we also consider he possibiliy of using a more general correlaion marix C o model he associaion among he error erms. We will specifically consider he case when he error erms are correlaed according o a AR(1) model. Tha is: y() Φ(), P, σ 2 MV N(Φ(W 1 ())d, σ 2 C), where 1 ρ ρ 2 ρ (M 1) ρ 1 ρ ρ (M 2) C = ρ (M 1) ρ 1 In his case, we will need a prior for ρ. We propose o use: ρ U[ 1, 1] 21

39 o declare ignorance of he srucure of he correlaion beween adjacen error erms. Assuming ha P, σ 2, ρ, d are independen given γ, and hen we are able o derive he poserior disribuion when he error erms are correlaed according o a AR(1) model: π(p, d, σ 2, ρ, γ y()) = f(p, σ2, ρ, d, γ, y()) f(y()) f(y() P, σ 2, ρ, d, γ) f(p, σ 2, ρ, d γ) f(γ) = f(y() P, σ 2, ρ, d, γ) f(p) f(σ 2 γ) f(ρ) f(d γ) P (γ) { } (de(σ 2 C)) 0.5 σ νγ 2 νγ λ γ exp 2σ 2 exp{ 0.5d (D γ RD γ ) 1 d}i{ 1 ρ 1} exp{ 0.5(y() Φ(W 1 ())d) L L(y() Φ(W 1 ())d)} n b (de(d γ RD γ )) 0.5 k=1 M 1 i=1 p (a ik 1) ik n b j=1 α γ j j (1 α j ) (1 γ j). Here L = (1 ρ 2 ) ρ σ (1 ρ 2 0 ρ 1 0 0, ) ρ 1 where L L is he Cholesky decomposiion of 1 σ 2 C 1 (Jones 2011), p ik denoes he heigh of he ih jump in he ime span of he kh basis funcion p k, a ik is he ih parameer in he vecor a of he Dirichle disribuion for he kh basis funcion, and α j is he probabiliy ha he jh basis funcion is ruly imporan (so ha γ j = 1) in he model. Noe ha we use he subscrip k o emphasize he fac ha a could vary across basis funcions. We also mus rewrie de(c). Leing L = σl, hen: de(c) = de((l L ) 1 ) = (de(l 1 )) 2 22

40 Afer some marix manipulaions and mahemaical inducion, we obain: ρ 1 ρ L 1 = ρ 2 ρ 1 ρ 2 1 ρ 2 0, ρ M 1 ρ 1 ρ 2 1 ρ 2 Therefore, he full poserior disribuion is: { } π(p, d, σ 2, ρ, γ y()) σ M νγ 2 νγ λ γ exp exp{ 0.5d (D 2σ 2 γ RD γ ) 1 d} exp{ 0.5(y() Φ(W 1 ())d) L L(y() Φ(W 1 ())d)} (1 ρ 2 ) 0.5(M 1) I{ 1 ρ 1}(de(D γ RD γ )) 0.5 n b k=1 M 1 i=1 p (a ik 1) ik n b j=1 α γ j j (1 α j ) (1 γ j). Noe ha when one assumes ha here is no apparen auocorrelaion relaionship in he error erms, L is replaced by he ideniy marix, and ρ by 0 in he expression above. Then he poserior disribuion of he jh elemen of γ, given he oher informaion, could be obained from: f(γ j y(), P, σ 2, ρ, d, γ (j) ) = f(γ j σ 2, d, γ (j) ) where γ (j) denoes he curren vecor of γ, excluding he jh elemen. I is obvious ha he poserior disribuion of (γ j oher informaion) is sill a Bernoulli disribuion, and he probabiliy of success is given by: P (γ j = 1 σ 2, d, γ (j) ) = f(σ2 γ j = 1, γ (j) )f(d γ j = 1, γ (j) )P (γ j = 1, γ (j) ) f(σ 2, d, γ (j) ) = u u + v, and similarly, he probabiliy of failure is: P (γ j = 0 σ 2, d, γ (j) ) = f(σ2 γ j = 0, γ (j) )f(d γ j = 0, γ (j) )P (γ j = 0, γ (j) ) f(σ 2, d, γ (j) ) = v u + v, where u is he numeraor of he probabiliy of success, and v is he numeraor of he probabiliy of failure. 23

41 The condiional poserior disribuion of (d oher parameers) is given by: f(d y(), ρ, σ 2, γ) exp{ 0.5(y() Φ(W 1 ())d) L L(y() Φ(W 1 ())d)} exp{ 0.5d (D γ RD γ ) 1 d} { exp 0.5d {Φ(W 1 ()) L LΦ(W 1 ()) + (D γ RD γ ) 1 }d } 2y() L LΦ(W 1 ()). I is obvious ha: where and d y(), p, σ 2, ρ, γ MV N(µ F, Σ), µ F = Σ[Φ(W 1 ()) L Ly()], Σ = [Φ(W 1 ())L LΦ(W 1 ()) + (D γ RD γ ) 1 ] 1. The poserior condiional disribuion of σ 2 is: Thus, f(σ 2 y(), P, ρ, d, γ) =f(σ 2 y(), ρ, d, γ) { σ M νγ 2 exp 1 ( ν 2σ 2 γ λ γ + (y() Φ(W 1 ())d) )} where L L (y() Φ(W 1 ())d) σ 2 y(), P, ρ, d, γ IG(α, β ), α = M + ν γ, 2. and β = 1 ( ) ν γ λ γ + (y() Φ(W 1 ())d) L L (y() Φ(W 1 ())d). 2 24

42 Noe ha he informaion abou basis funcion selecion is conained in he poserior disribuion of γ. We employ he poserior mode of γ as defining he mos desirable poenial model, as described in George and McCulloch (1993). Tha is, we sample from he join poserior disribuion in he following order: p 1, p 2,..., p nb, σ 2, d, γ, p 1, p 2,..., p nb, σ 2,.... We uilize he Gibbs sampler o sample γ, σ 2 and d, since heir condiional poserior disribuions are known and easy o sample from, and we use Meropolis-Hasings mehod wihin he Gibbs sampler o sample P and ρ, since heir condiional poserior disribuions are no in closed form. Due o he resricions on each p k vecor, we use a runcaed normal disribuion as he insrumenal disribuion. Le superscrip (i) represens parameer values sampled a he ih ieraion. For he kh basis funcion, one firs generaes an iniial p (0) k = (p (0) 1k, p(0) 2k,..., p(0) M 1k ) from a Dirichle prior. Then a ieraion i, one samples he elemens of p (i) k one in he following way: 1. Sample he firs elemen in p (i) N(p (i 1) 1k k, p(i) 1k one by, by drawing a random observaion from, σ2)i(0, 2 L 1 u). And hen adjus he las elemen in p (i 1) k, i.e., p (i 1) M 1k, o make he following holds: Denoe he adjused p (i 1) M 1k as p (i) p (i) M 2 1k + p (i 1) jk j=2 + p (i 1) M 1k = 1. M. Here 1k L1 u = p (i 1) 1k +p (i 1) M 1k. If he proposed p(i) 1k, denoed as p (i) 1k, is acceped by he Meropolis-Hasings algorihm, hen he firs and he las elemens in p (i) k are updaed as p (i) 1k are he same as hose in p (i 1). Oherwise, he enire vecor p (i) k p (i 1) k. 2. For j = 2, 3,..., (M 2), sample he jh elemen p (i) jk and p (i) M 1k, respecively, oher elemens remains he same as by drawing a random observaion from N(p (i 1) jk, σ2)i(0, 2 L j u), and adjus p (i) M 1k o make he following holds: M 2 j=1 Sill denoe he adjused p (i) M 1k as p (i) p (i) jk + p (i) M 1k = 1. M. Here 1k Lj u = p (i 1) jk +p (i) M 1k. If he proposed 25

43 p (i) jk, denoed as p (i) jk, is acceped by he Meropolis-Hasings algorihm, hen he jh and he las elemens in p (i) k are updaed as p (i) jk elemens are kep unchanged. Oherwise, he enire vecor p (i) k and p (i) M 1k, respecively, oher remains unchanged. Noe ha for any arbirary ieraion i, each ime we sample, say, he jh elemen, he upper bound of he runcaed normal disribuion is adjused based on he fac ha i has o be less han p (i 1) jk non-negaive adjused p (i) M 1k + p (i 1) M 1k in he vecor. For ieraion i, wih he proposed p jk in p (i) k is given by: a(p (i) jk, p(i 1) jk ) = π(p (i) jk ) π(p (i 1) or p(i 1) jk jk ) + p (i) M 1k, in order o obain a denoed as p (i) jk, he accepance raio q(p (i 1) jk p (i) jk ) q(p (i) jk p(i 1) jk ), where q( ) represens he runcaed normal disribuion N(, σ 2 2)I(0, L j u). Hence: q(p (i 1) jk p i jk) q(p i jk p(i 1) jk ) = Φ( L j u p (i 1) jk σ 2 Φ( Lj u p (i) jk σ 2 ) Φ( p (i 1) jk ) Φ( p σ 2 ) (i) jk σ 2 ) Here Φ denoes he cumulaive disribuion funcion of he sandard normal disribuion. Then i follows ha: log(a(p (i) jk, p(i 1) jk )) = ( L log(π(p (i) j jk )) + log u p (i 1) ) ( (i 1) ) jk p jk Φ Φ σ 2 σ 2 ( ( L log(π(p (i 1) j u p (i) ) ( (i) )) jk p jk jk )) + log Φ Φ σ 2 σ 2 = 0.5(y() Φ (i) (W 1 ())d) L L(y() Φ (i) (W 1 ())d) ( n b M 1 + log(p (i) (a ir 1) L j u p (i 1) ) ( (i 1) ) ir ) + log jk p Φ Φ r=1 i= (y() Φ (i 1) (W 1 ())d) L L(y() Φ (i 1) (W 1 ())d) ( n b M 1 log(p (i 1) (a ir 1) L j u p (i) ) ( (i) ) ir ) log jk p jk Φ Φ, r=1 i=1. σ 2 σ 2 jk σ 2 σ 2 26

44 where Φ (i) (W 1 ()) represens he adjused design marix in he ih sep according o p (i) jk. 2.4 Kno Selecion wih Reversible Jump MCMC The flexibiliy of B-splines for fiing funcional daa is based on he knos on he ime axis. Appropriaely chosen knos could resul in an exremely good fi, while naively chosen knos ha do no reflec he naure of he funcional curve could produce a poorly fied curve. Hence, we will consider boh cases of fixed knos and opimally seleced knos in he simulaion secion. Many mehods have been developed ha focus on selecing knos o improve he performance of B-splines. We will discuss wo approaches proposed by Denison e al. (1998) and DiMaeo e al. (2001) ha uilize Reversible Jump Markov Chain Mone Carlo (RJMCMC) (Green 1995) o simulae he poserior disribuion of (k, ξ), where k denoes he number of inerior knos used o connec B-splines, and ξ refers o he locaions of he k inerior knos on he ime axis in ascending order. RJMCMC is an exension o he Meropolis-Hasings mehod ha uses some proposal disribuion o propose candidae values for he parameers of ineres. However i includes a birh jumping probabiliy, and a deah jumping probabiliy o allow for he possibiliy of dimension change. I is paricularly useful in cases when he rue dimension of he parameers is unknown and simulaneous esimaion of he parameer values is needed along wih a dimension updae; common seings for RJMCMC include variable selecion or sep funcion esimaion. The mehod requires having he deailed balance propery saisfied for each updae o ensure he exisence of a seady sae disribuion. In he general scheme of boh Denison e al. (1998) and DiMaeo e al. (2001), when selecing he number and locaions of he knos in funcional curve fiing, he necessary seps involved in RJMCMC are he birh sep, deah sep and relocaion sep. In he firs wo seps, he chain eiher acceps an increase in 27

45 he dimension of he parameers by having an addiional proposed kno insered ino he exising kno sequence, acceps having one of he knos deleed from he sequence, or rejecs he proposed sae and says a he curren sae. In he relocaion sep, he chain deermines wheher o relocae one of he exising knos or no. Because he underlying rue model is no unique, we assume ha wih appropriaely chosen knos, he observed daa could be adequaely described wih a group of order r B-splines. The daa and he design marix formed by he B-splines evaluaed a hose measured ime poins are conneced via he following relaionship: y() Φ k,ξ (), σ 2 B MV N r+k (Φ k,ξ ()c, σ 2 BI), where c is he rue underlying vecor of coefficiens corresponding o he B-splines in he model, and σb 2 is he variance of he error erms. Here he subscrips in he noaion Φ k,ξ () are used o emphasize how boh he dimension and he values of he B-spline funcions depend on he number and locaions of he knos. Noe ha while independen errors are assumed here, one may cerainly include some auocorrelaion srucure o allow for he possibiliy of correlaed error erms. However, in a wosage procedure, where one firs selecs knos on he ime axis for opimal B-splines performance, and hen uses our Bayesian model described in secion 2.3 o furher improve he fi, he srucure of error erms for boh sages should coincide wih each oher. We use he scheme given by DiMaeo e al. (2001). The prior for k is Poisson, and ξ k follows Dir(1,1,..., 1). The prior for σb 2 could be chosen as inverse gamma or he improper prior π(σb) 2 1/σB. 2 The coefficien vecor has he following condiional prior: c k, ξ, σ 2 B MV N(0, σ 2 BM{Φ k,ξ () T Φ k,ξ ()} 1 ). In each ieraion, he hree aforemenioned probabiliies are given by: b k = g min{1, P (k + 1)/P (k)}, e k = g min{1, P (k 1)/P (k)}, h k = 1 b k e k 28

46 where b k is he probabiliy of jumping from k inerior knos o k + 1 inerior knos. Wheher he chain a he nex ieraion moves o he sae wih an addiional inerior kno depends on he accepance raio. Similarly, e k is he probabiliy of jumping from k inerior knos o he sae wih k 1 inerior knos. Such a move may or may no be made based on he accepance raio. h k is he probabiliy of relocaing one of he exising knos. The main discrepancy beween Denison e al. (1998) and DiMaeo e al. (2001) is ha he firs uilizes a no enirely Bayesian approach, bu raher a quasi-bayesian approach o generae he chain by using leas squares o calculae c values for each proposed updae of (k, ξ). According o DiMaeo e al. (2001), his would end o over-fi he curve o some exen. As a resul, he laer proposed deriving he disribuion of c k, ξ, Y() by inegraing ou σb 2 from he join poserior disribuion of c, σb k, 2 ξ, Y(). Based on our invesigaion, i appears ha even for a parameric model, he poserior disribuion of c k, ξ, Y() is unlikely o be in closed form. For he improper prior on σ 2 B adoped by DiMaeo e al. (2001), he corresponding disribuion is no recognizable. The aforemenioned poserior disribuion becomes a mulivariae non-cenral disribuion only when an inverse gamma prior wih equivalen shape and scale parameer values is employed for σb. 2 In fac, based on our limied simulaion sudies, even for irregular shaped daa curves, wih he maximum number of knos se large enough, boh of he aforemenioned approaches are capable of giving plausible fiing resuls in he sense ha he mean squared error is negligible compared wih he daa informaion. However, our primary goal is o improve he original B-spline fi of he curves, and o compare he improved fi wih he resul obained from our mehod. A model wih B-splines could achieve a perfec fi by using an ourageously large number of knos, bu his would produce a model oo complicaed o be of any pracical advanage. We wan o limi he maximum number of knos used in he procedure, and show in our simulaion sudies ha even wih 29

47 a limied number of knos, he performance of our approach is compeiive wih using B-splines wih a relaively large number of opimally seleced knos. We use leas squares o esimae c in each sep for compuaional ease and also o avoid a disribuion idenifiabiliy issue. Wih a limied number of knos and relaively complicaed funcional daa, he over-fiing concern of his quasi-bayesian approach noed by DiMaeo e al. (2001) would no longer be a problem. The proposal probabiliy for he jump from k knos o k + 1 knos is defined by: q(k + 1 k) = b k k k P B (ξi ξ i ). i=1 Here ξ i is he locaion of he ih kno, ξ i is he proposed new kno, which is cenered around ξ i, and P B ( ) is he proposal disribuion. DiMaeo e al. (2001) claimed ha he simulaion resuls are relaively robus o he choice of proposal disribuion; herefore, we use heir proposed Bea disribuion wih parameers ξ i ν and (1 ξ i )ν. For each exising kno, one firs randomly generaes a realizaion from he proposal disribuion ha is cenered a ha kno. Pick a random one kno, say, ξ p; calculae he aforemenioned probabiliy; calculae he accepance raio; and compare he accepance raio wih a randomly generaed uniformly disribued realizaion o deermine wheher he proposed kno cenered a ξ p is acceped or no. The proposal probabiliy for jumping from k o k 1 knos is given by: q(k 1 k) = e k /k. Similarly o he birh sep, one randomly picks an exising kno and uses he accepance raio o deermine wheher or no he seleced kno should be deleed from he sequence. The proposal probabiliy for relocaing one of he exising kno is: q(ξ p ξ p ) = h k 1 k P R(ξ p ξ p ). We se P R ( ) o be he same as P B ( ). If such a move is acceped, ξ p will be replaced by he proposed ξ p. Therefore, in each sep, he ransiion probabiliies of moving 30

48 from k o k 1 knos or from k o k + 1 knos are jus he aforemenioned proposal probabiliies imes he corresponding accepance raios. See DiMaeo e al. (2001) for deailed expression of accepance raios. Noe ha when calculaing he accepance raios in each sep, c needs o be updaed along wih he proposed (k, ξ) o mach he dimensions. 2.5 Simulaion Sudies In his secion, we simulae some irregularly shaped funcional daa. We carry ou our approach and compare our approach o several alernaive fiing mehods: (1) he original B-splines wih fixed equally spaced knos; (2) B-splines wih opimally seleced knos; (3) he Fourier basis sysem; and (4) he orhonormal se of wavele basis which include he Haar faher wavele and shifed and scaled moher waveles. We consider simulaing a group of observaions ha come from a mixure of a periodic signal, a smooh signal, and a spiky signal. In order o do ha, we deermine a number of Fourier basis funcions o be employed, randomly generae normal coefficiens for hese, and add independen Gaussian errors o form he periodic signal. We use a similar approach o generae a signal based on B-splines. The spiky daa is generaed by firs randomly pariioning he ime domain ino several inervals, and evaluaing he values of a baseline curve a he boundaries of hose inervals. The baseline curve is obained from linear combinaions of B-splines plus error erms. The widh of each inerval represens eiher he widh of a spike in he curve, or he spacing beween a pair of spikes. We herefore use wo exponenial disribuions wih differen means o model he spacings. The means of he exponenial disribuions are chosen so ha he simulaed curve has a moderae number of spikes, wih appropriae spacings beween hem. The heigh of each spike is deermined via he baseline value a he cener of each spike plus a randomly generaed U(0, 30) observaion. Then our mixed funcional daa is obained by aking a weighed average of he hree curve ypes. 31

49 Some examples are described below. Example 2.1. In his example we employ 11 Fourier basis funcions and 20 order-5 B-spline basis funcions wih equally spaced knos o generae our simulaed daa. The B-splines are measured over 500 equally spaced poins in he domain [1, 5]. Their corresponding coefficiens are generaed independenly as Gaussian wih mean 0 and sandard deviaion 2. Random errors from he same disribuion are added o boh curve ypes. The baseline curve for he spiky daa is he rue signal curve generaed by B-splines, plus independen N(0, 4) errors. The widhs of he spikes and he spacings beween each pair of spikes are exponenial wih respecive means 0.4 and 2/15. The weighs for he hree ypes of curves are 0.1, 0.1 and 0.8. Figure 2.3 shows he simulaed noisy observed curve (solid) superimposed on he rue signal curve (dashed). We measure he signal o noise raio by calculaing he raio of he sandard deviaion of he rue signal versus he sandard deviaion of he error funcion. From invesigaion, i appears ha roughly 2.5 o 4.5 are appropriae values of he aforemenioned raio, so ha he rue signals are no overwhelmed by noise. Afer sandardizaion of he simulaed daa curves o scale hem o exis wihin he domain [0, 1], we sar wih order-5 B-splines, wih 12 equally spaced knos on [0, 1], and carry ou our procedure o fi he daa wih ransformed B-splines. Figure 2.4 is he race plo of mean squared error over 1000 ieraions; we see ha he chain converges rapidly afer a few ieraions even hough he simulaed curve is irregular wih several spikes. Therefore, we employ 1000 ieraions for mos of our seings, since i is sufficien for us o sample from he poserior disribuion. Figure 2.5 shows a se of 15 ransformed B-splines based on he 15 order-5 B-splines wih a sequence of 12 equally spaced knos over [0, 1]. Each spline is wised somewha o accommodae he irregular shape of he curve, ye he domains over which he ransformed splines and he original ones are posiive are roughly he same. In order o obain smooh ransformed splines while also keeping flexibiliy o fi irregularly shaped curves, one 32

50 y() Figure 2.3: True versus fied simulaed curves. Dashed spiky curve: simulaed rue signal curve. Solid wiggly curve: corresponding observed curve. may wan o choose he values of he elemens in parameer a o be relaively small, ye no so small ha he wiggles in he splines look like sep funcions wih sudden jumps. In order o smooh our fied curve, we average he obained curves from MCMC ieraions 200 o 1000, roughly afer he area where he MSE begins o level off on he graph. This implied a burn-in period of 200 ieraions. Figure 2.6 and 2.7 are comparison plos ha superimposes several fied curves obained via differen mehods on he rue signal curve (black solid). We use TB o denoe our approach for funcional daa fiing, B o denoe he B-splines approach 33

51 MSE ieraions Figure 2.4: Mean squared error race plo of 1000 MCMC ieraions spline value Figure 2.5: Example of se of 15 ransformed B-splines obained from one MCMC ieraion wih equally spaced knos, SKB o denoe he B-splines approach wih knos seleced via RJMCMC, and Wave and Fourier represen fiing via he wavele basis or he Fourier basis. Here we are using 16 insead of 15 wavele basis funcions, since he number of wavele basis funcions employed in a fiing procedure mus be 2 res, where res is he resoluion value. STB denoes our approach wih reduced dimension. More specifically, o obain he STB fi for he observed curves, we do he following: For each ieraion, we obain a sequence of basis funcions seleced along wih oher parameer values. We hen calculae he frequencies of differen sequences 34

52 y() Truh TB B SKB Figure 2.6: True signal curve versus fied curves from hree compeing mehods. Black solid curve: rue signal curve. Red dashed curve: fied curve obained from Bayesian ransformed B-splines mehod. Blue doed curve: fied curve obained from B-splines basis funcions. Dashed green curve: fied curve obained from B-splines basis funcions wih seleced knos. of basis funcions being seleced. Usually here is one or wo ses of basis funcions ha are seleced wih overwhelming frequency. These are he poenial ses of basis funcions o be included in he model. Nex we selec he ieraions in he chain ha have he same se of basis funcions appearing mos frequenly. Following ha, we obain leas squares esimaes for he coefficiens of he basis for each ieraion in he subse, and hence obain esimaes for he original curves. Finally we average hose fied curves and ake ha as our STB fi. Noe ha even hough he basis funcion shapes change from ieraion o ieraion, afer a cerain poin (i.e., afer he MSE values level off), such change is negligible and only represens randomness 35

53 y() Truh TB Wave Fourier Figure 2.7: True signal curve versus fied curves from hree compeing mehods. Black solid curve: rue signal curve. Red dashed curve: fied curve obained from Bayesian ransformed B-splines mehod. Blue doed curve: fied curve obained from he wavele basis funcions. Dashed green curve: fied curve obained from he Fourier basis funcions. of he Markov chain. Similarly, he SKB fi is obained by firs checking he MSE plo o deermine he number of ieraions i afer which he MSE values become sabilized, obaining he number of knos k sel ha appear wih he greaes frequency in he chain, hen selecing all ieraions wih k sel knos seleced, and averaging he esimaed curves from he se of chosen ieraions afer he ih. Again, even wih he same number of knos seleced, he locaions of he knos vary for differen ieraions in he se. Ye sill, he esimaed curves sabilize afer he ih ieraion, as he MSE values sabilize. We se he maximum possible number of knos, k max, o be 12 in his seing, making he greaes possible dimension of he B-spline model 15. In pracice, for hese irregular simulaed daa, he RJM CM C procedure for kno 36

54 Table 2.1: MSE comparison able for five smoohing mehods. Top: MSE values calculaed wih respec o he observed curve. Boom: MSE values calculaed wih respec o he rue signal curve. TB STB B SKB Wave Fourier Obs Truh selecion always leads o he maximum number of knos appearing wih he greaes frequency, making he dimension of he SKB model exacly 15. SKB does a beer job han B in capuring he peaks and roughs of he curves, ye wih is limied dimension, i canno achieve a desirable fi, eiher. Among he several fied curves in he comparison plos, ours (red) performs he bes. Table 2.1 compares he MSE values (wih respec o he rue signal curve and o he observed curve) obained via he aforemenioned fiing approaches. Example 2.2. Using he same simulaion seing, wih differen weighings of he hree curve ypes, we now compare he performances of differen fiing mehods. Table 2.2 summarizes he MSE values resuling from he comparaive approaches. The hree numbers in he firs cell of each row represen he weighing of respecively: he periodic curve generaed from he Fourier basis, he smooh curve generaed from B-splines and he spiky curve obained from he mehod described previously. The number in each upper sub-cell in he inerior of he able is he MSE of he fied curve obained via he corresponding mehod wih respec o he observed curve, and he value in he boom sub-cell is he MSE wih respec o he rue signal curve. For differen weighings, he error erms added for each curve ype migh be differen, in order o mainain he signal-o-noise raio. The value in bold in each row is he row minimum, and he value in ialics is he second smalles MSE value in he row. Our approach almos always achieves a much smaller MSE compared wih oher fiing mehods. 37

55 Table 2.2: MSE comparison able for six smoohing mehods. Digis in he Weighs column represen he weighing of he simulaed periodic curve, smooh curve and spiky curve, respecively. Top row in each cell: MSE value calculaed wih respec o he observed curve. Boom row in each cell: MSE value calculaed wih respec o he rue signal curve. Weighs B SKB Wave Fourier TB STB Real Daa Applicaion In his secion we include some resuls for fiing an ocean wind daa se aken from he Naional Oceanic and Amospheric Adminisraion (NOAA). This daa se is analyzed in Hichcock, Booh and Casella (2007) for exploring he effecs of daa smoohing on daa clusering. The enire daa se is accessible via he hisorical daa page of Naional Buoy Daa Cener: hp:// (Hichcock e al. 2007). A each of four regional locaions (Norheas, Souheas, Easern Gulf and Wesern Gulf), average hourly wind speeds are colleced. The daa observaions could be viewed as funcional daa since wind speed is inrinsically coninuous and ha each daa poin is calculaed based on a coninuous measure of wind speed during hourly periods. For our analysis, we fi he same subse of 18 38

56 y() y^() Figure 2.8: Observed versus smoohed wind speed curves. Top: four observed wind speed curves. Boom: four corresponding wind speed curves smoohed wih Bayesian ransformed B-splines basis funcions. curves analyzed in Hichcock e al. (2007), where only he measuremens from he firs 7 days of 2005 are used. The firs graph in Figure 2.8 shows four observed sample wind speed curves. The original paerns of he curves are somewha masked by measuremen error. The second graph in Figure 2.8 exhibis four corresponding fied curves obained via our approach. Each pair of observed and smoohed curves are draw wih he same color. Figure 2.9 shows side-by-side boxplos of MSE values for fied curves, corresponding o 18 funcional ocean wind speed observaions, obained via differen mehods (i.e., 39

57 SKB B Wave Fourier TB Figure 2.9: Side by side boxplos of MSE values for five smoohing mehods. From lef o righ: boxplo of MSE values for 18 ocean wind curves smoohed wih he seleced knos B-splines (SKB); he B-splines wih equally seleced knos (B); he Wavele basis (Wave); he Fourier basis (Fourier) and he Bayesian ransformed B-splines (TB). B, SKB, Wave, Fourier, TB). For his daa se, we sar wih order-6 B-splines wih 8 knos o carry ou our analysis. Noe ha one should only choose an odd number of Fourier basis funcions so ha he same number of sine and cosine erms are used when fiing funcional daa; hus we use 13 Fourier basis funcions. Since he wavele basis requires he number of knos o be a power of 2, we use 16 basis funcions. For he SKB mehod, for each of he 18 observaions, he RJM CM C procedure ineviably chose he number of knos o be k max (i.e., 8) mos frequenly, due o he several oscillaions and spikes in he curves, and he small value of k max. Therefore, for all he observaions analyzed, he SKB mehod in fac used 12 basis funcions for daa fiing. Thus he comparison for MSE is relaively unprejudiced in erms of model dimension, excep ha he wavele basis and he Fourier basis have slighly higher dimensions. The 40

58 boxplos show ha our mehod has an obvious advanage, in erms of MSE values, in fiing he wind daa se. 2.7 Discussion We have proposed a Bayesian approach for funcional daa fiing via basis sysems. Our mehod surpasses he shape-dependen limiaions of currenly popular basis sysems, e.g., polynomial splines, B-Splines, Fourier Bases, wavele bases, in funcional daa fiing. I adaps splines flexibly in accordance wih a variey of shapes of curves. Using a daa-dependen ransformaion of individual splines, fi accuracy is improved compared o exising basis sysems, wih model dimension remaining small o moderae. In our approach, ime disorion is no longer applied o he enire observaion o align daa, bu raher on each individual spline funcion. The applicaion of he ime warping funcion o he splines allows he domain of he ransformed splines o be roughly he same as he original splines, making he design marix band-srucured for compuaional efficiency. In fac, as long as he ime grid on which ransformaion is applied is sufficienly fine relaive o he measured ime sequence, he poserior splines remain smooh, and shifs on he domains of he splines, if any, are negligible. In his aricle, our approach is based on B-splines, i.e., we impose ime ransformaions on individual B-splines o obain poserior splines, and we compare he fiing resuls wih he opimal performance of B-splines. Cerainly he mehod is no resriced o B-splines as he saring basis funcions; one may also consider oher possible ransformaion opions. The combinaion of ime ransformaion on individual basis funcions and Bayesian modeling is a novel approach o funcional daa fiing. I achieves similar goals as employing muliple resoluion basis sysems and carrying ou a basis selecion, ye avoids he heavy compuaional workload. 41

59 Chaper 3 Funcional regression and clusering wih funcional daa smoohing Summary: This chaper includes some invesigaion of funcional daa regression and clusering, in order o evaluae he effecs of our fiing mehod inroduced in chaper 2 in funcional daa-relaed applicaions. 3.1 Inroducion Funcional Regression As wih individual random variables whose variabiliy can be analyzed by sudying heir relaionships wih oher variables, funcional daa lend hemselves o similar mehods like regression or ANOVA in a funcional seing. Usually, linear models involve funcional variables in one of he following hree ways: eiher he response variable is funcional and he predicor(s) scalar, or some or all of he predicors are funcions and he response is scalar, or boh he response and one or more of he predicor variables are funcions. The main difference beween regression or ANOVA in he funcional seing and in he radiional sense is ha he inercep erm and all he coefficiens relaing o funcional predicors are also funcions. Typically, how he funcional regression model is buil mus depend on one s insigh abou how he response variable is relaed o he predicors, wheher funcional or scalar. The concurren model is frequenly employed when boh he response variable and some of he predicor variables are funcional, and when one believes ha he 42

60 response variable value measured a any poin in he ime inerval T is only relaed o he value a he equivalen ime wihin he funcional predicor. The concurren model a ime wih only one funcional predicor is as follows: y() = α() + x()β() + ɛ() Here he response variable is measured a a fine grid of ime poins in T. β() quanifies he impac of predicor x a ime on he value of y(). By carrying ou poinwise regression a each ime poin, we ge esimaes of α() and β(). And combining ˆα() and ˆβ() for all he measured ime poins in, we obain esimaed curves for he rue inercep funcion α( ), and for he coefficien funcion β( ). When one believes ha he response variable is relaed o he derivaive of one or more predicors, or here are some ineracions beween he predicors, or beween he derivaive of some predicor and some oher predicors, he ineracion model or derivaive model is implemened. There are many oher models in he realm of funcional regression, bu we will be mosly focusing on he models discussed above as our simulaions and real daa examples sui he aforemenioned model well. Funcional Clusering Anoher mosly used applicaion in he realm of funcional analysis is funcional clusering. Funcional clusering mehods are very similar o radiional cluser analysis mehods. One sars by calculaing disances beween all pairs of curves o be clusered o form a disance marix. One mos commonly used disance measure is he Euclidean disance, or he L 2 norm: d L2 [y i (), y j ()] = [y i () y j ()] 2 d = d ij Popular funcional clusering mehods can be direcly applied o he disance marix obained from he original se of curves. 43

61 Currenly, popular clusering mehods include bu are no limied o hierarchical based mehods wih differen ypes of linkages, pariioning mehods such as he K-means and he K-medoids mehods, and a hybrid mehod, Ward s mehod. The hierarchical mehods ypically sar wih each observaion being a individual cluser, and joining he wo closes clusers in each sep. Here, he closeness beween wo clusers is measured differenly depending on he linkage employed. The single linkage mehod joins he wo clusers C 1 and C 2 wih he smalles disance d s as measured by: d s = min (d ij ). i C 1,j C 2 Since i only depends on he minimal disance beween any pair of curves, he single linkage based hierarchical clusering resuls can be unreliable. As opposed o single linkage, complee linkage is based on he maximal disance d c for all pair of curves, which is given by: d c = max (d ij ). i C 1,j C 2 The complee linkage is beer han single linkage, since i depends on he farhes disances for all pairs of curves. Anoher linkage opion is called he average linkage, which joins he wo clusers based on he minimal of he following disance d a : d a = 1 n 1 n 2 i C 1 j C 2 (d ij ). Here n 1 and n 2 are he oal number of iems in clusers C 1 and C 2, respecively. This linkage uilizes he mos informaion possible from all he observaions o deermine disance beween clusers. Anoher caegory of clusering mehods is he pariioning mehods, and he mos popular clusering mehods in his caegory include K-means and K-medoids clusering. Unlike hierarchical mehods, which form ree ype resuls as par of he clusering procedure, he pariioning mehods ypically ry o search for an opimal pariion of he original observaions ino K clusers by minimizaion of some summed disance 44

62 measure. Such mehods sar from randomly pariioned K clusers, or K ceners, called cenroids, and repeaedly swich he clusering membership for each iem in he pool unil no new pariion could resul in a smaller summed disance measure defined for he corresponding approach. Ward s clusering mehod, on he oher hand, falls in a hird caegory, which is in fac a hybrid of he hierarchical mehod and he pariioning mehod, in he sense ha i sars from each observaion being is own cluser, and joins he wo clusers ha will lead o he smalles summed disance measure a each sep, unil all observaions are joined ino one big cluser. Tree cuing echniques can be used o obain clusers of desirable size. The fiing mehod we proposed in Chaper 2 provides an alernaive o commonly used smoohing mehods such as he B-splines, he Fourier basis, he wavele basis, ec. We are specifically ineresed in he applicaions of our fiing mehod, o see wheher i leads o improvemens in saisical analyses and how such improvemens compee wih hose resuling from employing oher fiing mehods before carrying ou funcional daa applicaions. Our invesigaion focused on funcional regression and funcional clusering. In secions 3.2 and 3.3, we discuss simulaed daa examples for funcional clusering and funcional regression, respecively. And in secions 3.4 and 3.5, we give wo real daa funcional regression examples. 3.2 Simulaion Sudies: Funcional Clusering This secion includes some invesigaion of he influence of our funcional daa smoohing mehod on funcional daa clusering, o see wheher our mehod gives compeiive resuls in erms of accuraely clusering he curves. We have simulaed a oal of four clusers of 23 curves, each of lengh 200, where hree of he clusers each have 6 curves, and one cluser has 5. Figure 3.1 shows he 23 raw daa curves, where curves coming from he same cluser are ploed wih he same color and line 45

63 y() Figure 3.1: Simulaed curves from four clusers. symbol. We fi all 23 curves using our ransformed B-spline approach, he regular B- spline approach, B-splines wih seleced knos, he wavele basis funcions and he Fourier basis funcions. Then we calculae, for each pair of curves, he Euclidean disances beween hem, o form a disance marix. And linkage based hierarchical clusering mehod wih single linkage, average linkage or complee linkage, Ward s mehod, k-medoids and he k-means clusering mehods are implemened based on he disance marix, and lasly he Rand index for using each clusering mehod wih each smoohing mehod is calculaed. Here, he Rand index is originally inroduced in Rand (1971), i measures he similariy beween wo cluserings C p1 and C p2 of a se of objecs S c. I s value is calculaed via he following formula: R c = n 1 + n 2 n 1 + n 2 + n 3 + n 4, 46

64 Table 3.1: Rand index values for five smoohing mehods based on regular disance marix TB B SKB Fourier Wave single average complee ward k-medoids k-means Table 3.2: Rand index values for five smoohing mehods based on sandardized disance marix TB B SKB Fourier Wave single average complee ward k-medoids k-means where n 1, n 2, n 3 and n 4 are he numbers of pairs of objecs from he sample S c ha are: assigned o he same cluser in C p1 and he same cluser in C p2, assigned o differen clusers in C p1 and differen clusers in C p2, assigned o he same cluser in C p1 bu differen clusers in C p2 and assigned o differen clusers in C p1 bu he same cluser in C p2, respecively. Hence, Rand index values close o 1 implies grea similariy of he wo cluserings C p1 and C p2 of he curves, oherwise, Rand index values close o 0 suggess grea discrepancies beween C p1 and C p2. Table 3.1 gives he Rand index values based on he regular disance marix, wih each of he aforemenioned smoohing mehods applied o he daa prior o daa clusering. And Table 3.2 gives he Rand index values based on he sandardized disance marix, in which each measured poin in each curve has he mean value for all 23 curves measured a he same ime poin subraced from i, and is divided by is sandard deviaion across he 23 curves. For boh ables, excep for he hierarchi- 47

65 cal clusering mehod wih single linkage (in which case he wavele basis smoohing seems o give he bes clusering accuracy) for all oher clusering mehods implemened, our ransformed B-spline basis smoohing produces clusers whose accuracy is no worse han all oher compeing mehods. Hence, from he funcional clusering perspecive, our smoohing mehod does produce compeiive resuls as measured by Rand index. 3.3 Simulaion Sudies: Funcional Regression We are specifically ineresed in exploring wheher smoohing funcional daa prior o funcional regression will lead o improvemen in parameer esimaion and predicion accuracy, and if so, which of he currenly popular mehods would lead o comparaively greaer improvemen. This may depend on wha crierion we use o judge wha beer means. In his sudy, he smoohing mehods under comparison are he same as hose used in Chaper 2, i.e., a B-splines fiing mehod wih equally spaced knos, a B- spline basis wih opimally chosen knos, he Fourier basis, he wavele basis, and our Bayesian approach. Inuiively, we may conjecure ha no applying pre-smoohing echniques on funcional daa prior o carrying ou regression analysis, i.e., he naive approach, would lead o he wors regression predicion and esimaion, since error erms in he daa may mask he rue signal, and good smoohing mehods end o separae error from daa. However, based on our simulaion invesigaions, he resuls do no necessarily mach hose inuiive expecaions. In he following simulaion sudies, we focus on he one predicor scenario, in which boh he response and he predicor variables are funcions of he same ime inerval T. Our rue predicor variable signal is simulaed o be a linear combinaion of periodic pars, a smooh par and a spiky par. The periodic par consiss of several sine and cosine funcions, he smooh par is simply a polynomial funcion, and he 48

66 spiky par is generaed via a random Poisson realizaion o deermine he number of spikes in each curve, exponenial realizaions o deermine he heighs of hose spikes in each curve, and uniform realizaions o deermine he widh of each spike. The inercep α( ) and coefficien funcion β( ) are specified as smooh funcions. We generaed 16 response and predicor curves using he same α( ) and β( ). Since for regression analysis, he spread of he values of he predicor influences he accuracy of esimaion, we adjused he ranges for he predicor curves o be somewha differen, by adding consan shifs o he original generaed predicor curves if necessary. Assuming ha boh he response and he predicor are observed over a relaively fine grid of ime poins in T, realizaions of he Ornsein-Uhlenbeck process are applied o generae he discreized error vecors, which are added o he signal response and predicor vecors o obain he observed vecors. The Ornsein-Uhlenbeck (O-U) process is a sochasic process ha used o model a coninuous random noise process. The O-U process is realized via he following differenial equaions: dx = θx d + dw, wih X 0 = x 0, and W is he Wiener process. For he O-U process, boh he ime and he disances δ beween X and X +δ play a role in deermining he srucure beween hem. Afer he daa are generaed, he smoohing mehods menioned above are applied on boh he response and predicor discreized funcions. For each measured ime poin, we carry ou a poinwise funcional regression analysis, where we concaenae he response values (eiher smoohed or nonsmoohed) of all 16 response curves evaluaed a o form he response vecor, and do he same for he predicor curves (eiher smoohed or non-smoohed) o obain a vecor of predicor values evaluaed a. Then we use OLS o obain he fied values of α() and β(). Afer carrying ou he poinwise regression for all, by concaenaing he esimaed α() and β() for all values, we obain he enire esimaed α and 49

67 Table 3.3: In-sample SSE comparison for funcional regression predicions based on simulaed curves NS TB Fourier Wave B Truh Obs β funcions. Model fi is assessed by: SSE = i (yi () ˆα() x i () ˆβ()) 2, where yi () is he ih rue signal response, x i () is he smoohed predicor evaluaed a poin, ˆβ() is he esimaed coefficien curve and ˆα() is he esimaed inercep curve, where some smoohing mehod is applied o ˆα() o correc for apparen roughness. In our comparison, we fix he number of basis funcions across differen smoohing mehods under comparison. I urns ou ha naive regression (i.e., wih no presmoohing on he curves) does well in esimaing β, and he SSE value is even smaller han mos compeing mehods ha apply pre-smoohing on he daa (as will be shown in Table 3.3). The reason could be ha he number of basis funcions employed for differen smoohing mehods is relaively small and is insufficien o capure he srucure of he simulaed daa; herefore, hey end o underfi he curves. On he oher hand, a closer look a he naive regression reveals ha i ends o overfi he daa, and hus may do poorly if one exrapolaes he measured ime poins ino an ou-of-sample se of poins in T. The resuls comparing he SSE values defined above for he prediced response curves wih respec o eiher he rue signal curves or he observed response curves, wihou smoohing versus wih presmoohing on boh he response and he predicor variables via differen smoohing mehods are shown in Table 3.3. Here, NS denoes no smoohing on eiher he response or he funcional predicor. Oher noaions are 50

68 he same as in Chaper 2. We can see ha he SSE values wih respec o he observed curve and he rue curve are boh smalles if he response and he predicor curves are fied using our mehod. Noe ha he SSE values are on a very large scale. This is because he values are calculaed as he sum of SSE s for 16 curves accumulaed over a oal of 100 ime poins. The SSE values for he individual fied curves are shown in Figure 3.2. This graph includes 9 boxplos of SSE values for 9 prediced response curves wih respec o he rue signal curves, wih presmoohing of he curves based on he five compeing mehods shown in Table 3.3. The red reference line on each boxplo gives he SSE value of he prediced response curves wih our presmoohing approach. Apparenly, for he majoriy of he cases, using our smoohing mehod prior o carrying ou he funcional regression model produces he prediced response curves wih relaively small SSE values, compared o oher compeing mehods such as NS (no smoohing), Fourier, Wave and B. Figures 3.3 and 3.4 give 9 rue signal response curves versus prediced response curves in scenarios when he response and predicor curves are used direcly in funcional regression, or when hey are presmoohed using TB, Fourier, Wave or B. On each individual plo of Figure 3.3, he black spiky curve represens he rue signal response curve, he purple wiggly curve represens he prediced response wih no presmoohing of he curves, he red and green dashed curves represen he scenario when he response and he prediced curves are presmoohed using he ransformed B-splines mehod or he Fourier basis funcions, respecively. On each individual plo of Figure 3.4, he black spiky curve and red dashed curve are he same from hose in Figure 3.3, and he green and blue dashed curves represen he prediced response wih presmoohing using he B-spline or he Wavele basis funcions, respecively. In all of hese individual plos, i is shown ha our approach in Chaper 2 ends o give he bes predicions in he sense ha hose prediced curves no only are smooh in 51

69 SSE 0e+00 3e+05 SSE SSE SSE SSE SSE SSE SSE SSE Figure 3.2: Boxplos of SSE values for he firs 9 prediced response curves wih no presmoohing or presmoohing using he ransformed B-splines, he Fourier basis, he Wavele basis and regular B-splines basis funcions on he curves. Red line: SSE value for he prediced response curves wih presmoohing on he curves using he ransformed B-splines. naure, bu hey are capable of capuring he spikes on he curves ha oher compeing mehods end o miss. The mehod employing he unsmoohed curves ends o overfi, alhough i has smaller SSE han he Fourier smooher, which ends o miss he peaks. We also check ou-of-sample predicion SSE for he five differen mehods. We 52

70 y() y() y() y() y() y() y() y() y() Figure 3.3: The firs 9 prediced response curves. Black spiky curves: rue signal response curves. Red long dashed curves: prediced curves wih presmoohing using he ransformed B-splines. Purple wiggly curves: prediced response curves wih no presmoohing on he curves. Green dashed curve: prediced response curves wih presmoohing using he Fourier basis funcions. 53

71 y() y() y() y() y() y() y() y() y() Figure 3.4: The firs 9 prediced response curves. Black spiky curves: rue signal response curves. Red long dashed curves: prediced curves wih presmoohing using he ransformed B-splines. Blue wiggly curves: prediced response curves wih wih presmoohing using he Wavele basis funcions. Green dashed curve: prediced response curves wih presmoohing using he regular B-spline basis funcions. 54

72 Table 3.4: Ou-of-sample SSE comparison for funcional regression predicions based on simulaed curves NS TB Fourier Wave B Truh have generaed a random sample of 100 ime poins across [0, 1], and have our rue and prediced response curves exrapolaed o hose ime poins, and calculaed he SSE values as defined earlier for he aforemenioned se of compeing mehods. See Table 3.4. Again our mehod gives he smalles SSE as compared wih ohers. Noe ha similarly o Chaper 2, we are uilizing roughly he same number of basis funcions for he differen smoohing procedures prior o funcional regression. In fac, one may ge nearly perfec fi of funcional daa via mos fiing mehods such as he wavele basis, as long as one allows a sufficienly large number of basis funcions o be employed. However, in pracice, one does no wan a model ha uilizes a number of basis funcions ha is oo large, due o compuaional speed and model parsimony concerns. Tha is he reason ha we are resric all compeing mehods o employ he same number of basis funcions for comparison purposes. For some oher currenly popular B-splines relaed mehods, such as he radiional B-spline basis or he seleced knos B-spline basis mehods, smooh and accurae fis could only be obained as long as one chooses he order of he basis funcions, he number and locaions of he knos wisely. However, kno selecion remains a difficul problem even oday. In mos cases he opimal knos seleced in pracice are no ruly opimal. Hence one common problem of hese fiing mehods is ha when given funcional curves ha are complicaed in srucure, especially hose ha are locally highly irregular in some regions while smooh in oher areas, i is hard o deermine he appropriae number of basis funcions o use o obain a desirable fi. One may choose a relaively large number of basis funcions, hoping o achieve an 55

73 accurae fi, ye a an accurae fi in local irregular areas may lead o over-fiing in oher smooh areas. On he oher hand, if one chooses a small o moderae number of basis funcions o begin wih, he rue curve may be poorly fied. However, our approach does no encouner problems like hese. A balance beween accuracy and smoohness can always be achieved even wihou ools like he roughness penaly erm. Relaive accuracy is achieved due o he flexibiliy aained brough by adding ime disorion o each of he basis funcions, even if one sars wih few basis funcions, and smoohness is obained by aking he average of he fied curves from hese sabilized MCMC ieraions once convergence is aained, since he randomness in he ime-warping from ieraion o ieraion is smoohed ou by his simple sep. This is one vial difference beween radiional fiing mehods and our approach, i.e., he basis funcions conain suble changes from ieraion o ieraion, and convergence of heir individual shapes is achieved when he number of ieraions is large enough. In his simulaion sudies, we have smoohed he esimaed α() funcions obained via differen smoohing mehods using a moderae number of B-spline basis funcions, since hey were all oscillaing and lacked smoohness. The esimaed α() funcions seem o play he role of an offse since he ˆβ() funcions are all relaively smooh. This sep is no necessarily vial, due o he fac ha he ˆα() s have relaively small range compared wih ha of he simulaed curves hemselves. In sum, his simulaion sudy gives us evidence of improvemen in predicion accuracy in funcional regression, when our approach is uilized o presmooh he curves, and such improvemen is compeiive wih many oher currenly popular smoohing mehods. However, one may be aware ha more accurae fiing of he curves are no always linked wih beer predicion accuracies. In fac, in some examples shown laer, a imes consisenly underfiing or overfiing he curves may lead o good predicion accuracy as well. Hence, if he goal is solely o sudy he paerns of he rue underlying funcional form of he curves, hen our approach is 56

74 a wise choice. Bu when he objecive is performing funcional applicaions such as funcional regression or clusering, hen our mehod serves as an alernaive fiing approach in he realm of funcional daa fiing mehods, which are expeced o improve he saisical inference resuls o some exen. Bu depending on he specific siuaion, here sill migh be circumsances when, surprisingly, no smoohing or some oher smoohing mehods (even hough hey may no give smooh and accurae fis of he curves) may lead o he bes performance; hence one mus no make sweeping conclusions. 3.4 Real Daa Applicaion: Funcional Regression Example 1 In he following wo secions, we explore he performance and poenial of our funcional daa smoohing mehod on real funcional regression applicaions. Our daa comes from he websie hp:// where he hisorical weaher daa on any given ciy can be found. We ake one year of weaher daa for Orlando, FL. In our daa se, here are daily emperaure, dew poin, humidiy, sea level pressure, visibiliy, ec. We carry ou a funcional regression using his daa se. We employ daily emperaure as he response variable, and since emperaure is inuiively mosly relaed o dew poin and(or) humidiy, we include boh dew poin and humidiy in he model, and we explore he effec of our and oher fiing mehods on he predicions of muliple funcional regression. We denoe he emperaure funcion as T ( ), dew poin as D( ), and humidiy as H( ). Our one year daa se is spli ino four 3-monh secions, and each of he 3-monh daily emperaure funcion serves as a response curve. Hence we have four pairs of 3-monh dew poin and humidiy funcions as predicor curves, accordingly. To es wheher our smoohing mehod leads o improvemen on his real daa se, we smooh all of he raw response and predicor curves, and for each mehod, we fi hree compeing models: T () = α 1 () + D()β 11 () + H()β 12 () + ɛ(), 57

75 Table 3.5: SSE comparison able: Model 1: boh dew poin and humidiy are predicors. Top: SSE for prediced response curves wih respec o he rue observed curve, no furher smoohing on α 1 (), β 11 () and β 12 (). Boom: SSE for prediced response curves wih respec o he rue signal curve, wih furher smoohing on α 1 (), β 11 () and β 12 () using regular B-spline basis funcions. NS TB Fourier Wave B Smoohed Unsmoohed Table 3.6: In-sample SSE comparison: Model 2: dew poin is he only predicor. Top: SSE for prediced response curves wih respec o he rue signal curve, no furher smoohing on α 2 () and β 1 (). Boom: SSE for prediced response curves wih respec o he rue observed curve, wih furher smoohing on α 2 () and β 1 () using regular B-splines basis funcions. NS TB Fourier Wave B Smoohed Unsmoohed T () = α 2 () + D()β 1 () + ɛ(), T () = α 3 () + H()β 2 () + ɛ(). Each of hese hree models is fied afer five compeing smoohing mehods are performed on he raw response and predicor curves, i.e., NS, TB, B, Wave and Fourier as defined earlier. Once he coefficien curves are obained, hey are eiher furher smoohed using regular B-spline basis funcions wih a sequence of equally spaced knos (order 10 basis funcions wih 15 knos locaed evenly across (0, 1)), or lef unsmoohed, and he esimaed coefficien funcions are used o produce predicions for all 4 response curves. Lasly, SSE values for all he prediced curves are calculaed for each compeing smoohing mehod, wih or wihou furher smoohing on he esimaed coefficien curves. The SSE values are given in Tables 3.5, 3.6 and 3.7. For all hree models, our mehod ends o give he smalles SSE, or beer predicions for he response curves. 58

76 y() y() y() y() Figure 3.5: True Orlando emperaure curves and prediced response curves wihou daa smoohing, or wih daa presmoohed using he ransformed B-spline basis and he Fourier basis funcions. Black solid curve: rue Orlando weaher curves. Green solid curve: prediced curves wihou any daa smoohing. Red solid curve: prediced curves wih daa presmoohed using he ransformed B-spline basis funcions. Purple dashed curve: prediced curves wih daa presmoohed using he Fourier basis funcions. 59

77 y() y() y() y() Figure 3.6: True Orlando emperaure curves and prediced response curves wih daa presmoohed using he ransformed B-spline basis, regular B-spline basis funcions and he wavele basis funcions. Black solid curve: rue Orlando weaher curves. Green solid curve: prediced curves presmoohed using he wavele basis funcions. Red solid curve: prediced curves wih daa presmoohed using he ransformed B-spline basis funcions. Purple dashed curve: prediced curves wih daa presmoohed using regular B-spline basis funcions. 60

78 Table 3.7: In-sample SSE comparison: Model 3: humidiy is he only predicor. Top: SSE for prediced response curves wih respec o he rue signal curve, no furher smoohing on α 3 () and β 2 (). Boom: SSE for prediced response curves wih respec o he rue observed curve, wih furher smoohing on α 3 () and β 2 () using regular B-splines basis funcions. NS TB Fourier Wave B Smoohed Unsmoohed Inuiively, furher smoohing on he esimaed inercep and slope curves could ge rid of heir unwaned local behaviors, making he prediced curves look beer and smooher. However, his is no always he case, as can be seen from hese ables: For eiher model and any compeing smoohing mehod, doing no furher smoohing on he esimaed coefficien curves ends o give a smaller SSE. Besides, simply from he SSE poin of view, cerainly he model wih boh predicors gives he smalles SSE values, bu using he principle of model parsimony, one could argue for he hird model, wih only humidiy as he predicor. Figures 3.5 and 3.6 give he prediced response curves versus rue Orlando weaher emperaure curves for he model wih only humidiy as a predicor, and wihou smoohing on he esimaed coefficien curves. The black curves on he individual plos of boh figures are he rue Orlando emperaure curves, and he red solid curves on boh figures are he prediced curves wih daa presmoohed using our approach in Chaper 2. The green curves on Figure 3.5 are he predicions wihou any daa smoohing before he funcional regression sep, and purple dashed curves are prediced curves wih daa presmoohed using he Fourier basis funcions. On Figure 3.6, he green solid curves are he prediced responses wih daa presmoohed using he wavele basis funcions, and he purple dashed curves are hose obained wih daa presmoohed using regular B-spline basis funcions. In all four curves, our approach produces smooh fis ha mosly capure he main paerns of he curves, while oher predicions wihou presmoohing or 61

79 Ocober 2015 Congaree waer level Figure 3.7: Upsream (Congaree gage) waer level measures for flood even of Ocober wih oher smoohing mehods on he daa are more wiggly, or a lile bi overfied or underfied. In his Orlando weaher example, we have showed ha presmoohing on he daa using our approach in Chaper 2 gives promising predicion resuls. However, furher smoohing on he esimaed coefficien curves may no improve he predicion accuracy as desired. We discuss his more in anoher real daa example ha follows. 3.5 Real Daa Applicaion: Funcional Regression Example 2 In his secion, he daa se we examine consiss of records of he waer levels during flood evens in Columbia, SC, occurring across several years. More specifically, wo funcional records of river sage levels measured by waer gages are colleced a boh he upsream and he downsream areas of he Congaree River during each of eigh 62

80 Ocober 2015 Cedar Creek waer level Figure 3.8: Downsream (Cedar Creek gage) waer level measures for flood even of Ocober flood evens beween Augus 1995 and Ocober Noe ha he sage is a measuremen of he heigh of he river s waer level a he locaion of he gage. We call he upsream gage he Congaree gage ( Cong ), and downsream gage he Cedar Creek gage ( Cedar ). For each flood even, he Congaree sage and he Cedar Creek sage are measured a he same sequence of ime poins excep for he las flood even, which occurred in Ocober As can be seen from Figure 3.7, he Congaree curve corresponding o he Ocober 2015 flood even has 648 sage readings from he sar of he measuremens. The waer level smoohly increases during he flood and gradually drops o normal wih a few wiggles afer he even. The Cedar Creek curve for his even is given in Figure 3.8; he response funcion (i.e., he waer level) increases gradually a he sar of he flood even, ye akes a few rapid and almos sraigh dips afer he peak. And he gage readings were all missing aferwards. 63

81 Figure 3.9: Downsream (Cedar Creek gage) and upsream (Congaree gage) waer level measures for six flood evens. We would like o esimae he whole curve of sage readings a he downsream gage. I is suspeced ha he gage los readings due o being suddenly broken in he flood, and he peak of he funcion seems o be he breaking poin. Therefore we keep only ha porion of he curve up o he observed peak, and he res of he measuremens will be considered missing. If we sandardize each curve o make each flood even of sandard lengh, so ha each curve has he same number of ime poins, we obain Figure 3.9 as follows. Six pairs of Congaree and Cedar Creek waer level curves for six flood evens are drawn in figure 3.9. And for each flood even, he Congaree curve and he Cedar Creek curve are shown in he same color, and drawn wih he same symbol ype, wih he upper and lower ones represening he upsream and downsream waer levels, respecively. The number of ime poins in he ime sequences for all flood evens are sandardized 64