BAYESIAN WITH FULL CONDITIONAL POSTERIOR DISTRIBUTION APPROACH FOR SOLUTION OF COMPLEX MODELS. Pudji Ismartini

Transcription

1 Proceedig of Iteratioal Coferece O Research, Implemetatio Ad Educatio Of Mathematics Ad Scieces 014, Yogyakarta State Uiversity, 18-0 May 014 BAYESIAN WIH FULL CONDIIONAL POSERIOR DISRIBUION APPROACH FOR SOLUION OF COMPLEX MODELS Pudi Ismartii BPS-Statistics Idoesia, Sekolah iggi Ilmu Statistik (SIS) Jakarta, Idoesia ismartii@bps.go.id, ismartii09@yahoo.com Abstract Complex models are ofte arise i social field applicatios. Model complexity might be caused by the umber of variables i the model, or due to the complex data structures. Problems are ofte ecoutered i complex models is a difficulty to obtai a solutio i the model parameter estimatio process. Bayesia methods ca overcome these problems with the moder approach of Bayesia aalysis usig special simulatio procedure based o the posterior distributio of parameters, i.e. the Markov Chai Mote Carlo (MCMC). Implemetatio of MCMC methods for Bayesia aalysis requires proper samplig algorithm i order to obtai a sample from a distributio. he algorithm which is efficiet ad ofte used by MCMC is Gibbs Samplig. Oe of the advatages of Gibbs samplig is the geeratio of radom variables is doe usig the cocept of oe-dimesioal distributio which are structured as a form of full coditioals, i.e. the full coditioal posterior distributio of parameter. his paper propose to describe the process of parameter estimatio for complex model usig Bayesia through full coditioal posterior distributio of parameters. Keywords: Bayesia, Complex models, MCMC, Gibbs Samplig, Full Coditioal Posterior Distributio M INRODUCION I a real world pheomea, it is ofte faced complex models due to some reasos. First, it is caused by big umber predictors used i the model. It is arise i may areas of sciece ad techology, especially i social sciece sice social pheomea are geerally iflueced by huge dimesios of life. hus modellig i social sciece might deal with huge predictors. Secod, complex model might occur because of the ucommo particular distributio of data. I geeral, data modelig usig the ormal distributio assumptio. However, the data are ofte ecoutered cases that are ot ormally distributed. For example, data of per capita household expediture which are log ormal distributed with three parameters. hird, complex models may be caused by the complex structure of the data. I the social field, i geeral, the data has a hierarchical structure where data is ested withi its territory. Estimatio methods commoly used i statistical modelig is the classical approach usig the data likelihood fuctio. I the classical approach, the model parameters to be estimated is assumed to be sigle-valued. Estimatio process with the classical approach, commoly doe to obtai parameter values that maximizes the Likelihood fuctio which is cosidered as a fuctio of these parameters. I some cases, the estimatio process with the classical methods geerally use umerical optimizatio techiques to obtai the solutio. However, i the case of complex models is ofte difficult to obtai the solutio of classical methods. Bayesia iferece approach is differet from the classical approach eve though both methods M-91

2 Pudi Ismartii /Bayesia with Full... ISBN use the data Likelihood fuctio. he Bayesia approach, all ukow parameters are viewed as radom variables characterized by prior distributios of the parameters (Ntzoufras, 009; Gelma et al., 004; Cogdo, 006). Prior distributio of the parameters stated variatios of these parameters. I cotrast to classical approaches, Bayesia methods do ot ivolve the optimizatio process i iferece. Bayesia approach is applyig Bayes' theorem based o the oit posterior distributio of all parameters (Kig, Morga, Gimeez ad Brooks, 010). Furthermore, the Bayesia method will perform parameter estimatio usig margial posterior distributios of the parameters. he margial posterior distributios obtaied by itegrate oit posterior distributio. At this stage, especially for the case of fairly complex models geerally arise the problems of the itegratio process becomes very complicated ad difficult to obtai the solutio. However, Bayesia methods ca overcome these problems. I this case, the method used i the moder approach of Bayesia aalysis is ot usig the itegratio of oit posterior distributio aalytically, but by usig the data simulatio procedure that follows a oit posterior distributio through the full coditioal form i order to obtai the margial posterior distributio of each parameter which will be estimated. hus, the optimizatio process is performed i a classical aalysis is replaced with the process of itegratio i the Bayesia aalysis approach (Kig et al., 010). Itegratio i Bayesia aalysis is ot doe aalytically to the oit posterior distributio of the parameters, but with a special approach to the simulatio procedure that produces a set of sample from the posterior distributio. his process became kow as the Markov Chai Mote Carlo (MCMC) (Koop 003; Kig et al., 010). his paper propose to describe the process of parameter estimatio for complex model usig Bayesia through full coditioal posterior distributio of parameters. MEHODS he Bayesia method is adopted from the ame of the ivetor of the method, amely homas Bayes ( ). However, the ew Bayesia method became kow i 1764 after homas Bayes died. Although Bayesia methods have existed sice the 18th cetury, but util the early 0th cetury, the Bayesia method is less popular tha classical methods (frequetist). he use of Bayesia methods begis to icrease as the developmet of iformatio techology so the difficulty i data aalysis i order to get the solutio aalytically, ca be obtaied usig computer simulatios (Gelma et al., 004)..1 Bayesia Method Bayesia method is based o the Bayes heorem (Box ad iao, 199; Gelma ad Hill, 007). Accordig to the rules of probability i the Bayes theorem, the posterior distributio of the parameters ca be expressed as followed: f( y θ) f( θ) f ( θ y), f ( y) where, θ ad y are both radom, 1 d y1 y y M-9 (1) θ is parameter vector da y deotes vector of observatios from the sample. f ( y ) is defied as ormalized costat with respect to θ. f ( θ ) is prior distributio of parameter θ ad f ( y θ ) is a likelihood fuctio of data. he, the posterior i Equatio (1) ca be represeted as a proportioal form as follows: f ( θ y) f ( y θ) f ( θ ). () It is show i Equatio (), the posterior is proportioal to the combiatio of prior iformatio

3 Proceedig of Iteratioal Coferece O Research, Implemetatio Ad Educatio Of Mathematics Ad Scieces 014, Yogyakarta State Uiversity, 18-0 May 014 ad curret iformatio of data. All iformatio about the ukow parameter of iterest is icluded i their oit posterior distributio.. Markov Chai Mote Carlo (MCMC) MCMC is a method that is widely implemeted i the Bayesia modelig approach to obtai solutios for existig posterior distributio which requires complicated calculatio sice it ivolves itegratio process with high dimesios (Carli ad Chib, 1995). he idea of MCMC is to geerate a set of parameter data based o Markov Chai process by usig Mote Carlo simulatio iteratively to obtai the posterior distributio which is statioary (steady state) (1) () ( ) (Ntzoufras, 009). Markov Chai adalah suatu proses stokastik dari θ, θ,..., θ sedemikia sehigga f ( 1) ( ) (1) ( 1) ( ) ( t t,..., ) ( t θ θ θ f θ θ t ). he MCMC algoritm is as follow (Ntzoufras, 009): 1. (0) Defie iitialize value of parameters θ. Geerate sample of parameters through iteratively suppose M times of iteratio 3. bur i process for the first B iteratio i order to reach equilibrium state ( B 1) ( B) ( M ) θ, θ,..., θ as a sample i posterior aalysis 5. Use.3 Gibbs Samplig Implemetatio of MCMC methods for Bayesia aalysis requires proper samplig algorithm to obtai a sample from a distributio. he algorithm is ofte used as a geerator of radom variables i the MCMC is the Gibbs samplig (Casella ad George, 199; Gelma et al., 004). Gibbs Samplig is a techique for geeratig radom variables from a margial distributio directly without havig to calculate the desity fuctio of the distributio (Casella ad George, 199). Gibbs samplig process is doe by takig a sample of Gibbs Sequece based o the basic properties of the Markov Chai (Casella ad George, 199). Gibbs Samplig algorithm is illustrated i Figure 1. Figure 1. Gibbs Samplig Algorithm M-93

4 Pudi Ismartii /Bayesia with Full... ISBN DISCUSSIONS Gibbs Samplig usig the cocept of phasig set of ukow parameters ad estimate the parameter or set of parameters oe at a time with a give value of the other parameters ad the data sample used (Lu, homas, Best, ad Spiegelhalter, 000). his is oe of the advatages of Gibbs Samplig because it geerates the radom variables by usig the cocept of uidimesioal distributios which are structured as a form of full coditioals. Assume θ is a vector of a set of parameters to be estimated, the full coditioal posterior for each parameter ca be formulated. he full coditioal posterior of parameter is specified from the oit posterior distributio of all parameters i the vector θ by settig costat value for other parameter tha, i.e. θ \ d, where θ is vector of \ parameter θ without ad d is umber of parameter to be estimated. hus, the full coditioal posterior distributio of, i.e. f ( θ\, y), is proportioal form of oit posterior distributio of all parameters which oly cotais the compoet parameters, while the other compoets are elimiated. he full coditioal posterior distributio for each parameter was the used i the iteratio process of parameter estimatio usig MCMC with Gibbs samplig. Suppose Y is radom variable which follow Log Normal hree Parameters distributio (LN3) or Y LN3(,, ). If 1/ is precisio parameter, the probability desity [ y] [ y] [ y] [ y] fuctio of LN3 is as follow (Norstad, 011): Sice E y 1 [ y] [ y] f ( y [ y], [ y], )= exp (l( y ) [ y] ). ( y ) l x β, the likelihood fuctio of LN3 radom variable is : L i i [ y] i [ y] i1 = f ( y β,, ) f( y β,, ) (3) i1 1 [ y] [ y] exp (l( yi ) xiβ) ( yi ) he prior of parameter, [ y], ad are defied as follow: (, ), (4) r N [ ] r [ ] r Gamma(a,b ), (5) [ y] [ [ y] ] [ [ y] ] N(, ). (6) [ ] [ ] he set of parameter to be estimated is the θ ( β, [ y], ) where β 0 1 k. Accordig to Equatio (), oit posterior distributio of parameters is formulated by usig Equatio (3), (4), (5) ad (6) : M-94

5 Proceedig of Iteratioal Coferece O Research, Implemetatio Ad Educatio Of Mathematics Ad Scieces 014, Yogyakarta State Uiversity, 18-0 May 014 f ( β,, y) f ( y β,, ) p( ) p( ) p( ) i1 [ y] i [ y] r [ y] i1 r0 k ( ) / k [ y] 1 / 1 exp [ y] (l( yi ) xi β) [ ] r exp [ ] r ( r [ ] r ) i1 r0 yi (7) exp exp ( ). [ y ] a 1 [ y] 1/ [ ] [ y] [ ] [ ] b[ ] [ τ[ y ]] he full coditioal posterior distributio of each parameter is a proportioal form of oit posterior distributio i Equatio (7) which is oly cotai a parameter that will be estimated. hus, other compoet of parameters are elimiated sice it is treated as a costat value. he full coditioal posterior distributio of each parameter is the defied as follow : k (i) 1 1 f( y, β r \ r,, [ y] ) exp [ y] (l( yi ) i ) exp [ ] r ( r [ ] r ), i1 x β (8) r0 (ii) (iii) 1 1 f( y, β, ) exp (l( ) x β ) exp ( ), (9) [ ] [ y] [ y] yi i [ ] i1 ( yi ) i1 / 1 [ [ ] ] 1 y [ y] f( [ y],, ) [ y] exp [ y] a τ y β (l( yi ) xi β ) [ y] exp i1 b [ [ y ]] (10) he process of parameter estimatio is coducted usig MCMC ad Gibbs Samplig. his process is iterative process that follows the Markov Chai process iteratively to estimate the parameters. his iterative estimatio process use a give value of the parameter iformatio from the previous iteratio step. his process will take place i full coditioals form of each parameter ad these are arraged alterately as the ext phase of iterative stochastic simulatios (Gelma, et al, 004). Gibbs samplig iteratio procedure is doe with the followig steps (Ismartii, Iriawa, Setiawa da Ulama, 01): (0) (0) (0) Step 1 : Defie iitial value for each parameter which will be estimated ( β, λ, τ ). [ y] Step : Geerate sample of each parameter by coducted M iteratio process through the full coditioal posterior distributio, i.e.: 4 CONCLUSIONS i) Geerate r usig Equatio (8), ii) Geerate usig Equatio (9), iii) Geerate [ y] usig Equatio (10). his step is coducted iteratively. M-95

6 Pudi Ismartii /Bayesia with Full... ISBN Parameter estimatio techique usig Bayesia approach with full coditioal posterior distributios of the parameters is a alterative solutio that ca be applied to complex modelig cases. his is possible sice the full coditioal posterior distributio is usig the cocept of uidimesioal distributio. hus, the solutio of a complex estimatio process ca be obtaied by usig this iterative method. REFERENCES Box, G.E.P da iao, G.C. (1973), Bayesia Iferece i Statistical Aalysis, Readig, MA: Addiso-Wesley. Carli, B.P. da Chib, S. (1995), Bayesia model choice via Markov Chai Mote Carlo methods, Joural of the Royal Statistical Society, Ser. B, 57(3): Casella, G. da George, E.I. (199), Explaiig the Gibbs Sampler. Joural of the America Statistical Associatio 46(3): Cogdo, P.D. (006), Bayesia Statistical Modellig, d editio, Joh Wiley & Sos, Eglad. Gelma, A., Carli, J B., Ster, H. S. da Rubi, D. B. (004), Bayesia Data Aalysis, d editio, Chapma & Hall, Florida. Gelma, A., da Hill, J. (007), Data Aalysis Usig Regressio ad Multilevel/Hierarchical Models, Cambridge Uiversity Press, UK. Kig, R., Morga, B.J., Gimeez, O., da Brooks, S.P. (010), Bayesia Aalysis for Populatio Ecology, Chapma & Hall/CRC, USA. Ismartii, P., Iriawa, N., Setiawa, da Ulama, B.S.S. (01c), oward a Hieararchical Bayesia Framework for Modellig the Effect of Regioal Diversity o Household Expediture, Joural of Mathematics ad Statistics, Sciece Publicatio, 8(): 83-91, 5 Jue 01. ISSN: , DOI: /mssp Koop, G. (003), Bayesia Ecoometrics, Joh Wiley & Sos, Chichester, Eglad. Lu, D.J., homas, A., Best, N., ad Spiegelhalter, D. (000), WiBUGS A Bayesia Modellig Framework: Cocepts, Structure ad Extesibility, Statistics ad Computig, 10, pp Norstad, J. (011), he Normal ad Logormal Distributios, upublished paper, (31 July 01). Ntzoufras, I. (009), Bayesia Modelig Usig WiBUGS. Wiley, New Jersey, USA. M-96