Our Calibrated Model has No Predictive Value: An Example from the Petroleum Industry

Transcription

1 Our Calibrated Model as No Predictive Value: An Example from te Petroleum Industry J.N. Carter a, P.J. Ballester a, Z. Tavassoli a and P.R. King a a Department of Eart Sciences and Engineering, Imperial College, London, SW7 2AZ, United Kingdom. j.n.carter@imperial.ac.uk. Abstract: It is often assumed tat once a model as been calibrated to measurements ten it will ave some level of predictive capability, altoug tis may be limited. If te model does not ave predictive capability ten te assumption is tat te model needs to be improved in some way. Using an example from te petroleum industry, we sow tat cases can exit were calibrated models ave no predictive capability. Tis occurs even wen tere is no modelling error present. It is also sown tat te introduction of a small modelling error can make it impossible to obtain any models wit useful predictive capability. We ave been unable to find ways of identifying wic calibrated models will ave some predictive capacity and tose wic will not. Keywords: Prediction, Calibration, Uncertainty, Petroleum 1. INTRODUCTION In many studies involving numeric models of complex real world situations, for example petroleum reservoirs and climate modelling, it is implicitly assumed tat if te model as been carefully calibrated to reproduce previously observed beaviour, ten te model will ave some predictive capacity. It is recognised tat predictability may only be acievable for a finite period of time, and tat any prediction will be uncertain to some extent. Two types of error are considered in most calibration exercises: measurement error and model error. Measurement errors are fixed at te time te measurement was made, tey generally ave well defined statistics and can be andled appropriately. Model errors are due to approximations, suc as a loss of spatial, or temporal, resolution, and te noninclusion of all of te relevant pysics. Te assumption tat is normally made is tat if te model errors are sufficiently unimportant, so tat wen te model as been calibrated to measurement data, ten we ave some level of acceptable predictability. If te model does not ave predictability, ten te model errors are assumed to be too large and we need to use a better model. Were better probably means improved resolution, spatial or temporal, and/or te inclusion of more pysics. In tis paper we present te results of a study, for a petroleum reservoir, were a well calibrated model as no predictive value. Even toug te calibration and trut models ad identical pysics and identical spatial and temporal resolution. A second study sows Correspondence to J.N. Carter

2 tat were tere are sligt differences in te pysics between te calibration and trut models, ten te problems encountered are even worse. In te next section te experimental set-up is described, tis is followed by te results for cases wit/witout modelling errors. Finally we draw some conclusions from our observations. 2. EXPERIMENTAL SET-UP In tis section we describe our tree parameter reservoir model and our metodology for calibrating te model against te available measurements Model Description Our model is a cross-section of a simple layered reservoir, wit a single vertical fault midway between an injector producer pair, as sown in figure 1. Te model tat we calibrate as tree parameters: te vertical displacement (trow) of te fault; te permeability of te poor quality sand; and te permeability of te good quality sand. Te geological layers are assumed to be omogeneous (ie tey ave constant pysical properties). Te trut case, wic is used to generate te measurements for te calibration, is a variant of te calibration model, but wit fixed parameter values. In te case of no model error, ten te trut case is a member of te set of all possible calibration models. Te size and type of model error is cosen by ow a specific calibration model is perturbed to obtain te trut case. In te work presented in tis paper, te model error is obtained by introducing small variations into te spatial properties of te geological layers. Te permeability and porosity in eac grid block are randomly perturbed. Te maximum variations tat are allowed is ±1% of te unperturbed mean values. Tese perturbations are muc lower tan would be expected for a real world rock tat ad been classified as omogeneous. A more extensive description of te model can be found a paper tat deals wit estimating model errors[4] Calibration Metodology Our procedure to produce a calibrated model is as follows: 1. Coose trut values for te tree model parameters; 2. Select te level of measurement and model error to be used; 3. From te trut case produce te measurements required for te calibration process (tree years of montly data); 4. Calibrate te model against te measurements; 5. Predict te beaviour for years 4-1.

3 Poor sand (Permeability k p ) Good sand (Permeability k g ) Trow () Figure 1. Reservoir model sowing te fault trow and te geological, and simulation, layers. We ave considered te trut case: = 1.4, k p = 1.31 and k g = wit and witout model error. No measurement error was added, but we assumed Gaussian noise wit a 1% standard deviation wen calculating te likeliood tat a proposed calibration matces te trut. In order to quantify te degree of te model calibration against measurements, we define first an objective function for te calibration period, m, as follows m = j=1 k=1 sim(j, k) obj(j, k) 2σ jk (1) were sim(j, k) is te simulated response for production series k of te model at time j, obj(j, k) is te corresponding true value and σ jk, an estimation of wat would be te associated measurement error. We consider tree production series: Oil Production Rate, Water Production Rate (or Water Cut) and Water Injection Rate. Likewise, te objective function for te prediction period, f, is f = j=37 k=1 sim(j, k) obj(j, k) 2σ jk (2) Te ranges tat te model parameters were allowed to take are: (, 6), k g (1, 2) and k p (, 5).

4 exp( Dm/.15) exp( Df/.15) Figure 2. Calibrations of te model (wit no modelling error) to a) istory period, b) prediction period Genetic Algoritm Our cosen searc metod is a Steady-state Real-parameter Genetic Algoritm. Tis is used because we need to searc for multiple good optima witin a parameter space tat seems to contain very many local optima. It is a development of a previously publised study [1] and as been developed to solve te type of problem described in tis paper. In brief te details are: a steady-state population of 5 individuals is used, parents are selected randomly (witout reference to teir fitness), crossover is performed using vsbx[1, 3], and culling is carried out using a form of tournament selection involving 1 individuals, a total of 7 individuals are generated. 3. RESULTS In tis section we present te results of two studies: te first is wit no modelling error present; te second as a low level of modelling error Calibration wit No Modelling Error Figure 2a sows te result of calibrating te model against te data for te first 36 monts. Te trut model as exactly te same pysics and structure as te calibration models, and te trut model is a member of te set of possible calibration models. Te very large spike, wit 1, corresponds to te trut case. We can also see notable local optima wit < < 8, 3 < < 38 and 4 < < 45. Te global optimum as a small basin of attraction around it and as proved difficult to identify in previous work[2], te easiest optimum to find as been te one wit 3 < < 38. Te rater noisy structure of te objective surface is largely an artifact of te of te way tat k g is sampled. Any point wit an acceptable objective value is plotted no matter wat value of k g was used. Tis means tat it is possible for two points to ave identical values for and k p but different values of te objective function. Hence a vertical line would be plotted. Figure 3 sows a contour plot, centred on = 5. and k p = 1.65, of te objective

5 Dm(opt) Figure 3. Surface plot for m, were k g as been optimised so as to minimise m, (3.5, 6.5) and k p (1.4, 1.9) function m. Te figure was generated by conducting a grid searc on a fine grid. At eac point on te grid, k g was optimised, tis results in a muc smooter representation of te objective. Figure 2b sows te result of calibrating te model to te prediction period. Te only substantial point found corresponds to te trut model. All of te oter local optima tat can be seen in figure 2a are unable to matc te observations during te prediction period. We conclude tat for tis model you can only obtain a good prediction from te trut case, and tat good matces from te istory matcing pase ave no predictive value Calibration wit Modelling Error Te result of matcing te calibration model to data generated by a trut case tat includes modelling errors is sown in figure 4a. Superficially te figure is similar to figure 2a. Te important difference is tat te global optima now occurs for 32. Tis is witin te largest basin of attraction and is usually found by most searc algoritms. Te optima associated wit te true parameter values is of muc lower quality. If we now look at te calibration to te prediction period, figure 4b, we see tat te global optimum for te istory matcing period as no predictive value. None of te models tat ave some predictive value correspond to te trut case (te spike at 1 as te wrong values for k p and k g ). Te objective values obtained are low compared to tose in figure 2b. 4. CONCLUSIONS In tis paper we ave examined, for a particular case, our ability to calibrate a model and ten to make accurate predictions. Tis as been carried out for cases wit and witout modelling errors, but no measurement error. From tese studies we make te following observations:

6 exp( Dm/.15) exp( Df/.15) Figure 4. Calibrations of te model (wit modelling error) to a) istory period, b) prediction period. Te basin of attraction a round a global optimum may be sufficiently small tat searc algoritms may not find tem. Te basins of attraction associated wit oter local optima may be muc larger and ence easier to find. Wen tere is no modelling error present, some of te non-global optima may be of quite good quality. However only te global optimum is able to make an accurate prediction. Wen small amounts of modelling error are present, ten te global optimum is no longer associated wit te trut. Te local optimum tat as parameter values of te trut case is not of significant quality and could easily be disregarded. None of te models tested in te presence of modelling errors ave valuable predictive power. In particular te global optima from te istory matcing period was unable to provide an accurate prediction. In summary: in te absence of model errors, and wit very low measurement errors, it is possible to obtain calibrated models tat do not ave any predictive capability; suc models may be significantly easier to identify tan te correct model; we are unable to differentiate between calibrated models wit or witout predictive capabilities; te introduction of even small model errors may make it impossible to obtain a calibrated model wit predictive value. In tis analysis tere is noting tat seems to be unique to tis model. In particular tere is te issue of data availability, adding more measurements does not appear to offer a guaranty of avoiding tis dilemma. If te observations made wit tis model are not unique to te model, and we ave no reason to believe tat te model is unique, ten tis presents a potentially serious obstacle to te use of models of tis type for prediction. Our concern is tat if we cannot successfully calibrate and make predictions wit a model as simple as tis, were does tis leave us wen are models are more complex, ave substantive modelling errors, and we ave poor quality measurement data.

7 REFERENCES 1. P.J. Ballester and J.N. Carter,Real-parameter Genetic Algoritms for Finding Multiple Optimal Solutions in Multi-modal Optimisation, Genetic and Evolutionary Computation Conference 23:76, Lecture Notes in Computer Science 2723, Pub Springer-Verlag, Heidelberg, M.D. Bus and J.N. Carter, Applications of a Modified Genetic Algoritm to Parameter Estimation in te Petroleum Industry, Intelligent Engineering Systems Troug Artificial Neural Networks 6:397, J.N. Carter, Introduction to using Genetic Algoritms, in Soft Computing and Intelligent Data Analysis in Oil Exploration, Eds M. Nikraves, F. Aminzade and L.A. Zade, Developments in Petroleum Science 51, Elsevier Science, J.N. Carter, Using Bayesian Statistics to Capture te Effects of Modelling Errors in Inverse Problems, Matematical Geology, 36:187, 24.