CLOUD RACKING FOR SOLAR IRRADIANCE REDICION Ming-Ching Chang, Yi Yao, Guan Li, Yan ong, ete u GE Global Reseach Cente, Niskayuna, NY USA ABSRAC We popose a video analytic system to segment and tack clouds fo the pupose of sola iadiance pediction. Goundbased imaging sensos ae used to monito potential sun occlusions fo sola iadiance dop pediction, which can be used to assist powe amp contol stategies. Sky images ae fist ectified to emove fisheye atifacts. Cloud pixels ae segmented and classified into low-to-high tanspaencies. Evolution of cloud boundaies ae tacked using optical flow. acking extapolation pedicts futue cloud movements and defomations as well as potential sun occlusions. o accuately estimate sola iadiance dop, we popose a novel scheme based on back-pojecting the pedicted sun occlusion onto the evolving cloud bounday to count fo cloud tanspaency and iadiance dop estimation. Expeimental esults show that shot-tem sola iadiance dop is pedictable with easonable accuacy. Index ems sola iadiance pediction, cloud tacking, optical flow, back pojection.. INRODUCION A majo issue of sola powe is its fluctuation due to unstable sunlight iadiance. Cloud occlusions ae known to be a key contibuto to this effect [, 2]. he effects of clouds on sola iadiance dop depend on the cloud s volume, height, and thickness [3]. In this pape, we pesent a shot-tem sola foecasting system based on visual tacking of clouds fom a gound-based sky image. By segmentation and tacking cloud pixels, we show that sun occlusion can be estimated fo iadiance dop pediction. Ou appoach can effectively impove sola amp contol [4] and opeational cost. Ou video analytic pipeline consists of the following steps (summaized in Fig.: (i One o moe wide-angle cameas ae used to captue lage egions of the sky. (ii he sky image is ectified to emove fisheye distotions, so that cloud movements and defomations can be tacked and pedicted using linea pojective models. (iii Cloud pixels ae segmented and thei tanspaencies ae estimated. (iv Hon/Schunck optical flow is applied to calculate the motion field of the cloud Ming-Ching Chang is cuently with the Electical and Compute Engineeing Depatment, Univesity at Albany, SUNY, NY, USA. Yi Yao is cuently with Sanoff Copoation, NJ, USA. Guan Li is cuently with Zillow Goup, Seattle, USA Yan ong is cuently with Univ. of South Caolina. Fig.. oposed pipeline of visual cloud tacking fo sola pediction. bounday. (v Cloud defomations and movements ae pedicted via extapolation of the motion field. (vi Occlusion of the sun is pedicted along with iadiance dop estimation based on estimates of cloud tanspaency. he main contibution of this pape is a video analytic system focused on cloud tacking and sun-occlusion analysis fo sola iadiance pediction. he system integates a wide ange of pactical compute vision techniques including camea calibation, image ectification, segmentation, classification, optical flow, tacking and pediction. We popose a novel back-pojection method fo a quantitative pediction of sola occlusion by consideing both the cloud motion field and cloud tanspaencies. Related Woks: hee exists extensive eseach on sky images and cloud tacking fo sola foecasting [5]. Analysis typically focuses on cloud type classification, thickness, tanspaency, velocity, height, and elative positions w..t. the sun. Most existing methods ae based on simple image pocessing methods to detemine cloud egions. Relatively little effot was been devoted to developing an end-to-end solution fo sola occlusion pediction. he UCSD otal Sky Image [6, 4, 7] segmented and detemined cloud type by classifying sky pixels (clea sky, optically thin and thick clouds using the RGB ed-blue atio. heshold paametes wee chosen using manually annotated images. Cloud motion vectos wee calculated using coss-coelation between consecutive images, whee only ough global motion is estimated. he wok of [8] employs a simila pipeline of cloud pixel segmentation and Lucas-Kanade optical flow [9] fo sun occlusion estimation up to 40 seconds. In contast, we pefom a pobabilistic classification of cloud types (in 2, and we estimate cloud motion using globally efined Hon/Schunck optical flow [0] (in 3 to geneate a longe pediction (as shown in 5. 2. SKY IMAGER SEU, CLOUD SEGMENAION AND RANSARENCY ANALYSIS We fist descibe ou sky image hadwae configuation. o avoid camea ove-exposue due to diect obsevation of the sun, an effective way fo sun position estimation o block-
(a (b Fig. 2. (a Input wide-angle fisheye and the view afte ectification. (b Calibation using a standad checkeboad patten using OCamCalib [], whee main steps include cone extaction, e-pojection eo analysis, and the ecovey of camea paametes and checkeboad positions in 3D. (a Image I (i sigmoid(h (b fbmg (j fh (c pbmg (k p(h (d fb (l v (e pb (m vn (f p(cloud (n fl (g segmentation (o p(l (h hue (p tanspaency Fig. 3. Cloud segmentation and tanspaency classification. (a Input sky image. (b fbmg = b min(, g. (c pbmg = sigmoid(fbmg. (d fb = b with intensity scaling by multiplication of 00 and visualized in [0, 255]. (e pb = sigmoid(fb. (f Cloud mask p(cloud. (g Cloud segmentation mask afte thesholding and mophological filteing. (h Hue map h. (i sigmoid(h. (j High-tanspaency featue fh = p(cloud [ sigmoid(h]. (k High-tanspaency map p(h = sigmoid(fh. (l Intensity map v. (m Nomalized intensity map vn. (n Low-tanspaency featue fl = p(cloud [ sigmoid(vn ]. (o Low-tanspaency map p(l = sigmoid(fl. (p Final cloud tanspaency label map. ing is equied. he UCSD system [4, 7] uses a mechanical moving stipe to physically block the sun. In contast, we exploit a softwae solution by calculating the sun location in the image view based on the Sola osition Algoithm (SA [2] and camea calibation. Specifically, given the cuent UC time and the site location in tems of global longitude and latitude, we follow Jean Meeus s algoithm [3] detailed in [2] to pecisely calculate the sola zenith and azimuth angles, and convet the sun location to pixel coodinates in ou sky image using calibated camea (intinsic and extinsic paametes. Fig.4c depicts an example esult, whee the two geen ellipses indicate the sun mask and an enlaged ove-exposed egion. We denote S τ the sun mask at time τ fo late use in occlusion estimation. We use a wide angle lens to ensue full sun coveage thoughout the day. hus fisheye ectification is pefomed, such that linea velocity models can be assumed fo cloud motion pediction. We use the OCamCalib [] to calculate the ectification fom a set of checke boad images, whee both the adial and de-centeing distotions ae estimated [4, 5] via nonlinea optimization. Camea intinsic and extinsic paametes ae deived by minimizing the e-pojection eos of the image gids. Fig.2 summaizes the main steps. We descibe an empiical ule-based, pobabilistic appoach to pe-fame cloud pixel analysis fo joint segmentation and classification. Senso-dependent paametes ae empiically detemined at the optics setup stage. We fist detemine whethe a pixel belongs to the cloud o the backgound (sky o othe. Fo a cloud pixel Ix, we define a pixel-wise classification function (Ix that classifies Ix into fou types: high- (h, mid- (m, and low- (l of tanspaencies and the backgound (b. Given an input image I, the RGB and HSV fo a pixel Ix ae denoted as Ix = (, g, b and Ix = [h, s, v] espectively, whee 0, g, b, v 255, 0 h 360, and 0 s. In the liteatue [, 6, 2, 3], the ed/blue atio ( b [4,, 6, 2] and ed-blue diffeence ( b [3] ae often used fo cloud segmentation, since clea sky usually scattes moe blue than ed light, and the clouds scatte both blue and ed light almost equally. Howeve, we found this simple appoach did not pefom well when faced with typical envionmental vaiations. Instead, we adopt the pobabilistic logistic sigmoid fo obust classification. Specifically, we define two featues based on (i the diffeence between the blue and the minimum of ed and geen, fbmg = b min(, g, and (ii flip the ed/blue atio, fb = b. Both featues ae mapped to a nomalized ange of [0, ] using a sigmoid function, pbmg = fbmg µbmg, whee µbmg and σbmg epesent +exp( σbmg the mean and standad deviation of cloud pixels, espectively. pb = is calculated similaly. Ou soft f µ +exp( bσ b b cloud segmentation is deived as: p(cloud = pbmg pb. hesholding followed by mophological filteing esults in the final segmentation map, see Fig.3a-g. Detemination of high-tanspaency clouds: We obseved that both the RGB and hue h of a tanspaent cloud ae close to that of clea sky, while the cloud can still be obustly segmented, as in Fig.3a,h. We thus intesect the cloud mask and the hue map to identify the tanspaent clouds. Specifically, the diffeence between pbmg and the sigmoid of hue ph = epesents a useful feah µh +exp( σh tue. Finally, a high-tanspaency cloud featue is defined as f (h = pbmg ph. Detemination of low-tanspaency clouds: We obseved that low-tanspaency clouds ae typically dak and thick, and hence have lowe pixel bightness as in Fig.3a,l.
Fig. 4. (a Velocity estimation by aveaging out the bounday optical flow field. (b An example velocity estimation map B t at time t. Red cuves outline the detected cloud egion. Blue aows indicate estimated velocity field on the cloud boundaies. he length of the aows ae popotional to the magnitude of the estimated velocities. (c Cloud motion pediction mask Ĉτ at a futue time τ = t + t. (d,e sun occlusion calculation using back-pojection (see text. We use the diffeence between cloud map and the bightness v as a featue fo this type of cloud. We fist nomalize the v map by v n = v vmin v max v min as in Fig.3m. We then calculate p vn = +exp( vn µvn σvn. A low-tanspaency featue is defined as f( l = p bmg p vn. Finally, the pobability of high- p( h and low- p( l tanspaencies can be calculated by nomalizing f( h and f( l to the ange of [0, ] using additional sigmoids. Afte futhe mophological filteing, the emaining cloud pixels belong to the mid-tanspaency m. Fig.3 illustates ou complete ule-based cloud segmentation pipeline. Fo each input image I, connected component labeling is pefomed to detemine the set of cloud egions {C i } b i= I, whee b is the total numbe of egions. 3. CLOUD MOION ESIMAION Afte each cloud egion C i is defined, such egions can be visually tacked based on appeaance and shape matching. Since cloud evolution is typically non-igid and can split and mege, we dynamically tack the bounday B i of each cloud C i using Hon/Schunck optical flow [0]. he displacement is calculated fo fine pixel gids between consecutive fames, thus the aw estimated velocity is mostly noisy. o this end, we aggegate velocity estimation fom a lage bounday egion fo obust estimation as in Fig.4a. Specifically, fo each cloud bounday point of inteest I x B i (ed dot in Fig.4a, we detemine the velocity v(i x by calculating the mean velocity of all points {x} in the bounday stip Ω x aound I x, x Ωx v(ix Ω x, whee Ω x denotes stip size n. i.e., v(i x = Fig.4b shows an example of the esulting velocity field. Fo each cloud bounday B i at the t-th fame, given the estimated bounday velocity filed Vi t = {v(it x} of B i, the next step is to pedict cloud bounday movements in the futue τ = t + t fame, whee the pediction inteval t is a pespecified system paamete. Ou goal is to estimate the pobability of any pixel Îτ x in the τ-th fame belonging to a cloud bounday. We add the ˆ symbol (e.g., Î to indicate a pedicted futue quantity thoughout this pape. We define a label function L(Îτ x, which takes value of if the pixel Îτ x at time τ belongs to a cloud bounday and 0 othewise. he cloud bounday is pedicted by intesecting all individual pedictions, whee each individual pediction p ( L(Îτ x = v(i t x follows a Gaussian distibution fom extapolating the velocity v(i t x with inceasing vaiance: p(l(îτ = V t = [ ( ] p L(Îτ x = v(ix t. x I Specifically, the Gaussian mean µ is deived fom the velocity magnitude and the pediction ( inteval, following a simple linea motion model µ v(îτ x = Ix t + v(ix t. t he Gaussian vaiance gows as the pediction inteval t inceases. he intesection of Gaussian pediction bounday compises a pobabilistic map indicating the pobability of any given pixel Îτ x at the futue fame τ belonging to a pedicted cloud bounday. hesholding this map yields the pedicted cloud bounday mask L(Îτ. Fig.4d depicts an illustative example. Connected component inteio filling is applied to poduce the set of pedicted cloud egions {Ĉτ i }b i=. We expeimented with pe-ecoded videos, so that we could veify the cloud motion pediction {Ĉτ i }b i= ovelaid on top of the tue futue cloud image I τ. Fig.4c illustates an example esult, whee each pedicted cloud egion Ĉτ i is highlighted in light ed. he goundtuth cloud bounday B τ in the futue fame ae visualized in two colos. Blue indicates successful pedictions whee the goundtuth bounday falls within the pedicted mask, and solid ed indicates failue pedictions. Obseve that the majoity of the successful pediction pixels, indicating a high pediction accuacy. 4. SOLAR OCCLUSION REDICION Given the set of pedicted cloud egions {Ĉτ i }b i= and the obtained sun mask S τ, we can pedict sun-cloud occlusion as well as estimate the amount of the anticipated sola iadiance dop. We use a flag function F (τ to indicate the pediction of an occlusion event that will cause sola iadiance dop, i.e., F (τ = suggests a sun occlusion and 0 othewise. A naive solution is to count the total sun-cloud intesection pixels o aea A τ = b i= Ĉτ i S τ followed by a applying a theshold θ A, i.e., F (τ = if A τ > θ A and 0 othewise. We obseve that in pactice the calculation of A τ is vey inaccuate. Simple thesholding of A τ esults in excessive switches of the flag function, thus esulting in false tiggeing of events. o obtain a moe obust sola occlusion estimation, we enfoce the constaint that F (τ can only switch values when the above thesholding condition A τ > θ A is accumulated fo moe than sufficient numbe of consecutive fames. o this end, we intoduce two additional accumulation vaiables c and c + and apply two espective accumulation thesholds θ c+ and θ c fo obust pediction of sun occlusion events:
v(ci = I Bi and Ix Is ds x v(ix Ix Bi and Ix Is ds moved to I si = Is tv(ci (2 he sun cente is in ode to geneate a back-pojected sun egion S i. If S i intesects with cloud egion Ci, the iadiance decease caused by Ci is given by Ii = (a (b Fig. 5. Evaluations. Compaison between the ecoded goundtuth and pedicted iadiance measues fo (a a 40-minutes sequence at two pedictions of 2.5 minutes ( t = 50s and 5 minutes ( t = 300s, whee (-4 depicts obseved images. (b a 7.5-hous sequence ecoded at 0. FS. he fist view shows the input image with detected cloud bounday and the optical flow motion field. he second view shows the cloud segmentation with estimated tanspaency classes. he thid view shows the futue fame, the calculated sun mask and the pedicted cloud bounday. Red cuve depicts gound tuth. ink cuve depicts the inteaction-aea based occlusion. Cyan cuve depicts the inteaction-aea consideing cloud tanspaency. Yellow cuve depicts the sun iadiance dop estimation cuve. If Aτ θa, incease c+ by one and eset c = 0. If Aτ < θa, incease c by one and eset c+ = 0. If c+ > θc+, set F (t = and eset c+ = 0. If c > θc, set F (t = 0 and eset c = 0. Sola iadiance dop pediction: he above accumulated intesection-aea based sun occlusion estimation can be impoved by consideing cloud tanspaency that bette coelates with sun iadiance dop. Recall that each cloud pixel is classified into thee categoies: high-, medium-, and lowtanspaency. Fo each categoy, we assign a fixed weighted amount of iadiance dop when a sun occlusion occus. We define such type-specific iadiance dop estimations as: if (Ix = h 2 if (Ix = m. I (Ix = ( 3 if (Ix = l he pedicted iadiance dop is the aveaged ove all pixelwise estimations I = i Ix S i S C τ I i C iτ (Ix. 4.. Estimate sola iadiance dop using back-pojection Note that the above intesection-aea based sun occlusion estimation only consides a single cloud occlude event. Obseve in Fig.4e that multiple cloud occludes o fonteis can occu at the same time, and ovelapping occlusions need to be count individually. Also, the dynamic natue of cloud motion makes it had to extapolate the tanspaency estimation. o this end, we popose a novel algoithm to back-poject the pedicted sun occlusion to the cloud bounday fo occlusion easoning. In this way occlusions can be accuately and quantitatively accumulated fo iadiance dop estimation. Fo each detected cloud egion Ci, an aveage speed v(ci is computed based on all the velocities of the bounday pixels that ae sufficiently close to the sun image (i.e. < theshold ds. Let Is denote the sun cente pixel. Ix S i S i Ci I (Ix. Consideing all detected cloud Ci Ix S i C I (Ix i I = i. Ci i S i egions, we have Fig.4d,e illustates the back-pojection sun occlusion calculation. Ou method estimates the iadiance dops (i.e., changes of iadiance ove time (instead of an absolute value. We shift the esulting iadiance measue to zeo using clea sky i.e. when the sun is not obscued at all. 5. EXERIMENAL VALIDAION We implemented the descibed system in C++ on a fan-less embedded industial compute box, which ae deployed next to GE s sola testing site at Schenectady, NY. he pipeline can un up to 0 FS, howeve 0. FS is sufficient fo ou sola foecasting application. We collected sky video sequences using a Nikon camea with esolution 936 296. We also use an iadiance senso to collect goundtuth simultaneously at a fequency of one sample pe second. All sigmid paametes fo cloud segmentation and classification in 2 ae detemined empiically, e.g., µbmg = 5, σbmg = 5, µb =.0, σb = 0.3, µh = 0.2, σh = 0.05, µvn = 0.9, σhn = 0.2. We tested ou algoithm with two pediction intevals t = 50s and t = 300s with identical paamete settings. Fig.5a epots esults on a sequence contains 426 images (45 minutes, whee images ae taken evey 2 minutes. At time (, the cloud on the left begins to occlude the sun, wheeas at time (2 the cloud completely occludes the sun. As the clouds keep moving towad ight, the dake cloud begins to block the sun at time (3. he sun is fully coveed at time (4. We define the pe-fame cloud bounday pediction accuacy τ i C iτ i Bi fo qualitative evaluation. Fo a video, A= i Biτ the aveage of A s ove all fames ae used. We obtained A = 0.85 fo t = 50s, and A = 0.86 fo t = 300s, which demonstated the effectiveness of ou method. Fig. 5b epots esults on a longe sequence about 7.6 hous at 0. FS. Obseve how the pediction cuves align and coelate with the goundtuth iadiance cuves. Note that ou esulting iadiance pediction cuve (yellow can be futhe scaled by the oveall iadiance of a clea sky fom measues thoughout a day, to yield a long-tem iadiance evaluation that bette matches the (ed goundtuth envelope cuve. he Bid Clea Sky Model [7] authoed by Richad Bid seves well fo such scaling pupose. 6. CONCLUSIONS We pesent a video analytic system to segment and tack clouds fo shot-tem sola iadiance dop pediction. Futue wok includes fine classification of cloud types and tanspaencies.
7. REFERENCES [] G. fiste, R. L. McKenziea, J. B. Lileya, A. homasa, B. W. Foganb, and C. N. Long, Cloud coveage based on all-sky imaging and its impact on suface sola iadiance, Jounal of Applied Meteoology, vol. 42, no. 0, pp. 42434, 2003. [2] A. Keute, M. Zangel, M. Schwazmann, and M. Blumthale, All-sky imaging: a simple, vesatile system fo atmospheic eseach, Applied Optics, vol. 48, no. 6, pp. 09 097, 2009. [3] A. Heinle, A. Macke, and A. Sivastav, Automatic cloud classification of whole sky images, Atmospheic Measuement echniques, vol. 3, no. 3, pp. 557 567, 200. [4] M. S. Ghonima, B. Uquhat, C. W. Chow, J. E. Shields, A. Cazola, and J. Kleissl, A method fo cloud detection and opacity classification based on gound based sky imagey, Atmos. Meas. ech. Discuss., vol. 5, pp. 4535 4569, 202. [2] Ibahim Reda and Afshin Adneas, Sola position algoithm fo sola adiation applications, ech. Rep., NREL, http://www.nel.gov/docs/fy08osti/34302.pdf, 2008. [3] J. Meeus, Astonomical Algoithms, Willmann-Bell Inc., Richmond, Viginia, USA, 2 edition, 998. [4] Z. Zhang, On the epipola geomety between two images with lens distotion, Intenational Confeence on atten Recognition, vol., pp. 407 4, 996. [5] D.C. Bown, Decenteing distotion of lenses, hotogammetic Engineeing, vol. 7, pp. 444 462, 966. [6] C. N. Long, J. M. Sabbug, J. Calbó, and D. agès, Retieving cloud chaacteistics fom gound-based daytime colo all-sky images, Jounal of Atmospheic and Oceanic echnology, vol. 23, no. 5, pp. 633 652, 2006. [7] R. E. Bid and R. L. Hulstom, Simplified clea sky model fo diect and diffuse insolation on hoizontal sufaces, echnical Repot No. SERI/R-642-76, Golden, CO: Sola Enegy Reseach Institute, 98. [5] D. K. Chatuvedi and Isha, Sola powe foecasting: A eview, Intenational Jounal of Compute Applications, vol. 45, no. 6, 206. [6] Chi Wai Chow, Sege Belongie, and Jan Kleissl, Cloud motion and stability estimation fo inta-hou sola foecasting, Sola Enegy, vol. 5, pp. 645 655, 205. [7] Chi Wai Chow, Byan Uquhat, Matthew Lave, Anthony Dominguez, Jan Kleissl, janet Shields, and Byon Washom, Inta-hou foecasting with a total sky image at the UC San Diego sola enegy testbed, vol. 85, no., 20. [8] hilip Wood-Badley, Jose Zapata, and John ye, Cloud tacking with optical flow fo shot-tem sola foecasting, in Austalia and New Zealand Sola Enegy Society Confeence, 202. [9] B. Lucas and. Kanade, An iteative image egistation technique with an application to steeo vision, in Intenational Joint Confeence on Atificial Intelligence, 98, pp. 674 679. [0] B. K.. Hon and B. G. Schunck, Detemining optical flow., Atificial Intelligence, vol. 7, pp. 85 203, 98. [] D. Scaamuzza, A. Matinelli, and R. Siegwat, A toolbox fo easy calibating omnidiectional cameas, IEEE Intenational Confeence on Intelligent Robots and Systems (IROS, pp. 297 30, 2006.