Super-resolution Frame Reconstruction Using Subpixel Motion Estimation

Transcription

1 Super-resolution Frame Reconstruction Using Subpixel Motion Estimation ABSTRACT When vieo ata is use for forensic analysis, it may transpire that the level of etail available is insufficient. This is particularly the case when it is erive from sensors eploye in the fiel since they have limite resolution. For example, a vehicle license plate number may not be legible in vieo capture uring a traffic violation. In such cases, it is esirable to enhance the resolution of selecte frames. However, obtaining the imum resolution possible is computationally prohibitive. Forensic analysts nee a tool where the traeoff between etail extracte an the running time can be controlle. We escribe a parametrize algorithm for obtaining higher resolution frames using temporal context in the vieo stream. By varying the framework s variables, the quality of the output an the cost of computing it can be trae. We briefly report on our proof of concept implementation as well. 1. INTRODUCTION Vieo is routinely use to recor events for analysis at a later time. Surveillance cameras are use to monitor a range of locations, incluing prouct shelves in shops, traffic on roas, entrances to builings, an users of ATMs (automatic teller machines). When a crime occurs, the recore footage may be use to ientify the perpetrator. Since the cost of a vieo sensor is irectly relate to the resolution at which it captures single frames, the size chosen is usually a compromise between cost an function. As a result, when the ata is being use for forensic applications, it may be necessary to enhance its resolution. For example, this may be the only way of reaing the license plate on a passing car. The cost of using temporal context to improve the resolution of a single frame is significant. Hence, a forensic analyst woul typically prefer to select the extent of improvement in resolution that is require, since this woul also control how much time the process woul take. Our algorithm an prototype implementation provie this Backgroun In the context of igital vieo, an approximation occurs uring the recoring process. The image space in R 2 is approximate with a representation gri in N 2, which we call the approximate image space. Thus, regions in R 2 are represente by a single ata point, a pixel, as illustrate in Figure 1. By repeating the spatial registration process at iscrete points in time, a vieo sequence is forme. Recoring the instant in time with a temporal inex is accomplishe by a strictly increasing monotonic mapping from R to N. Finally, the set of points that unergoes spatial projection will vary from frame to frame. Vieo compression algorithms rely on the eye s inabaility to iscern small variations. For example, quantization maps a range of intensity levels to a single one since it can be represente more succinctly. Our goal is to extract an utilize this variation instea. It will provie us with extra information about regions in temporally proximal frames. X XO O XO # # # X X X O O O # # # X X X O O O # # # R Points enote with X are elements of S t+n Points entoe with O are elements of St Points enote with # are elements of S t-n O O X X X X X X X X X # # # O St+n Figure 1. The image sequence that recors the region R, registers similar but not ientical sets of points in object space (i.e. the original scene). S t S t-n

2 1.2. Problem Specification Given a vieo sequence, we wish to efficiently enhance the resolution of the representation gri in approximate image space, such that the result is the equivalent of having mappe a larger set of points, from the object space to the image space, than actually occurre in any given frame. Further, we woul like the resulting set s size to be a (preferably linear) function of the computation expene Approach We note that by approaching the problem using only spatial context, we can only obtain guesses of the the intensity values that shoul be assigne to the newly efine points (that result from the higher resolution). Numerous methos exist to o this such as linear interpolation, natural spline interpolation, or the application of a Gaussian filter to a region of points aroun the one whose intensity is being estimate, for each of the new points. However, sub-pixel features in the original scene may not be recovere by this process. Therefore, we use the insight that in consecutive frames, f t,...,f t+n, of the vieo sequence, the same region R in the object space is capture uring the recoring process, by mapping ifferent sets of points, S t,...,s t+n, of the object space onto the image space, as illustrate in Figure 2. Thus Camera Registration Gri Points on surface mappe to first frame Points on surface mappe to secon frame Points on surface mappe to thir frame Surface of object Figure 2. We assume that the camera samples a ifferent set of points on the object in each frame. the sets S t,...,s t+n, are very close approximations of each other, but not ientical. This is because the nature of analogue to igital conversion is that of a many to one mapping of an uncountable set onto a countable set. We will procee by retrieving information about the region in the object space R, corresponing to the image space I t uner consieration, from representations of the same region R in previous an future frames, I t n,...,i t 1,I t+1,...,i t+n. With a larger set of points mappe from the object space to the image space, we will be in a position to create a higher resolution vieo frame that captures sub-pixel features of the original object Overview of Algorithm Vieo Frames Motion Vectors (x,y) Determine For Each Block a b c Similar Blocks Sub-Pixel Motion Re-Estimation (x,y) abc abc abc abc abc abc abc abc abc Spline Interpolation Scattere Data Points Figure 3. Overview of algorithm. High Resolution Frame Spline Surface Evaluation Spline Surface Each frame of the vieo sequence is enhance inepenently in serial orer. To process a given frame, a set of frames is forme by choosing several frames that are immeiately previous an several that are immeiately after the frame uner consieration. The frame that is being improve is then broken up into rectangular blocks. Extra ata about each block in the frame is then extracte. For a given block, this is one by performing a search for one block from each of the frames in the above efine set that best matches it. A sub-pixel isplacement that will minimize the error between each of the contextual blocks an the block uner consieration is analytically calculate for each of the blocks in the frame. The ata thus obtaine can be viewe as a set of points of the form (x,y,z), where z is the image intensity at a point (x,y) in

3 the XY plane. The set thus obtaine will correspon to points scattere in the XY plane. To map this back to a gri, a bivariate spline surface, compose of tensor prouct basis splines, that approximates this ata is compute an evaluate. Since this is one for each frame in the sequence, a sequence of similar length to the input is prouce as output. 2. EXTRACTION OF TEMPORAL CONTEXT Our algorithm enhances the resolution of the vieo sequence by increasing the size of the ata set associate with each frame. Consiering one reference frame, whose resolution we are currently improving, without loss of generality we will consier the problem of extracting information of relevance from only one other frame, the contextual frame. The same process may be applie to a set of other frames immeiately before an after the reference frame in the vieo sequence in orer to obtain the increase size ata set representing the region in object space that was mappe to the reference frame. Ieally, whenever an object in the reference frame appears in the contextual frame, we woul like to be able to ientify the exact set of pixels that correspon to the object, extract the set an incorporate it into the representation of the object in the reference frame. To effect this search, we use a technique from vieo compression, calle motion estimation Initial Motion Estimation Vieo compression achieves higher compression ratios than still image intraframe coing, the process of reucing spatial reunancy in a frame, by hybriization with interframe coing, which exploits temporal correlation in the ata. This is one by compensating for the motion of objects, from frame to frame, by ientifying an object, estimating its motion, an then encoing the movement as a vector representing an affine transformation. Currently research on computing optical flow that woul yiel such representations exists, but for computationally efficient solutions 1 that give a goo approximation, stanars such as MPEG 2 an H.261 have aopte a ifferent approach. In the general case, we woul allow for a range of motion by the object, taking into account scaling, rotation, an warping in aition to translation. The implication of this is that the geometry of the omain an range of the motion mapping nee not be the same. This leas to complex geometrical concerns as well as intensive computational requirements. Stanars such as MPEG eal with this by arbitrarily ecomposing a frame into blocks an working at the granularity of blocks 3 with the assumption that there is no rotation occurring. This is the equivalent of imposing a uniform motion vector fiel on all the pixels in the block. To ensure the correctness of the algorithm, a means of encoing the motion of the pixels that o not have the same motion as the block is introuce in the form of a resiual coe, which is the ifference of the actual representation an that impose on the pixel by the block granularity motion vector. The reference frame is broken into blocks of fixe size. (The MPEG stanar specifies the use of size blocks. 4 We leave the block imensions as a parameter of the algorithm to etermine the best imensions for our purpose.) The size of the block must be etermine as a function of several factors that necessitate a traeoff: By using smaller blocks, the finer granularity allows for local movements to be better represente. The uniform motion fiel is force on a smaller number of pixels. The isavantage is the increase in computational requirements. Depening on the matching criterion, smaller blocks may also result in more incorrect matches, since bigger blocks provie more context increasing the probability of correctly etermining an object s motion vector. For each of these blocks, a search for a block of ientical imensions is performe in the contextual frame. A metric that associates cost inversely with the closeness of the match is use to ientify the block with the lowest cost. The vector connecting a point on the reference block to the corresponing point of the best match block foun in the contextual frame, is etermine as the initial motion estimate for that block.

4 Matching Criteria If a given vieo frame is of with F w an height F h, an the sequence is T frames long, then a pixel at position (m,n) in the f th frame, is specifie by V S(m,n,f), where m {1,...,F w }, n {1,...,F h }, an f {1,...,T }. If a given block is of with B w an height B h, then a block in the reference frame with its top left pixel at (x,y) is enote RB(x,y,t), where x {1,...,(F w B w +1)}, y {1,...,(F h B h +1)}, an t is the temporal inex. For a fixe reference frame block enote by RB(x,y,t), the block in the contextual frame that has its top left pixel locate at the position (x + a,y + b) is enote by CB(a,b,u), where (x + a) {1,...,(F w B w + 1)}, (y +b) {1,...,(F h B h +1)}, an where the contextual frame has temporal inex (t+u). The etermination that the object, represente in the block with its top left corner at (x,y) at time t, has been translate in exactly u frames by the vector (a,b), is associate with a cost given by the function C(RB(x,y,t),CB(a,b,u)). We chose to use Mean Absolute Error (MAE), or Mean Absolute Difference (MAD) as the cost function on the basis of empirical stuies in the literature on MPEG stanars 1,5 that have shown it works as effectively as more computationally intensive criteria, where MAD(RB(x,y,t),CB(a,b,u)) = (1) 1 B w B h is the specific cost function. (B w 1) i=0 (B h 1) j=0 V S(x + i,y + j,t) V S(x + a + i,y + b + j,t + u) Fining the Motion Vector Given a reference block RB(x, y, t), we wish to etermine the block CB(a,b,u), that is the best match possible in the (t + u) th frame, with the motion vector (a,b) constraine to a fixe rectangular search space centere at (x, y). This is illustrate in Figure 4. If this space s imensions are specifie as the parameters p an q of the algorithm, then a {(x p),...,(x + p)}, an b {(y q),...,(y + q)} must hol. For sequences with little motion these values can be mae small for computational efficiency. If the motion characteristics of the sequence are unknown, p an q may be set so that the entire frame is searche. The only metho that can etermine the best choice CB(a min,b min,u) with probability one, such that F h B h F w B w Outline of reference frame Outline of contextual block Object in contextual block Object in reference frame Figure 4. During the first stage, the contextual block containing the object in the reference block is etermine with integral pixel accuracy. a {( x + 1),...,(F w B w x + 1)}, (2) b {( y + 1),...,(F h B h y + 1)}, C(RB(x,y,t),CB(a min,b min,u)) C(RB(x,y,t),CB(a,b,u)) is full search, 6 which calculates C(RB(x,y,t),CB(a,b,u)) for all values of a an b in the above mentione ranges, an keeps track of the pair (a,b) that has resulte in the lowest value of the cost function. This algorithm has complexity O(pqB w B h ) if p an q are specifie, an O(F w F h B w B h ) if the most general form is use. For general purpose vieo compression, numerous other algorithms have been evelope that can reuce the complexity of the search. The problem with the various alternatives to full search is that they o not guarantee that the correct best match will be returne. They will be useful to us only when there is almost no similarity between the various possible blocks within a given frame, minimizing the chance of incorrect motion estimation. When vieo compression is being performe an the motion vector is use in conjunction with a resiual coe, this is not a problem since the error will be compensate for. However, in our application we are searching

5 for the specific part of image space that represents the area of object space uner consieration. We therefore prefer to use a full search algorithm with limits on p an q if there are computational constraints, since this yiels a more effective strategy for our purpose. In aition, the complexity of context extraction 7 is significantly lower than that of the context integration stage of the algorithm, making it possible to use full search here without noticeably affecting the performance of the implementation Re-estimation With Subpixel Accuracy Objects o not move in integral pixel isplacements. As a result we wish to estimate the motion vector at a finer resolution. Therefore, the initial estimate as outline above is the first part of a two stage hierarchical search. We have evelope an analytical technique to estimate the motion of a block with sub-pixel accuracy. Its implementation provies the secon part. as illustrate in Figure 5. By moeling the cost function, MSE, as a separable function, we evelop a measure for what real value isplacements along each orthogonal axis will reuce the value of the mean absolute error. Let the function F(x, y) contain a representation of the reference frame with temporal inex t, that is x {1,...,F w }, y {1,...,F h } (3) F(x,y) = V S(x,y,t) F w B w an let the function G(x, y) contain a representation of the contextual block for which we are etermining the motion vector with sub-pixel accuracy. If RB(x 0,y 0,t) was the reference block for which the previous stage of the algorithm foun CB(a min,b min,u) to be the contextual block with the closest match in frame (t + u), then efine F h B h x y x min = x 0 + a min, x = x 0 + a min + B w 1 (4) y min = y 0 + b min, y = y 0 + b min + B h 1 an further note that the appropriate efinition of G(x,y) that is require is given by x {x min,...,x }, y {y min,...,y } (5) G(x,y) = V S(x,y,u) Figure 5. The secon stage etermines a sub-pixel isplacement of the contextual block such that there is an improve match with the reference area. Instea of (a min,b min ), which is the estimate that results from the first stage of the search, we seek to etermine the actual motion of the object, which can be represente by (a min + δa min,b min + δb min ). Therefore, at this stage the vector (δa min,δb min ), which we call the sub-pixel motion isplacement, must be foun. Lemma 2.1. δx can be calculate in O(log (Y B w B h )) space an O(B w B h ) time. Proof. This statement hols true upto the approximation that intensity values between two ajacent pixels vary linearly. Above, Y is the range of possible values of the function (or, equivalently the range of possible values that a pixel s color can assume). If we estimate the sub-pixel motion isplacement as (δx,δy), then the cost function is C(RB(x,y,t),CB(a+ δx,b + δy,u)). Thus if we use mean square error as our matching criterion, an we treat it as an inepenent function of δx an δy, we obtain the equation MSE(δx) = [F(x,y) G(x + δx,y)] 2 (6)

6 For the purpose of calculating F an G with non-integral parameters, we will use linear interpolation. In orer to perform such an estimation, the function must be known at the closest integral value parameters. Since we have values of the pixels ajacent to the reference block easily available, an since this is not true for the contextual block, we reformulate the above equation without loss of generality as MSE(δx) = By expaning Equation 7 we obtain [F(x + δx,y) G(x,y)] 2 (7) M SE(δx) = (8) [ F 2 (x + δx,y) + G 2 (x,y) 2F(x + δx,y)g(x,y) ] This is the equivalent of using a fixe gri (the contextual block) an variably isplacing the entire reference frame, in orer to fin the best sub-pixel isplacement, (instea of the intuitive fixing of the image frame an movement of the block whose motion we are etermining). Assuming that the optimal sub-pixel isplacement along the x axis of the contextual block uner consieration is the δx that minimizes the function MSE(δx) in Equation 8 which moels C(RB(x 1,y 1,t),CB(a+δx,b)), we procee to calculate MSE(δx) = (9) (δx) [ ] 2F(x + δx,y) F(x + δx,y) 2G(x,y) F(x + δx,y) (δx) (δx) Since we wish to etermine the minimum of MSE(δx), we will solve for δx using the constraint MSE(δx) = 0 (10) (δx) = [ [F(x + δx,y) G(x,y)] ] F(x + δx,y) (δx) Since F(x, y) is a iscrete function, we use linear interpolation to approximate it as a continuous function for the purpose of representing F(x + δx,y) an computing F(x + δx,y). We consier the case of a positive δx first. δx, 0 δx 1, Applying Equations 11 an 12 to Equation 10, we obtain { (δx) F(x + δx,y) = F(x,y) + δx [F(x + 1,y) F(x,y)] (11) F(x + δx,y) = F(x + 1,y) F(x,y) (δx) (12) [ [F(x + 1,y) F(x,y)] (13) [ F(x,y) G(x,y) + δx [F(x + 1,y) F(x,y)] ] ] } = 0 Further expaning, an grouping to separate terms with δx, { [ δx [F(x + 1,y) F(x,y)] 2 (14) + F(x + 1,y)F(x,y) F 2 (x,y) + F(x,y)G(x,y) F(x + 1,y)G(x,y) ] } = 0

7 Therefore, by rearranging terms an rewriting the above, we can obtain the close form solution δx = (15) x y [ [F(x + 1,y) F(x,y)] [F(x,y) G(x,y)] ] x y [F(x + 1,y) F(x,y)] 2 Similarly an inepenent calculation can be performe to attempt to ascertain what negative value of δx is optimal. The solution in the the case of a negative δx is a minor variation of the above an is outline below δx, 1 δx 0, Apply Equations 16 an 17 to Equation 10 to obtain { F(x + δx,y) = F(x,y) + δx [F(x,y) F(x 1,y)] (16) (F(x + δx,y)) = F(x,y) F(x 1,y) (δx) (17) After algebraic manipulation, we obtain δx = [ [F(x,y) F(x 1,y)] (18) [ F(x,y) G(x,y) + δx [F(x,y) F(x 1,y)] ] } = 0 x y [ [F(x,y) F(x 1,y)] [F(x,y) G(x,y)] ] x y [F(x,y) F(x 1,y)] 2 (19) Finally, we compute the MSE between the reference an contextual block, with each of the two δx s. The δx that results in the lower MSE is etermine to be the correct one. The close form solution for each δx as B w B h terms in both the numerator an the enominator. Each term requires two subtractions an one multiplication. This is followe by computing the final quotient of the numerator an enominator summations. The time complexity is therefore O(B w B h ). We note that computing the MSE is possible within this boun as well. Since the function values range upto Y, an only the running sum of the numerator an enominator nee to be store, the space neee is O(log Y B w B h ). Next, we note that it is possible to obtain a better representation of a block in the reference frame by completing the analysis of sub-pixel isplacements along the orthogonal axis. Lemma 2.2. The carinality of the set of points that represents the block uner consieration from the reference frame is non-ecreasing. Proof. After a δx has been etermine, the block must be translate by a quantity δy along the orthogonal axis. It is important to perform this calculation after the δx translation has been applie, since it guarantees that the MSE after both translations is no more than the MSE after the first translation, ensuring the correctness of the algorithm. We can etermine the δy in a manner analogous to the one use to etermine δx, using the representation of the MSE below, with the efinition x = x + δx: MSE(δy) = This is in turn equivalent to (by the same argument provie for the δx case): Solving for δy is achieve using: MSE(δy) = [F(x,y) G(x,y + δy)] 2 (20) [F(x,y + δy) G(x,y)] 2 (21) δy, 0 δy 1, δy = (22)

8 x y [ [F(x,y + 1) F(x,y)] [F(x,y) G(x,y)] ] x y [F(x,y + 1) F(x,y)] 2 δy, 1 δy 0, δy = (23) x y [ [F(x,y) F(x,y 1)] [F(x,y) G(x,y)] ] x y [F(x,y) F(x,y 1)] 2 If the sub-pixel isplacement (δx, δy) etermine results in an MSE between the contextual an reference blocks exceeing a given threshol, then the extra information is not incorporate into the current set representing the reference block. This prevents the contamination of the set with spurious information. It also completes the proof that either a set of points that enhances the current set is ae, or none are - since this yiels a non-ecreasing carinality for the set representing the block that is being processe. By performing this analysis inepenently for each of the contextual blocks that correspons to each of the reference blocks, we obtain a scattere ata set representing the frame whose resolution we are enhancing. If the sampling gri ha infinite resolution, an we inspecte the values registere on it at the points we have etermine above to be in the image space, we woul fin that these ata points are a goo approximation. By repeating this process with a number of contextual frames for each of the reference frames, we can extract a large set of extra ata points in the image space of the reference frame. At this stage, we have analytically enhance the resolution of the vieo frame. However, the format of scattere ata is not an acceptable format for most isplay meia such as vieo monitors. Therefore we must process this information further to make it usable Framework 3. CREATING A COHERENT FRAME We shall efine a uniform gri to be the set H P,k V Q,l of lines in R 2, where H P,k = {x = kα k {0,1,2,...,P }, α R, P N} specifies a set of vertical lines an V Q,l = {y = lβ l {0,1,2,...,Q}, β R, Q N} specifies a set of horizontal lines. Specifying the values of P,Q,α, an β etermines the uniform gri uniquely. Given a set S of points of the form (x,y,z), where z represents the intensity of the point (x,y) in image space, if there exist M,N,α, an β such that all the (x,y) of the points in the set S lie on the associate uniform gri, an if every ata point (x,y) on the uniform gri has an intensity (that is a z component) associate with it, then we call the set S a coherent frame. Each of the frames in the original vieo sequence was a coherent frame. We seek to create a coherent frame from the ata set D, that we have obtaine through the process escribe in Section 2, with the constraint that the number of points in the coherent frame shoul be the same as that of the ata set D. The general principle we use to effect the transformation from scattere ata to a coherent frame is to first construct a surface in R 3 that passes through the input ata. By representing this surface in a functional form with the x an y coorinates as parameters, it can then be evaluate at uniformly space points in the XY plane for the prouction of a coherent frame. There are numerous methos available for the purpose, the traeoff being the increase computational complexity neee to guarantee a greater level of accuracy of the mapping. While the efficiency of the proceure is important, in the light of our concerns in Section 1.3, we woul like to make our approximation as close a fit as possible. We note that although we will be force to use interpolation techniques at this stage, we are oing so with a ata set that has been increase in size as escribe in Section 2, so this is not equivalent to performing interpolation at the outset an is certainly an improvement over that. We use B-spline interpolation with egree k which can be set as a parameter to our algorithm Implementation Issues Given an arbitrary scattere ata set, we can construct a coherent frame that provies a very goo approximation of the surface specifie by the original ata set. However, if we were to work with the entire ata set at han, our algorithm woul not be scalable. This is ue to the fact that memory requirement woul excee that which is available on toay s computer systems. Noting that splines use only a fixe number of neighboring points, we employ the technique of ecomposing the ata set into spatially relate sets of fixe size. Each set contains all the points within a block in image

9 space. The isavantage of working with such subsets is that visible artifacts evelop at the bounaries in the image space of these blocks. To avoi this we compose the blocks by using ata from ajacent blocks to create a borer of ata points aroun the block in question, so that the spline surface constructe for a block is continuous with the surfaces of the ajacent blocks. Working with a block at a time in this manner, we construct surfaces for each region in the image space, an evaluate the surface on a uniform gri, to obtain the representation we esire. When this is one for the entire set, we have obtaine a coherent frame. Lemma 3.1. Creating a block in a coherent frame requires O(B w B h σ(k)t) operations using ata extracte from temporal context. Proof. Here T frames are incorporate, an egree k polynomial base B-spline tensor proucts are use to perform the transformation from scattere to grie ata. Lemma 2.2 guarantees that the input set oes not egenerate. The complexity of splining epens on the number of points, which is B w B h, the prouct of the block with an height. σ(k) is the cost to perform the splining operations per point in the ata set. Since B-splines have the property of local support, that is only a fixe number of ajacent B-splines are require for the evaluation of any given point in the space spanne by them (such as the surface being represente), an each B-spline can be represente as a fixe length vector of coefficients, the approximation of a surface specifie by a set of points has time complexity that is only boun by the egree of the polynomials use an the multiplicity of the knots. 8,9 The above result follows immeiately from this Time Complexity 4. RESULTS Theorem 4.1. The resolution of a vieo sequence can be enhance in O(n) time, where n is the size (in bits) of the raw vieo ata, if: (a) the egree of splines use in interpolation is fixe, (b) a constant number of frames of temporal context is use, an (c) the range of motion estimation is limite to a fixe multiple of the block size. Proof. We assume that there are L frames to process, in each of which there are FwF h B wb h blocks. For each block, it takes µ(b w,b h ) time to perform motion estimation (assuming fixe range exhaustive search) an re-estimation of the motion with sub-pixel accuracy, by Lemma 2.1. Transforming the ata set obtaine for a block takes O(B w B h σ(k)t) time, by Lemma 3.1. Therefore, processing an L frame vieo sequence using T frames of temporal context to enhance the resolution of each frame, yiels the higher resolution version in O(F w F h L[ µ(bw,b h) B wb h + σ(k)t]) time. By limiting the range of motion estimation to a fixe multiple of the block size, µ(b w,b h ) = O(1). Using a constant number of frames of temporal context results in T = O(1). Finally, while in theory B-spline interpolation has complexity O(k log 2 k), constructing a B-spline as well as evaluating it along with all its erivatives can be one in O(k 2 ) operations in practice. However, if k is fixe, then σ(k) = O(1). Since n = F w F h L an if the above specifie constraints hol, then enhancing the resolution of the vieo sequence has a time complexity boun of O(n) Application GROW The program GROW was evelope as an implementation of the algorithm. It provies a flexible comman line interface that allows the iniviual specification of numerous parameters such as the horizontal an vertical imensions of the blocks use for motion estimation, the blocks use for spline interpolation, the extra borer use for composing blocks for spline interpolation, the egrees of the splines use, the factors by which the output frame is scale up from the original, the imum range in which to perform motion estimation, the number of previous an later frames use to enhance any given frame, the error threshol that is acceptable uring motion estimation an that for spline interpolation. The parameters entere serve to guie but not enforce the algorithm. For example, splining starts with the egree the user enters but automatically rops to lower egrees (as less context is use) when the surface returne is not close enough to the points it is approximating. The error threshol specifie for splining is use to scale up bouns that are calculate using heuristics from the literature. 10 Intermeiate results, such as the sub-pixel isplacements, are calculate an kept in high precision floating point format. When error threshols

10 Figure 6. Original frames. are crosse, the ata in question is not use. Thus occlusion an peripheral loss of objects is ealt with by the effective result of using only reference image ata for the relevant region Experiments The methoology employe to test the program is specifie below. A vieo sequence was sub-sample, on a frame by frame basis. The new lower resolution sequence was use as ata, an the output compare to the original sequence. If G represents the high resolution vieo sequence, an F represents the high resolution sequence obtaine by running GROW on the low resolution version associate with G, where the low resolution version was obtaine through the sub-sampling of G, then the Signal to Noise Ratio is efine as: 1 x y B wb h F(x,y) SNR = 10log 1 x y (B wb h ) 2 [F(x,y) G(x,y)] 2 (24) Using this metric to measure istortion, we compare the strength of the signal of the sequence prouce by GROW uner various cicrumstances. As an example, note the parts of five smaller frames in Figure 4.2 that are part of the original vieo sequence, garen. By using all of them as input to GROW, the higher resolution output corresponing to the the thir original frame is generate an shown. Figure 4.3 shows versions constructe with GROW as well as with spline interpolation, along with the original frame. Artifacts that arise in the final images are ue to errors in initial block matching an splining. We note that the improvement over spline interpolation by the aition of ata obtaine from temporal context is one at little practical cost. This is emonstrate in the ata in Figure 7 by the fact that the amount of time spent on this is minimal compare to time spent on the spline construction an evaluation. The effect of varying the parameters of the algorithm is an observable as well as measurable change in the quality of the output images, as measure by the SNR heuristic. In Table 1, the first line represent the use of only spline interpolation without any proximal frames. The secon line uses 2 previous an 2 later frames. The SNR is noticeably higher. The thir line also uses a total of 5 frames, but the Figure 7. Time spent broken own by splining an temporal context extraction. Secons GROW SGF TCE Frame Count 6

11 Figure 8. The frame whose resolution is being increase. Figure 9. The same frame with its resolution increase using only spline interpolation. Figure 10. Higher resolution version of the central frame obtaine using GROW. motion estimation blocks are 4x4 instea of 8x8 as in the secon case or 16x16 in the first case. This creates a further improvement in the SNR (at the cost of increase computation). Only 4 parameters are shown - the range for initial motion estimation on each axis an the number of earlier an later frames use, respectively. However, GROW inclues 13 other parameters that can be ajuste. We escribe them in Appenix A. s t j k x y c p l m n a b z u e SNR Table 1. Effect of varying the parameters on the output frame s SNR As mentione in the overview, certain aspects such as spline egree selection have been automate, while others such as splining error bouns are semi-automatically calculate, using heuristics an minimal user input. To get an optimal image sequence, though, it is necessary for the user to manually ajust the values fe into the algorithm. Finally, we note that by han tuning GROW, we can obtain a significantly better result than possible by spline interpolation without using temporal context. This can be note from the table as the row which uses no past an future frames effects this kin of spline interpolation, but oes not yiel as strong a signal as the best case output of GROW. 5. RELATION TO PREVIOUS WORK Ientifying points in object space using image sequences has been ealt with extensively in the literature on computer vision, geometric moeling, image processing, an pattern recognition. Work has even been one on tracking objects with a moving camera. 11 To eal with the problems of iscerning rotation from translation when the motion is small, as well the sensitivity of the measurements of epth (to etermine object shape) to noise, factorization methos that analytically separate the motion from the object have been evelope for the cases where the motion is planar, 12 or in R 3, 13 as well as when the projection is either orthographic, 14,15 uner perspective 16 or the paraperspective case. 17 The analyses have been carrie out within a framework where either there exists a fixe moel of camera or object motion, or the stuy seeks to ientify the parameters 18 that constitute the mapping M, from object space to image space. Applying such methos woul involve ientifying M explicitly an computing M 1, or at the very least etermining M 1 empirically. This woul be neee in orer to capture representations in object space, that coul then be transforme to image space. By increasing the information available, a higher resolution frame coul then be constructe. However, we seek to evelop an efficient algorithm for vieo resolution enhancement,

12 by which we mean that we wish to obviate the nee for explicit mappings back to object space. Our algorithm will achieve the extraction of extra ata points in any given frame by working irectly with the frame s image space, an the associate temporal context, the past an future frames. Digital signal processing techniques target the problem of enhancing the resolution of the vieo sequence. 19 In contrast to other approaches, our work is explicitly parametrize so that cost an quality can be trae as neee. Further, we unertake an explicit analysis of the time complexity of the reconstruction process. Our approach oes not account for factors such as camera blur, occlusion or the aperture problem CONCLUSION This paper escribes how temporally proximal frames of a vieo sequence can be utilize to ai the forensic analyst by enhancing the resolution of iniviual frames. The framework evelope allows the analyst to vary a number of parameters so that a traeoff may be mae between the quality of the reconstructe frame an the time it takes to effect the process. The algorithm s time complexity is analyze as it is being evelope. Aitionally, we escribe our implementation of it as a C program that takes a vieo sequence as input an prouces a higher resolution version as the output. Finally, we escribe an example of its application, where varying the parameters results in an increase signal-to-noise ratio in the output as compare to the input. REFERENCES 1. V. Bhaskaran an K. Konstantinies, Image an Vieo Compression Stanars: Algorithms an Architectures, Kulwer Acaemic Publishers, Boston, MA, K. Challapali, X. Lebegue, J. Lim, W. Paik, R. S. Girons, E. Petajan, V. Sathe, P. Snopko, an J. Zepski, The gran alliance for us htv, Proceeings of the IEEE, D. L. Gall, Mpeg: A vieo compression stanar for multimeia applications, Communications of the ACM, L. Chiariglione, The evelopment of an integrate auiovisual coing stanar: Mpeg, Proceeings of the IEEE, T. Markas, Data Compression: Algorithms an Architectures. PhD thesis, Duke University, K.-W. Wang, R.-L. Wang, an Y.-X. Yan, Optoelectronic hybri system for the full-search block-matching motion compensation algorithm. 7. P. Pirsch, N. Demassieux, an W. Gehrke, Vlsi architectures for vieo compression - a survey, Proceeings of the IEEE, C. e Boor, A Practical Guie to Splines, Springer-Verlag, New York NY, L. Schumaker, Spline Functions Basic Theory, John Wiley an Sons, New York, NY, P. Dierckx, Curve an Surface Fitting with Splines, Oxfor University Press, New York, NY, Burt, Bergen, Hingorani, Kolczynski, Lee, Leung, Lubin, an Shvayster, Object tracking with a moving camera - an application of ynamic motion analysis, Proceeings of the IEEE, C. Tomasi an T. Kanae, Shape an motion from image streams: A factorization metho - planar motion, tech. rep., Carnegie Mellon University, C. Tomasi an T. Kanae, Shape an motion from image streams: A factorization metho - point features in 3 motion, tech. rep., Carnegie Mellon University, C. Tomasi an T. Kanae, Shape an motion from image streams: A factorization metho - full report on the orthographic case, tech. rep., Carnegie Mellon University, C. Tomasi an T. Kanae, Shape an motion from image streams uner orthography: A factorization metho, International Journal of Computer Vision, C. Tomasi, Pictures an trails: A new framework for the computation of shape an motion from perspecitve image sequences, tech. rep., Cornell University, C. Poelman an T. Kanae, A paraperspective factorization metho for shape an motion recovery. 18. T.-H. Wu an R. Chellapa, Experiments on estimating motion an structure parameters using long monocular image sequences, tech. rep., University of Marylan, College Park, N. R. Shah an A. Zakhor, Resolution enhancement of color vieo sequences, IEEE Transactions on Image Processing 8(6), pp , S. Borman, Topics in Multi-frame Super-resolution Restoration. PhD thesis, University of Notre Dame, 2004.

13 APPENDIX A. GROW PARAMETERS Usage: grow -f <first-frame> -g <last-frame> -h <frame-height> -w <frame-with> -i <input-files> -o <output-files> [-s <x-imension-of-splining-blocks>] [-t <y-im-spl-blks>] [-j <x-im-extra-pixels-for-splining>] [-k <y-im-extra-pixels>] [-x <x-im-of-motion-estimation-blocks>] [-y <y-im-mot-est-blks>] [-c <horizontal-scaling-factor>] [- <vertical-scaling-factor>] [-p <number-of-previous-frames-use-as-context>] [-l <num-later-frms>] [-m <imum-x-isplacement-uring-motion-est>] [-n <-y-isp>] [-a <x-imension-spline-egree>] [-b <y-imension-spline-egree>] [-z <spline-acceptable-error-factor>] [-u <class-of--accept-err>] [-e <imum-error-uring-motion-est>] [-v (verbose)] -f The inex number of the first frame where processing of the vieo sequence is to begin. For example, if the entire sequence of files frame1.y,...,frame20.y is to be processe then the value use as a parameter of -f woul be 1. -g The inex number of the last frame to be processe. In the above example, the parameter of -g woul be 20. -h The integer number of pixels in the vertical irection of each iniviual frame in the vieo sequence. -w The integer number of pixels in the horizontal irection of each iniviual frame in the vieo sequence. -i The string name of the input sequence of files to be use. For example, to process the sequence frame1.y,..., frame20.y, the parameter frame woul be use for the -i option. -o The string name of the output vieo sequence. For example, if the parameter newframe was use for the -o option, then if there are 5 output frames, they will be store as newframe0.y,...,newframe4.y. The options above must be specifie by the user. The ones below have efaults. -s The integer parameter supplie is the horizontal imension of the block size use uring spline interpolation an evaluation. -t The integer parameter is the vertical imension of the block size use uring spline interpolation an evaluation. -j During spline interpolation of a block, pixels from ajacent blocks are use to sew the blocks together. The integer parameter supplie is the horizontal with of the borer that runs parallel to the vertical axis. -k The integer parameter supplie is the vertical with of the above borer that runs parallel to the horizontal axis. -x The integer parameter supplie is the horizontal imension of the block size use uring motion estimation. -y The integer parameter is the vertical imension of the block size use uring motion estimation.

14 -c The integer parameter supplie specifies the factor by which the horizontal resolution is increase when evaluating the spline surface representing the image. - The integer parameter supplie specifies the factor by which the vertical resolution is increase when evaluating the spline surface. -p The integer parameter is the number of previous frames that are use uring temporal context extraction to a information to the ata set representing the frame that is currently being processe. -l The integer parameter is the number of later frames that are use to a information for the frame that is currently being processe. -m The integer parameter supplie is the imum horizontal irection in which to search for a motion vector. If this is set to 0, then a full search throughout the entire image is performe. -n The integer parameter supplie is the imum vertical irection in which to search for a motion vector. If this is set to 0, then a full search throughout the entire image is performe. -a The integer parameter is the imum egree of the polynomials use in the horizontal irection. If this yiels unacceptable results, the application will automatically try lower egree polynomials as well. -b The integer parameter is the imum egree of the polynomials use in the vertical irection. If this yiels unacceptable results, the application will automatically try lower egree polynomials as well. -z During spline interpolation the smoothing norm is subject to a least mean squares constraint. The floating point parameter supplie here specifies that value. -u The spline routines from SURFIT provie a return value inicating what kin of approximation was performe. The integer parameter supplie here specifies the level of approximation consiere acceptable. This shoul usually be left at 0. -e During motion estimation, if the MAE between the reference an currently consiere blocks is below the integer parameter supplie here, then an acceptable match is consiere to be foun. -v When this option is use the application sens etaile information about what it is currently oing to the stanar output.