PAPER Reduction of Computational Cost of POC-Based Methods for Displacement Estimation in Old Film Sequences

Transcription

1 IEICE TRANS. FUNDAMENTALS, VOL.E94 A, NO.7 JULY PAPER Reduction of Computational Cost of POC-Based Methods for Displacement Estimation in Old Film Sequences Xiaoyong ZHANG a), Nonmember, Masahide ABE b), and Masayuki KAWAMATA c), Members SUMMARY This paper proposes a new method that reduces the computational cost of the phase-only correlation (POC)-based methods for displacement estimation in old film sequences. Conventional POC-based methods calculate all the points of the POC and only use the highest peak of the POC and its neighboring points to estimate the displacement with subpixel accuracy. Our proposed method reduces the computational cost by calculating the POC in a small region, instead of all the points of the POC. The proposed method combines a displacement pre-estimation with a modified inverse discrete Fourier transform (IDFT). The displacement pre-estimation uses the 1-D POCs of frame projections to pre-estimate the displacement with pixel accuracy and chooses a small region in the POC including the desired points for displacement estimation. The modified IDFT is then used to calculate the points in this small region for displacement estimation. Experimental results show that use of the proposed method can effectively reduce the computational cost of the POC-based methods without compromising the accuracy. key words: old film restoration, subpixel displacement, phase-only correlation, DFT pruning 1. Introduction The twentieth century is the century of the moving images. Since invention of cinema at the end of the nineteenth century, it has developed as one of the most important manifestations of popular culture and mass media [1]. A huge number of films have been produced which are valuable records of history and culture. However, many old films have been degraded due to poor storage conditions and chemical instability of film materials. In recent years, there is a growing awareness of the urgency to restore these old films before they are lost. In general, there are several typical defects in old film sequences: frame displacement, intensity flicker, blotches, scratches, etc [2]. These defects not only result in the loss of the original information but also are annoying to watch. In addition, the compression standard such as MPEG is poorly suited to coding old film sequences suffering from these defects [3]. Digital restoration is an efficient approach to correct or remove these defects and presents the restored films in a digital format. In this paper, the focus is on the frame displacement Manuscript received October 13, Manuscript revised March 4, The authors are with the Department of Electronic Engineering, Graduate School of Engineering, Tohoku University, Sendaishi, Japan. a) zhang@mk.ecei.tohoku.ac.jp b) masahide@mk.ecei.tohoku.ac.jp c) kawamata@mk.ecei.tohoku.ac.jp DOI: /transfun.E94.A.1497 estimation, which is an essential step for correcting the displacement in old film sequences. Frame displacement is defined as the unwanted global motion of the interframe in the vertical and horizontal directions. It arises from the deterioration of the film perforations and the inaccurate film transporting system in the film camera [2]. In order to correct frame displacement in old film sequences, the displacement value between two consecutive frames should be estimated in advance. In addition, displacement correction also greatly affects subsequent restoration processes, such as flicker correction [4] and blotch removal [5]. Phase-only correlation (POC) is a powerful method used in image registration due to its robust performance and computational simplicity. The displacement between two images can be estimated with pixel accuracy by finding the location of the highest peak of the POC [6]. POCbased methods can also be applied to rotation and scaling estimation via the log-polar transform [7]. In recent years, POC-based methods have been extended to subpixel accuracy [8] [11]. These methods can be classified as two categories: non-interpolation methods and interpolation methods. Some typical non-interpolation methods are introduced in [8] [10]. In Ref. [8], a closed-form solution is provided by modeling the subpixel displacement as the result of image down-sampling. The subpixel displacement is then estimated using the highest peak of the POC and its two neighboring points. This method is extended in Ref. [9], the highest peak of the POC and its four neighboring points being used to estimate the subpixel displacement under noisy conditions. In Ref. [10], the height difference between two secondary side-peaks of the POC is taken into account to improve the accuracy of POC-based methods. Though good results have been reported, these non-interpolation methods are generally sensitive to noise. On the other hand, a high-accuracy interpolation method is proposed in [11] to estimate the displacement in old film sequences. This method uses the inverse discrete Fourier transform (IDFT) of a zero-padded normalized cross power spectrum to obtain an interpolated POC. The displacement is then estimated by locating the highest peak of a quadratic curve fitted to the highest peak of the interpolated POC and its neighboring points. Simulation results showed that this interpolation method could achieve outstanding accuracy in displacement estimation under noisy conditions. However, since the normalized cross power spectrum is extended to a spectrum of larger size due to zero padding, the Copyright c 2011 The Institute of Electronics, Information and Communication Engineers

2 1498 computational cost of the IDFT is greatly increased. It should be noted that the IDFT in the interpolation method [11] is inherently inefficient becausetheinput of the IDFT has a large number of zeros while the number of the output of the IDFT required for displacement estimation is small. A reasonable strategy for reducing the computational cost is to eliminate or considerably reduce the computations on the padded zeros and those for the undesired output of IDFT. In this paper, the interpolation method [11] is extended and a new method to reduce the computational cost of this method is proposed. This paper is organized as follows: Sect. 2 briefly introduces the POC-based methods for frame displacement estimation. Section 3 presents our proposed method in detail. In Sect. 4, the computational cost of the proposed method is discussed in comparison with the interpolation method [11]. Section 5 presents some experimental results to evaluate the performance of the proposed method. 2. POC-Based Methods for Displacement Estimation Let I n (t 1, t 2 ) and I n 1 (t 1, t 2 ) be two two-dimensional (2- D) continuous-space signals representing two consecutive frames in a film sequence, where t 1 and t 2 are continuous real numbers, and the subscripts n and n 1 denote the frame indices of two consecutive frames. I n (n 1, n 2 )andi n 1 (n 1, n 2 ) are two discrete-space signals obtained by sampling I n (t 1, t 2 ) and I n 1 (t 1, t 2 ) as follows: I i=n,n 1 (n 1 T 1, n 2 T 2 ) = I i=n,n 1 (t 1, t 2 ) t1 =n 1 T 1,t 2 =n 2 T 2 (1) where T 1 and T 2 are sampling periods along the vertical and horizontal directions, respectively. For simplicity, we set T 1 = T 2 = 1 to obtain the discrete signals I n (n 1, n 2 ) and I n 1 (n 1, n 2 ). The phase-only correlation (POC) of two frames, denoted by g(n 1, n 2 ), is defined as the inverse Fourier transform of the normalized cross power spectrum of two frames [6]. In practice, we use the DFT to calculate the POC of two frames as follows: g(n 1, n 2 ) = IDFT [G(k 1, k 2 )] = IDFT F n(k 1, k 2 ) Fn 1 (k 1, k 2 ) F n (k 1, k 2 ) Fn 1 (k 1, k 2 ) (2) where G(k 1, k 2 ) denotes the normalized cross power spectrum, F n (k 1, k 2 ) and F n 1 (k 1, k 2 ) are the 2-D DFTs of I n (n 1, n 2 )andi n 1 (n 1, n 2 ), respectively, k 1 = 0, 1,, N 1 1 and k 2 = 0, 1,, N 2 1 are the discrete frequency indices in the frequency domain, and the asterisk denotes the complex conjugate. The POC can be used for estimating the displacement between two frames. Suppose that I n (t 1, t 2 ) is a shifted replica of I n 1 (t 1, t 2 ), i.e., I n (t 1, t 2 ) = I n 1 (t 1 δ 1, t 2 δ 2 ), where δ 1 and δ 2 are real numbers. If δ 1 = τ 1 T 1 and δ 2 = τ 2 T 2 and τ 1 and τ 2 are integer numbers, the POC of two frames is a shifted delta function given by g(n 1, n 2 ) = IDFT F n(k 1, k 2 ) Fn 1 (k 1, k 2 ) F n (k 1, k 2 ) Fn 1 (k 1, k 2 ) IEICE TRANS. FUNDAMENTALS, VOL.E94 A, NO.7 JULY 2011 = IDFT [ e ] j2π(k 1τ 1 /N 1 +k 2 τ 2 /N 2 ) 1, (n 1, n 2 ) = ( τ 1, τ 2 ) = 0, otherwise where the location of the peak identifies the values of τ 1 and τ 2. Figures 1(a) and 1(b) show an example of a POC of two frames when (τ 1,τ 2 ) = (1, 1). The displacement between the two frames can be estimated with pixel accuracy by finding the location of the peak in the POC. Although the digital frames are represented by pixels, subpixel displacement between frames occurs when analog signals are sampled. If the amounts of the displacement are subpixel values, the POC is not a delta function. Figures 1(c) and 1(d) show the POC of two frames when (τ 1,τ 2 ) = (0.3, 0.3). The peak coordinates are (0, 0), even though the actual displacement values are (0.3, 0.3). When τ 1 and τ 2 are decimals, the POC of two frames in Eq. (2) are derived as follows: g(n 1, n 2 ) = IDFT [ e ] j2π(k 1τ 1 /N 1 +k 2 τ 2 /N 2 ) = 1 sin(π(n 1 +τ 1 )) N 1 N 2 sin ( π N 1 (n 1 +τ 1 ) ) sin(π(n 2+τ 2 )) sin ( π N 2 (n 2 +τ 2 ) ) (4) under the condition 2π k i τ i = 2π (N i k i )τ i, i = 1, 2 (5) N i N i since I n and I n 1 are real numbers. When π N 1 (n 1 + τ 1 ) and π N 2 (n 2 + τ 2 ) are very small, g(n 1, n 2 ) can be approximated as a sampled 2-D sinc function given by g(n 1, n 2 ) sin(π(n 1 + τ 1 )) sin(π(n 2 + τ 2 )) π(n 1 + τ 1 ) π(n 2 + τ 2 ) = sinc(n 1 + τ 1 )sinc(n 2 + τ 2 ) (6) where sinc(x) isdefinedas sinc(x) = sin(πx). (7) πx Figure 1(d) shows sinc(n 1 + τ 1 )whenτ 1 = 0.3. The displacement between two frames can be estimated with subpixel accuracy by finding the location of the highest peak in the approximated sinc function. 3. Proposed Method for Reducing the Computational Cost of POC-Based Methods As mentioned in Sect. 1, the interpolation method [11] uses zero padding in the frequency domain to obtain an interpolated POC in the spacial domain. This method introduces a heavy computational cost into the IDFT. In this section, we present the proposed method for reducing the computational cost of the interpolation method [11]. The proposed method Making artificial subpixel displacement between two frames is introduced in Sect. 5. (3)

3 ZHANG et al.: REDUCTION OF COMPUTATIONAL COST OF POC-BASED METHODS FOR DISPLACEMENT ESTIMATION IN OLD FILM SEQUENCES 1499 Fig. 1 Examples of the POC of two frames. (a) The POC of two frames when (τ 1,τ 2 ) = (1, 1). (b) The peak of the POC and its neighboring points along the n 1 axis in (a). (c) The POC of two frames when (τ 1,τ 2 ) = (0.3, 0.3). (d) The peak of the POC and its neighboring points along the n 1 axis in (c), these discrete points can be approximated as a sampled sinc function sinc(x + τ 1 ) plotted as the dashed curve. consists of three steps as follows. First, we use two slices in the normalized cross power spectrum to pre-estimate the displacement with pixel accuracy. This pre-estimated result allows us to determine a small region in which the highest peak of the interpolated POC and its neighboring points are included. Then, we modify the standard IDFT to selectively calculate the points in the small region. Finally, we adopt a quadratic curve fitting method to estimate the displacement with subpixel accuracy. Each step is described in the following subsections. 3.1 Displacement Pre-Estimation with Pixel Accuracy We extract two slices from G(k 1, k 2 ) in Eq. (2) along the k 1 -axis and k 2 -axis to obtain two 1-D signals, denoted by G h (k 1, 0) and G v (0, k 2 ). G h (k 1, 0) and G v (0, k 2 ) can also be expressed as follows: G h (k 1, 0) = F n(k 1, 0) Fn 1 (k 1, 0) F n (k 1, 0) Fn 1 (k 1, 0) (8) = e j2πk 1τ 1 /N 1 G v (0, k 2 ) = F n(0, k 2 ) Fn 1 (0, k 2) F n (0, k 2 ) Fn 1 (0, k 2) (9) = e j2πk 2τ 2 /N 2 where F n (k 1, 0) and F n 1 (k 1, 0) are the slices through F n (k 1, k 2 )andf n 1 (k 1, k 2 ) along the k 1 -axis, respectively, and F n (0, k 2 )andf n 1 (0, k 2 ) are the slices through F n (k 1, k 2 ) and F n 1 (k 1, k 2 ) along the k 2 -axis, respectively. According to the definition of the 2-D DFT, F n (k 1, 0) and F n 1 (k 1, 0) can be obtained by replacing k 2 with0in F n (k 1, k 2 )andf n 1 (k 1, k 2 ) as follows: F n (k 1, 0) = F n 1 (k 1, 0) = N 1 1 n 1 =0 N 2 1 I n (n 1, n 2 ) e j2πk 1n 1 /N 1 (10) N 1 1 n 1 =0 n 2 =0 N 2 1 I n 1 (n 1, n 2 ) e j2πk 1n 1 /N 1 (11) n 2 =0 where the terms in the square brackets are the projections of I n (n 1, n 2 )andi n 1 (n 1, n 2 ) along the horizontal directions (n 2 - axis). Therefore, F n (k 1, 0) and F n 1 (k 1, 0) are the 1-D DFTs of the horizontal projections of I n (n 1, n 2 )andi n 1 (n 1, n 2 ), respectively. Accordingly, G h (k 1, 0) is the 1-D normalized cross power spectrum of the horizontal projections of two frames. We can use the 1-D IDFT of G h (k 1, 0) to obtain the 1-D POC of the horizontal projections of two frames, denoted by g h (n 1 ), given by g h (n 1 ) = IDFT [G h (k 1, 0)] = IDFT [ ] e j2πk 1τ 1 /N 1. (12) Similarly, the 1-D POC of the vertical projections of two frames, denoted by g v (n 2 ), can be obtained from the 1- DIDFTofG v (0, k 2 ), given by

4 1500 IEICE TRANS. FUNDAMENTALS, VOL.E94 A, NO.7 JULY 2011 Fig. 2 Zero padding in the normalized cross power spectrum for POC interpolation. (a) the normalized cross power spectrum of size N 1 N 2. (b) the zero padded normalized cross power spectrum, denoted by G (k 1, k 2 ), of size MN 1 MN 2. (c) a 3M 3M region in the interpolated POC. Under the guidance of (p 1, p 2 ), this small region includes the highest peak of the interpolated POC and its neighboring points. g v (n 2 ) = IDFT [G v (0, k 2 )] = IDFT [ ] e j2πk 2τ 2 /N 2. (13) Under the condition of Eq. (5), g h (n 1 )andg v (n 2 ) can be approximated as two sample 1-D sinc functions as follows: g h (n 1 ) sinc(n 1 + τ 1 ) (14) g v (n 2 ) sinc(n 2 + τ 2 ) (15) Then, τ 1 and τ 2 can be estimated by finding the locations of the highest peaks in Eq. (14) and Eq. (15), respectively. The subpixel displacement estimation using the 1-D POCs of frame projections can be found in [12], [13]. Since projecting the 2-D frame into 1-D frame projections leads to information loss, the accuracy of this 1-D POCs-based method is inevitably deteriorated. However, experimental results in Ref. [13] show that the displacement estimation using 1-D POCs of frame projections can achieve at least one-pixel accuracy. Therefore, the peak coordinates of the 2-D POC, denoted by (p 1, p 2 ), can be pre-estimated as follows: p 1 = arg max g h (n 1 ) (16) n 1 p 2 = arg max n 2 g v (n 2 ). (17) 3.2 Modified IDFT for Calculating a Small Region in the Interpolated POC In this subsection, we modify the standard IDFT to selectively calculate the interpolated POC in a small region for reducing the computational cost of the interpolation method [11]. Consider that if an N 1 N 2 -point POC needs to be enlarged to an interpolated POC of size MN 1 MN 2,the normalized cross power spectrum of size N 1 N 2 should be padded with zeros to size MN 1 MN 2 before application of the IDFT. Figure 2(a) shows the N 1 N 2 -point normalized cross power spectrum, and Fig. 2(b) shows the zero padded normalized cross power spectrum, denoted by G (k 1, k 2 )ofsizemn 1 MN 2, where the squares inside the dashed line represent the padded zeros. The interpolated POC, denoted by g (n 1, n 2 ), is obtained from the standard IDFT of G (k 1, k 2 )givenby g (n 1, n 2 ) = IDFT [ G (k 1, k 2 ) ] = 1 N 1 N 2 MN 1 1 k 1 =0 MN 2 1 k 2 =0 G (k 1, k 2 )W k 1n 1 MN 1 W k 2n 2 MN 2 (18) where W MNi = e j2π/(mn i), i = 1, 2. By interpolation, the 3 3-point region including the highest peak of the POC and its neighboring points is enlarged M times to a 3M 3M-point region in the interpolated POC. Under the guidance of the pre-estimated results p 1 and p 2, we can predetermine the location of this small region in the interpolated POC as follows: C(m 1, m 2 ) = g (Mp 1 + m 1, Mp 2 + m 2 ) (19) where m 1, m 2 [ 2M + 1, 2M + 2,, M 1, M]. Figure 2(c) shows a small region where the circles inside the dashed line represent the points of C(m 1, m 2 ). According to the DFT pruning techniques [14] [16], when the input of IDFT has a large number of zeros and the number of output of IDFT to be computed is small, the computational cost of the IDFT can be significantly reduced by DFT pruning. Motivated by the DFT pruning techniques, the standard IDFT in Eq. (18) is modified to calculate C(m 1, m 2 ) as follows: C(m 1, m 2 ) = IDFT MOD [G(k 1, k 2 )] N N 2 1 = G(k 1, k 2 )W k 1(Mp 1 +m 1 ) MN N 1 N 1 W k 2(Mp 2 +m 2 ) MN 2 2 k 1 =0 k 2 =0 (20)

5 ZHANG et al.: REDUCTION OF COMPUTATIONAL COST OF POC-BASED METHODS FOR DISPLACEMENT ESTIMATION IN OLD FILM SEQUENCES 1501 where IDFT MOD stands for the modified IDFT. Comparing Eq. (20) with Eq. (18), the modified IDFT uses N 1 N 2 -point G(k 1, k 2 ) to calculate the interpolated POC limited in a 3M 3M region. This modified IDFT eliminates the computations on the zeros in G (k 1, k 2 )aswellas those for the points outside the small region C(m 1, m 2 ). 3.3 Displacement Estimation with Subpixel Accuracy Finally, the quadratic curve fitting method [11] is adopted to estimate the subpixel displacement values using C(m 1, m 2 ). The basic idea of this method is that a quadratic curve can be approximately fitted to the highest peak of a sinc function and its neighboring points. Therefore, the displacement values can be estimated by locating the highest peak of this fitted quadratic curve. Details of this method can be found in [11]. 4. Computational Cost Analysis In this section, we analyze the computational cost of the proposed method compared with that of the interpolation method [11]. Figure 3 shows the computation process of our proposed method. In this figure, I n 1 and I n are weighted by two 2-D Hann windows prior to the 2-D DFTs for preventing the edge effects in the 2-D DFTs. Consider that the observed frames are of size N N and the interpolated POC is of size MN MN. As N and MN are assumed to be powers of 2, all the DFTs except the modified IDFT are implemented by the fast Fourier transforms (FFT). Table 1 summarizes the numbers of the required arithmetic operations (additions, multiplications, divisions and square roots) in the interpolation method [11] and the proposed method. From this table, it can be seen that the proposed method can achieve a larger reduction of arithmetic operations through the modified IDFT as opposed to the 2-D IFFT, though the 1-D IFFTs for pre-estimating the displacement with pixel accuracy lead to more arithmetic operations. Figure 4 shows the numbers of multiplications as functions of N when M is 4. This figure shows that the reduction of the computational cost achieved by the proposed method becomes more significant as the size N becomes larger. To illustrate the reduction of computational cost involved, Table 2 gives the theoretical and relative numbers of the arithmetic operations of two methods when N = 1024 and M = 4. Table 1 Theoretical numbers of arithmetic operations (additions, multiplications, divisions and square roots) in the interpolation method [11] and the proposed method. Interpolation method [11] Proposed method Windows Mul 2N 2 2N 2 2-D FFTs 2-D NCPS 1-D IFFTs Mul 2N 2 log 2 N 2N 2 log 2 N Add 4N 2 log 2 N 4N 2 log 2 N Mul 2N 2 2N 2 Div N 2 N 2 Sqrt N 2 N 2 Mul 0 2Nlog 2 N Add 0 4Nlog 2 N 2-D IFFT or Mul M 2 N 2 log 2 MN 9M 2 N 2 modified IDFT Add 2M 2 N 2 log 2 MN 9M 2 (N 2 1) Mul: Multiplication, Add: Addition, Div: Division, Sqrt: Square root. Fig. 4 The numbers of multiplications as function of N when M = 4. Table 2 The theoretical and relative numbers of arithmetic operations for the interpolation methods [11] and the proposed method when N = 1024 and M = 4. Fig. 3 Computation process of the proposed method. NCPS denotes the normalized cross power spectrum. Add Mul Div Sqrt Interpolation method [11] Proposed method 444, 596, , 978, 800 (100%) (43.4%) 226, 492, , 278, 400 (100%) (73.5%) 1, 048, 576 1, 048, 576 (100%) (100%) 1, 048, 576 1, 048, 576 (100%) (100%)

6 1502 IEICE TRANS. FUNDAMENTALS, VOL.E94 A, NO.7 JULY Experiments In this section, some experiments are presented to evaluate the performance of the proposed method in terms of accuracy and computational time. The experimental data consists of five artificially degraded image sequences and an actual old film sequence. Experiments were performed on an Intel Xeon 3.4 GHz computer with Linux OS using MAT- LAB. In the MATLAB programs, all the DFTs except the modified IDFT were performed by the MATLAB build-in FFT function. 5.1 Artificially Degraded Image Sequences Artificially degraded image sequences are generated from five original images (Lena, Baboon, Sailboat, Man, Pentagon). Each sequence consists of nine frames with subpixel displacement values from 0.1 to 0.9 pixel. Since the old film sequences are generally degraded by some of the defects mentioned in Sect. 1, generating the displaced image sequences should take these defects into account. The degraded image sequences are obtained by the following methods: 1. Subpixel displacement is made as follows: The original images are magnified 10 times by zero padding in the frequency domain, and the magnified images are displaced by integer values from 1 to 9. The subpixel displaced images are then obtained by downsampling the pixel-displaced magnified images. 2. The intensity flicker is added based on a linear flicker model. Details of flicker generation and the parameters setting can be found in Ref. [4]. 3. The blotches are added by the method in Ref. [17]. Figure 5 shows the artificially degraded sequences used for displacement estimation. In order to show that the interpolation method [11] achieves a higher accuracy than the non-interpolation method [8], the non-interpolation method [8] was also performed in this experiment. Also, the normalized cross power spectrum in the interpolation method [11] and in the proposed method was enlarged 4 times by zero padding, i.e., M = 4 in Eq. (18) and Eq. (20). Here, the root mean square error (RMSE) was employed as the criterion to evaluate the accuracy of the displacement estimation. Table 3 summarizes the RMSEs of three methods where Δτ 1 and Δτ 2 represent the RMSEs of the vertical and horizontal displacement, respectively. This table shows that the accuracy of the interpolation method [11] is higher than that of the non-interpolation method [11]. Meanwhile, the proposed method is found to achieve the same accuracy as the interpolation method [11]. In addition, Table 4 gives the average computational time [sec/frame] of the interpolation method [11] and that of the proposed method when image resolutions are and This table also gives the relative computational time of the two methods. Comparing Table 4 with Table 1, the relative computational time of the proposed method is lower than the relative numbers of arithmetic operations. 5.2 Actual Old Film Sequences Experiments were also performed on actual old film se- Table 3 Errors[pixel] in displacement estimation. Non-interpolation Interpolation Proposed method [8] method [11] method SEq. Δτ 1 Δτ 2 Δτ 1 Δτ 2 Δτ 1 Δτ 2 Lena Baboon Sailboat Man Pentagon Table 4 Average computational time [sec/frame] in displacement estimation. Image resolution Interpolation method [11] Proposed method (100%) (9.1%) (100%) (7.4%) Fig. 5 Artificial degraded image sequences: (a) Lena ( ), (b) Baboon ( ), (c) Sailboat ( ), (d) Man ( ), and (e) Pentagon ( )

7 ZHANG et al.: REDUCTION OF COMPUTATIONAL COST OF POC-BASED METHODS FOR DISPLACEMENT ESTIMATION IN OLD FILM SEQUENCES 1503 sented that reduces the computational cost of an interpolation method used for displacement estimation in old film sequences. The contribution of the proposed method is reduction of the computational cost of the interpolation method without compromising its accuracy. Experimental results showed the efficiency of the proposed method. In future work, the proposed method will be extended to estimating the rotation and scaling in image sequences. References Fig. 7 Fig. 6 A frame in the old film sequences Sendai. Comparison of estimated results in old film sequences Sendai. quences from Sendai. We choose 100 frames of size pixels in a shot as the experimental data. Figure 6 shows one frame of this old film sequence. It is difficult to evaluate the accuracy of the displacement estimation on an actual old film sequence since the real displacement values are unknown. Hence, the estimated results obtained from the interpolation method [11] and the proposed method are directly given. In this experiment, the normalized cross power spectrum in the interpolation method [11] and that in the proposed method were also enlarged 4 times by the zero padding. Figure 7 presents the estimated displacement values obtained from these two methods. From this figure, it can be seen that our results are identical to the results obtained from the interpolation method [11]. On the other hand, the average computational time of the proposed method for estimating the displacement between two frames is about 1.15 seconds compared with the seconds of the interpolation method [11]. That is to say, the computational cost of our proposed method is about 5.3% of the interpolation method [11]. 6. Conclusions As an extension of Ref. [11], a new method was herein pre- The original film is provided courtesy of the Sendai City Museum. [1] P. Read and M. Meyer, Restoration of motion picture film, Butterworth-Heinemann, [2] A. Kokaram, Motion picture restoration: digital algorithms for artefact suppression in degraded motion picture film and video, Springer-Verlag London, UK, [3] P. Richardson and D. Suter, Restoration of historic film for digital compression: A case study, IEEE International Conference on Image Processing, pp.49 52, Oct [4] R. Kawamata, M. Abe, and M. Kawamata, Fast flicker removal using m-esimate with reference images created in consideration of the effects of flickers and blotches in film sequences, J. Signal Process., vol.14, no.1, pp.61 72, Jan [5] S.Nam,M.Abe,andM.Kawamata, Fastandefficient MRF-based detection algorithm of missing data in degraded image sequences, IEICE Trans. Fundamentals, vol.e91-a, no.8, pp , Aug [6] C. Kuglin and D. Hines, The phase correlation image alignment method, IEEE Conference on Cybernetics and Society, pp , Sept [7] G. Wolberg and S. Zokai, Robust image registration using log-polar transform, Proc. IEEE International Conference on Image Processing, pp , [8] H. Foroosh, J. Zerubia, and M. Berthod, Extension of phase correlation to subpixel registration, IEEE Trans. Image Process., vol.11, no.3, pp , March [9] L. Chen and K.H. Yap, An effective technique for subpixel image registration under noisy conditions, IEEE Trans. Syst. Man. Cybern. A, Syst. Humans, vol.38, no.4, pp , July [10] J. Ren, J. Jiang, and T. Vlachos, High-accuracy sub-pixel motion estimation from noisy images in Fourier domain, IEEE Trans. Image Process., vol.19, no.5, pp , May [11] M. Hagiwara, M. Abe, and M. Kawamata, Estimation method of frame displacement for old films using phase-only correlation, J. Signal Process., vol.8, no.5, pp , Sept [12] J. Ren, T. Vlachos, and J. Jiang, Subspace extension to phase correlation approach for fast image registration, IEEE International Conference on Image Processing, pp , Oct [13] X. Zhang, M. Abe, and M. Kawamata, An efficient subpixel image registration based on the phase-only correlations of image projections, 2010 International Symposium on Communications and Information Technologies, pp , Oct [14] H. Sorensen and C. Burrus, Efficient computation of the DFT with only a subset of input or output points, IEEE Trans. Signal Process., vol.41, no.3, pp , March [15] J. Markel, FFT pruning, IEEE Trans. Audio Electroacoustics, vol.19, no.4, pp , Dec [16] T. Sreenivas and P. Rao, FFT algorithm for both input and output pruning, IEEE Trans. Acoust. Speech Signal Process., vol.assp- 27, no.3, pp , June [17] A. Kokaram, R. Morris, W. Fitzgerald, and P. Rayner, Detection of missing data in image sequences, IEEE Trans. Image Process., vol.4, no.11, pp , Nov

8 1504 IEICE TRANS. FUNDAMENTALS, VOL.E94 A, NO.7 JULY 2011 Xiaoyong Zhang received his B.E. degree in electrical engineering from Guizhou University of Technology, Guiyang, China, in 2000, and his M.E. degree in electrical engineering from Guizhou University, Guiyang, China, in He is currently working toward a D.E. degree at Tohoku University, Sendai, Japan. His research interests image and video processing. He received the Best Paper Prize of IEEE Sendai Section Student Award in 2009 and the Student Award of the International Symposium on Communications and Information Technologies (ISCIT) in He is a student member of the IEEE. Masahide Abe received his Bachelor of Engineering, Master of Information Sciences, and Doctor of Engineering degrees from Tohoku University, Sendai, Japan in 1994, 1996, and 1999, respectively. In 1999, he joined the Graduate School of Engineering at Tohoku University, Sendai, Japan, where he is currently an Associate Professor. His research interests include digital signal processing, image processing, adaptive digital filtering and evolutionary computation. He received the Young Engineer Award from the Institute of Electrics, Information and Communication Engineers (IEICE) of Japan in 1997, and the Young Excellent Author Award of the 13th IEICE Workshop on Circuits and Systems in Karuizawa in He is a member of the IEEE, the Society of Instrument and Control Engineers of Japan, and the Research Institute of Signal Processing, Japan. Masayuki Kawamata received his B.E., M.E., and D.E. degrees in electronic engineering from Tohoku University, Sendai, Japan, in 1977, 1979, and 1982, respectively. He was an Associate Professor in the Graduate School of Information Sciences at Tohoku University and is currently a Professor in the Graduate School of Engineering at Tohoku University. His research interests include 1-D and multidimensional digital signal processing, intelligent signal processing, control theory, and linear system theory. He received the Outstanding Transaction Award from the Society of Instrument and Control Engineers of Japan in 1984 (with T. Higuchi), the Outstanding Literary Work Award from the Society of Instrument and Control Engineers of Japan in 1996 (with T. Higuchi), and the 11th IBM Japan Scientific Award in Electronics in He is a member of the IEEE, the Society of Instrument and Control Engineers of Japan, the Information Processing Society of Japan, the Institute of Image Information and Television Engineers of Japan. He is an IEEE Senior Member.