Basis Iterative Pursuit for Image Sequence Denoising
Brian Eriksson ECE 738 Final Project Abstract An iterative method is purposed in this paper using the basis pursuit algorithm for spatial denoising, coupled with temporal wavelet denoising to result in a denoised video signal. Introduction Several new techniques have been developed recently for the purposes of denoising images. The most promising of these techniques have been the curvelet and undecimated wavelet transforms. Using a basis pursuit algorithm, the advantages of each algorithm can be taken advantage of to produce a spatially denoised image. When working with image sequences, the correlation between frames can be used to denoise images. Considering each pixel as a time-domain signal across multiple frames, this signal can be denoised using hard thresholding of wavelets. This paper purposes a iterative method of basis pursuit spatial denoising combined with temporal denoising using wavelets to produce a denoised image sequence signal. Curvelet Algorithm The Curvelet transform (original purposed in [4]) consists of an overcomplete representation of an image using a series of L2 energy measurements ranging across scale, orientation, and position. Each curvelet consists of a tight frame constrained over a slice of the fourier domain. In the spatial domain, the curvelet is a scaled and rotated gabor signal along the width and is a scaled and rotated gaussian signal along the length. One of the most important properties of the curvelet is length = width^2 (length = 2^-J, width = 2^-(2*J), J = scale). This allows for the curvelet to act like a needle at fine scale representations. A brief overview of the mathematical framework from [2] is now presented to give the reader a formal representation of curvelets. Each curvelet is defined by three parameters (J, K, L). J = Scale L = orientation K = location
Brian Eriksson Iterative Basis Pursuit for Image Sequence Denoising
2
Parabolic Scaling Matrix:
22 j DJ = 0
0 2J
Rotation Angle:
J = 2 * 2 - J * l
Translation Parameter:
k = (k1 * 1 , k2 * 2 )
1 , 2 = normalizing _ cons tan ts
( x1 , x2 ) = ( x1 ) * ( x2 ) ( x1 ) = Gabor ( x1 ) ( x2 ) = Gaussian( x2 )
Finally, the curvelet parameterized by (J,K,L) can be defined as
3j 2
Curvelet Basis Function:
( j ,l ,k ) ( x1 , x2 ) = 2 * ( DJ * R J * ( x1 , x2 ) - k )
Curvelet Properties: -Anisotropy Scaling Law
width length 2
-Directional Sensitivity
# _ of _ Orientations =
1 J
-Spatial Localization -Curvelet coefficients form a Cartesian coordinate grid with spacing proportional to the length in
J
direction and the width of the
curvelet in the normal (respective to
J ) direction.
-Oscillatory Nature -Across a ridge, a curve displays an oscillatory nature. This is shown by the use of the gabor waveform in the normal direction.
Brian Eriksson Iterative Basis Pursuit for Image Sequence Denoising
3
Now a more intuitive graphical explaination of the curvelet transform is presented.
(Spatial Domain)
(Curvelet Coefficient Domain)
Figure Example of a group of curvelets at a fixed orientation and scale and its relationship to the curvelet transform. Each coefficient relates to an L2 energy measurement of the oval region in the spatial domain.
(A)
(B)
(C)
Figure Example of curvelets with fixed scale and location, varying along orientation Using curvelets of varying orientation (figure above) the resulting coefficients would be very close to zero for (A) due to only a small amount of edge energy included in the curvelet area. (B) would have a slightly larger coefficient but still relatively small in comparison to the coefficient related to figure (C) which would be quite large due to the tight frame of the curvelet being tangential to the edge.
Brian Eriksson Iterative Basis Pursuit for Image Sequence Denoising
4
(A) (B) (C) Figure Example of curvelets with fixed orientation and location, varying across scale Using curvelets of varying scale aligned with an edge (figure), one can see how the changes in scale will affect the resulting coefficient. For a coarse scale decomposition (A), the curvelet coefficient resulting will be of moderate size due to the frame being tangential to the edge, but would contain a large amount of area without any edge energy. The medium scale decomposition (B) will be larger due to the length of the curvelet having increased therefore containing more of the edge. The fine scale representation (C) will result in the largest coefficient due to almost the entire curvelet frame being filled with the edge energy.
(A)
(B)
(C)
Figure Example of curvelets with fixed orientation and scale, varying across location Using curvelets of varying location the strength of the algorithm becomes obvious as figures (A) and (B) would result in a coefficient very close to zero (curvelets are not compactly supported), while the coefficient related to (C) would be very large due to the edge being contained in the curvelet frame.
Brian Eriksson Iterative Basis Pursuit for Image Sequence Denoising
5
In [1] it is shown that curvelets give a sparse representation of edges in C2 space. The problem with curvelets is their poor presentation of point singularities, which relates to points or corners in an image. This is shown at length in [2], as using the curvelet transform to reconstruct a noisy version of the Lena image results in very high quality performance on the elongated potions of the image (hat edges, outline of lena's face), but performs quite poorly on the fine scale edges (eyes, mouth). Undecimated Wavelet Algorithm The standard one-dimensional A Trous Wavelet filter structure has the appearance of:
HPF LPF
2 2
HPF LPF
2 2
HPF LPF
2 2
Figure - Analysis Phase 2 2 2 2 2 2
HPF HPF HPF
+
LPF
+
LPF
+
LPF
Figure - Synthesis Phase This is the structure that is used when a normal orthogonal discrete wavelet transform is performed. The undecimated wavelet transform is performed by removing all the upsampling and downsampling blocks from the filter bank. In this representation, the signal is never decimated (downsampled), so the wavelet coefficients at each level are
Brian Eriksson Iterative Basis Pursuit for Image Sequence Denoising
6
the same size as the original signal. This is a change from the critically sampled orthogonal wavelet transform, where the size of the coefficients after filtering is the same size of the original image. For an image transformed under the Undecimated Discrete Wavelet Transform (UDWT), decomposition and reconstruction under one level would have the filter bank representation of:
LPF Rows
LPF Cols HPF Cols LPF Cols HPF Cols
LPF Cols HPF Cols LPF Cols HPF Cols
LPF Rows
HPF Rows
HPF Rows
Figure Undecimated Wavelet Transform It should be noted that after one level of analysis, the size of the wavelet coefficients would be 4 times larger than the original image size. Like Orthogonal wavelets, the Undecimated wavelet transform will obtain a sparse representation of singularities (corners in an image) while performing quite poorly on the elongated portions (long edges). This is shown in detail in [2], as the reconstruction of the noisy Lena image results in very good performance in the fine detail edges (facial features), while doing quite poorly on the long edges (outline of Lena, hat edges).
Basis Pursuit Method Currently we have presented the curvelet method that works well for long curves but poorly for sharp edges, and the undecimated wavelet method that works well for sharp edges but poorly for long curves. Intuitively, one would want to use what the both methods do well, and leave out what they do poorly. This is the
Brian Eriksson Iterative Basis Pursuit for Image Sequence Denoising
7
intention of the Basis Pursuit method. A generalized method of basis pursuit is described in [2], below is the modified version implemented in this paper.
Twave = Undecimated _ Wavelet _ Transform T -1 wave = Inverse _ Undecimated _ Wavelet _ Transform Tcurve = Curvelet _ Transform T -1 curve = Inverse _ Curvelet _ Transform s = noisy _ image =1
=
1 iterations
wave Noise = Twave ( s ) curve Noise = Tcurve ( s ) u = zero _ image
errorval = std _ noise significantval = 2 * std _ noise
for k = 1:iterations
wave = Twave (u )
residualwave = wave
Noise
- wave
if residual wave ( k , j ) > errorval &
wave
Noise
(k , j ) > significantval
wave (k , j ) = wave
End u = T -1 wave ( wave )
Noise
(k , j )
wave ( k , j ) = soft _ threshold ( wave ( k , j ), )
curve = Tcurve (u )
residual curve = curve
if
Noise
- curve
& curve
Noise
residual curve (k , j ) > errorval
(k , j ) > significantval
Brian Eriksson Iterative Basis Pursuit for Image Sequence Denoising
8
curve (k , j ) = curve Noise ( k , j )
curve (k , j ) = soft _ threshold ( curve (k , j ), )
end u = T -1 curve ( curve ) u = clip _ negative _ values(u ) = -
end
The ideal results of this algorithm would be an image that reconstructs the long edges of an image like the curvelet transform, while reconstructing the sharp edges like the undecimated wavelet transform. Temporal Wavelet Algorithm With an image sequence, there is structured redundancy across multiple frames. This redundancy can be related to the background or a slow moving object. Ideally we would want to average the background pixels and modify them to values as constant as possible while still allowing each pixel to change for movement.
(A)
Pixel Value
Frame Number
Brian Eriksson Iterative Basis Pursuit for Image Sequence Denoising
9
(B) Figure Pixel observed across a group of frames The peaks of the pixel value graph are assumed to be the result of motion in the image, all other values are assumed to be background that is under the effect of noise. Using a standard (decimated/orthogonal) haar wavelet transform, the pixel graph is denoised using hard All thresholding. coefficients below a certain range (value determined by the noise model) are thresholded to zero. The result should appear like the graph below.
Figure Wavelet denoised pixel signal Ideally the result of this method would be background pixels averaged out across all frames, while pixels that are the result of motion would be maintained at their original value. Iterative Basis Pursuit Algorithm Description An iterative method is now purposed using both basis pursuit and the temporal wavelet method. Prior to this algorithm, the groups of frames are denoised using a light wiener filter and block averaging. original_frames = Noisy group of frames std = standard deviation of additive noise
Frame Number
Pixel Value
Brian Eriksson Iterative Basis Pursuit for Image Sequence Denoising
10
wiener _ frames = Light _ Wiener _ Filter (original _ frames ) block _ frames = Block _ Averaging _ Filter ( wiener _ frames ) new _ frames = Basis _ Persuit _ A lg oirhtm(block _ frames, std ) std _ array = [0.3 * std , 0.15 * std , 0.075 * std ]
For k = 1 : 3
std = std _ array (k ) basis _ frames = Basis _ Pursuit _ a lg orithm(new _ frames,3* std ) new _ frames = Temporal _ Wavelet _ a lg orithm(basis _ frames,1.2 * std )
End
Noisy Image Noisy Image Base Denoising Techniques
Base
Basis Pursuit Basis Pursuit Temporal Wavelet Temporal Denoising
Update Updat Assumed Noise e
Denoised Image Denoised
The idea behind this is that basis pursuit denoising will reconstruct the frame image while also reconstructing some of the noise. The temporal wavelet denoising technique is very good at removing noise in redundant pixels, but will add its own noise caused by movement in the image. By iterating multiple times and decreasing the amount of assumed noising each time, the movement noise caused by the temporal wavelet denoising will be removed by the basis pursuit algorithm and a majority of the noise reconstructed by basis pursuit will be eliminated by temporal wavelets.
Brian Eriksson Iterative Basis Pursuit for Image Sequence Denoising
11
Experiments The iterative basis pursuit method was tested using two image sequence sets. One set, the MOBCAL sequence, consists a real video sequence with many different objects moving simultaneously. The other set, the cartoon sequence, is a series of drawings with a large amount of constant background with characters running across the screen. The sequences were tested using a light noise condition (variance = 100) and a heavy noise condition (variance = 900). Results After examining the preliminary results it was determined that the interesting cases were the MOBCAL sequence under light noise and the CARTOON sequence under heavy noise. The first results received were very optimistic for the heavy noise cases, with iterative method PSNR performing almost 1 dB better than all other methods. The light noise case showed a significantly different story as the iterative basis pursuit method was far too drastic with its denoising and ended up blurring the image. The resulting PSNR values were more than 1 dB away from the best method for this case (the base denoising methods). These observations were taken into consideration and the iterative method's threshold values were modified in order to even up the performance results.
Figure: MOBCAL Sequence with additive noise standard deviation = 10. (Plus sign) = Iterative Basis Pursuit Method, (Circle) = Single Basis Pursuit, (Square) = Curvelet Denoising, (Diamond) = Base Denoising methods
Brian Eriksson Iterative Basis Pursuit for Image Sequence Denoising
12
As shown in the figure, the iterative basis pursuit method still performs below the base denoising methods in this case. After modifying the method, the results improved from 1 dB away to only roughly 0.4 dB away from the best denoising method.
Figure: (A) The original frame image, (B) Frame image after additive white noise
Brian Eriksson Iterative Basis Pursuit for Image Sequence Denoising
13
Figure: (A) Frame denoised using Basis Pursuit spatial method, (B) Frame denoised using Curvelet denoising, (C) Frame denoised after iterative basis pursuit method From a distance the iterative basis pursuit method appears to perform the best out of all three methods. In the iterative basis image all the smooth areas appear to be completely denoised with very little variation along smooth patches.
Brian Eriksson Iterative Basis Pursuit for Image Sequence Denoising
14
Figure: (A) Frame denoised using Basis Pursuit spatial method, (B) Frame denoised using Curvelet denoising, (C) Frame denoised after iterative basis pursuit method Upon zooming in on an area containing a majority of smooth areas, the power of both basis pursuit methods (A,C) can be seen. Specifically compare the noise contained on the surface of the ball and on the edge of the calendar between the basis pursuit methods (A,C) and the curvelet denoising method (B). By comparing the dark sections of the wall, the difference between the spatial basis pursuit (A) and the iterative basis pursuit method (C) can be seen.
Brian Eriksson Iterative Basis Pursuit for Image Sequence Denoising
15
Figure: (A) Frame denoised using Basis Pursuit spatial method, (B) Frame denoised using Curvelet denoising, (C) Frame denoised after iterative basis pursuit method The failure of the iterative method can be seen in the figure above. Notice the detail of the pigs on the wallpaper in the curvelet denoised method (B). Now contrast again the iterative denoised image in (C). Although the method does fairly well in denoising the smooth portions of the wallpaper, the details on the pigs is almost completely eliminated. This causes the drop in PSNR for the iterative basis pursuit method.
Brian Eriksson Iterative Basis Pursuit for Image Sequence Denoising
16
Figure: Cartoon Sequence with additive noise standard deviation = 30. (Plus sign) = Iterative Basis Pursuit Method, (Ci...