Scissors Intelligent for Image Composition
Eric N. Mortensen1 William A. Barrett2
Brigham Young University
Abstract
We present a new, interactive tool called Intelligent Scissors which we use for image segmentation and composition. Fully automated segmentation is an unsolved problem, while manual tracing is inaccurate and laboriously unacceptable. However, Intelligent Scissors allow objects within digital images to be extracted quickly and accurately using simple gesture motions with a mouse. When the gestured mouse position comes in proximity to an object edge, a live-wire boundary snaps to, and wraps around the object of interest. Live-wire boundary detection formulates discrete dynamic programming (DP) as a two-dimensional graph searching problem. DP provides mathematically optimal boundaries while greatly reducing sensitivity to local noise or other intervening structures. Robustness is further enhanced with on-the-y training which causes the boundary to adhere to the specic type of edge currently being followed, rather than simply the strongest edge in the neighborhood. Boundary cooling automatically freezes unchanging segments and automates input of additional seed points. Cooling also allows the user to be much more free with the gesture path, thereby increasing the efciency and nesse with which boundaries can be extracted. Extracted objects can be scaled, rotated, and composited using live-wire masks and spatial frequency equivalencing. Frequency equivalencing is performed by applying a Butterworth lter which matches the lowest frequency spectra to all other image components. Intelligent Scissors allow creation of convincing compositions from existing images while dramatically increasing the speed and precision with which objects can be extracted.
sequences. Further, due to the wide variety of image types and content, most current computer based segmentation techniques are slow, inaccurate, and require signicant user input to initialize or control the segmentation process. This paper describes a new, interactive, digital image segmentation tool called Intelligent Scissors which allows rapid object extraction from arbitrarily complex backgrounds. Intelligent Scissors boundary detection formulates discrete dynamic programming (DP) as a two-dimensional graph searching problem. Presented as part of this tool are boundary cooling and on-the-y training, which reduce user input and dynamically adapt the tool to specic types of edges. Finally, we present live-wire masking and spatial frequency equivalencing for convincing image compositions.
2. Background
Digital image segmentation techniques are used to extract image components from their surrounding natural background. However, currently available computer based segmentation tools are typically primitive and often offer little more advantage than manual tracing. Region based magic wands, provided in many desktop applications, use an interactively selected seed point to grow a region by adding adjacent neighboring pixels. Since this type of region growing does not provide interactive visual feedback, resulting region boundaries must usually be edited or modied. Other popular boundary denition methods use active contours or snakes[1, 5, 8, 15] to improve a manually entered rough approximation. After being initialized with a rough boundary approximation, snakes iteratively adjust the boundary points in parallel in an attempt to minimize an energy functional and achieve an optimal boundary. The energy functional is a combination of internal forces, such as boundary curvature, and external forces, like image gradient magnitude. Snakes can track frame-to-frame boundary motion provided the boundary hasnt moved drastically. However, active contours follow a pattern of initialization followed by energy minimization; as a result, the user does not know what the nal boundary will look like when the rough approximation is input. If the resulting boundary is not satisfactory, the process must be repeated or the boundary must be manually edited. We provide a detailed comparison of snakes and Intelligent Scissors in section 3.6. Another class of image segmentation techniques use a graph searching formulation of DP (or similar concepts) to nd globally optimal boundaries [2, 4, 10, 11, 14]. These techniques differ from snakes in that boundary points are generated in a stage-wise optimal cost fashion whereas snakes iteratively minimize an energy functional for all points on a contour in parallel (giving the appearance of wiggling). However, like snakes, these graph searching techniques typically require a boundary template--in the form of a manually entered rough approximation, a gure of merit, etc.--which is used to impose directional sampling and/or searching constraints. This limits these techniques to a boundary search with one degree of freedom within a window about the two-dimensional boundary template. Thus, boundary extraction using previous graph searching techniques is non-interactive (beyond template specication), losing the benets of further human guidance and expertise.
1. Introduction
Digital image composition has recently received much attention for special effects in movies and in a variety of desktop applications. In movies, image composition, combined with other digital manipulation techniques, has also been used to realistically blend old lm into a new script. The goal of image composition is to combine objects or regions from various still photographs or movie frames to create a seamless, believable, image or image sequence which appears convincing and real. Fig. 9(d) shows a believable composition created by combining objects extracted from three images, Fig. 9(a-c). These objects were digitally extracted and combined in a few minutes using a new, interactive tool called Intelligent Scissors. When using existing images, objects of interest must be extracted and segmented from a surrounding background of unpredictable complexity. Manual segmentation is tedious and time consuming, lacking in precision, and impractical when applied to long image
1 2
[e-mail], Dept. of Comp. Sci., BYU, Provo, UT 84602 [phone] [e-mail], Dept. of Comp. Sci., BYU, Provo, UT 84602 [phone]
The most important difference between previous boundary nding techniques and Intelligent Scissors presented here lies not in the boundary dening criteria per se, but in the method of interaction. Namely, previous methods exhibit a pattern of boundary approximation followed by boundary renement, whereas Intelligent Scissors allow the user to interactively select the most suitable boundary from a set of all optimal boundaries emanating from a seed point. In addition, previous approaches do not incorporate onthe-y training or cooling, and are not as computationally efcient. Finally, it appears that the problem of automated matching of spatial frequencies for digital image composition has not been addressed previously.
Since the laplacian zero-crossing creates a binary feature, fZ(q) does not distinguish between strong, high gradient edges and weak, low gradient edges. However, gradient magnitude provides a direct correlation between edge strength and local cost. If Ix and Iy represent the partials of an image I in x and y respectively, then the gradient magnitude G is approximated with
G
=
2 Ix
+ I2. y
The gradient is scaled and inverted so high gradients produce low costs and vice-versa. Thus, the gradient component function is
fG
3. Intelligent Scissors
Boundary denition via dynamic programming can be formulated as a graph searching problem [10] where the goal is to nd the optimal path between a start node and a set of goal nodes. As applied to image boundary nding, the graph search consists of nding the globally optimal path from a start pixel to a goal pixel-in particular, pixels represent nodes and edges are created between each pixel and its 8 neighbors. For this paper, optimality is dened as the minimum cumulative cost path from a start pixel to a goal pixel where the cumulative cost of a path is the sum of the local edge (or link) costs on the path. 3.1. Local Costs Since a minimum cost path should correspond to an image component boundary, pixels (or more accurately, links between neighboring pixels) that exhibit strong edge features should have low local costs and vice-versa. Thus, local component costs are created from the various edge features: Image Feature Laplacian Zero-Crossing Gradient Magnitude Gradient Direction Formulation fZ fG fD
=
m ax
m ax
(G ) G = (G )
1
G m ax
(G )
(3)
giving an inverse linear ramp function. Finally, gradient magnitude costs are scaled by Euclidean distance. To keep the resulting maximum gradient at unity, fG(q) is scaled by 1 if q is a diagonal neighbor to p and by 1/2 if q is a horizontal or vertical neighbor. The gradient direction adds a smoothness constraint to the boundary by associating a high cost for sharp changes in boundary direction. The gradient direction is the unit vector dened by Ix and Iy. Letting D(p) be the unit vector perpendicular (rotated 90 degrees clockwise) to the gradient direction at point p (i.e., for D(p) = (Iy(p), -Ix(p))), the formulation of the gradient direction feature cost is
fD (p, q )
=
1
{ c o s [ d p ( p , q ) ] 1 +
cos
[ d q ( p , q ) ] 1 }
(4)
where
d p (p, q ) d q (p, q )
= =
D' L
(p ) L (p, q )
(p, q ) D ' (q )
are vector dot products and
L
(p, q ) = {
q p
p; q;
if if
D' D'
(p ) (q p ) 0 (p ) (q p ) < 0
(5)
The local costs are computed as a weighted sum of these component functionals. Letting l(p,q) represents the local cost on the directed link from pixel p to a neighboring pixel q, the local cost function is
l (p, q )
= Z f Z ( q ) + D f D ( p , q ) + G f G ( q )
(1)
where each is the weight of the corresponding feature function. (Empirically, weights of Z = 0.43, D = 0.43, and G = 0.14 seem to work well in a wide range of images.) The laplacian zero-crossing is a binary edge feature used for edge localization [7, 9]. Convolution of an image with a laplacian kernel approximates the 2nd partial derivative of the image. The laplacian image zero-crossing corresponds to points of maximal (or minimal) gradient magnitude. Thus, laplacian zero-crossings represent good edge properties and should therefore have a low local cost. If IL(q) is the laplacian of an image I at pixel q, then
fZ (q )
is the bidirectional link or edge vector between pixels p and q. Links are either horizontal, vertical, or diagonal (relative to the position of q in ps neighborhood) and point such that the dot product of D(p) and L(p, q) is positive, as noted in (5). The neighborhood link direction associates a high cost to an edge or link between two pixels that have similar gradient directions but are perpendicular, or near perpendicular, to the link between them. Therefore, the direction feature cost is low when the gradient direction of the two pixels are similar to each other and the link between them. 3.2. Two-Dimensional Dynamic Programming As mentioned, dynamic programming can be formulated as a directed graph search for an optimal path. This paper utilizes an optimal graph search similar to that presented by Dijkstra [6] and extended by Nilsson [13]; further, this technique builds on and extends previous boundary tracking methods in 4 important ways: 1. It imposes no directional sampling or searching constraints. 2. It utilizes a new set of edge features and costs: laplacian zero-crossing, multiple gradient kernels. 3. The active list is sorted with an O(N) sort for N nodes/pixels. 4. No a priori goal nodes/pixels are specied. First, formulation of boundary nding as a 2-D graph search eliminates the directed sampling and searching restrictions of previous implementations, thereby allowing boundaries of arbitrary com-
= {
0 ; if 1 ; if
IL (q ) IL (q )
=
0
0
(2)
However, application of a discrete laplacian kernel to a digital image produces very few zero-valued pixels. Rather, a zero-crossing is represented by two neighboring pixels that change from positive to negative. Of the two pixels, the one closest to zero is used to represent the zero-crossing. The resulting feature cost contains single-pixel wide cost canyons used for boundary localization.
plexity to be extracted. Second, the edge features used here are more robust and comprehensive than previous implementations: we maximize over different gradient kernels sizes to encompass the various edge types and scales while simultaneously attempting to balance edge detail with noise suppression [7], and we use the laplacian zero-crossing for boundary localization and ne detail livewire snapping. Third, the discrete, bounded nature of the local edge costs permit the use of a specialized sorting algorithm that inserts points into a sorted list (called the active list) in constant time. Fourth, the live-wire tool is free to dene a goal pixel interactively, at any free point in the image, after minimum cost paths are computed to all pixels. The latter happens fast enough that the free point almost always falls within an expanding cost wavefront and interactivity is not impeded. The Live-Wire 2-D dynamic programming (DP) graph search algorithm is as follows:
Algorithm: Live-Wire 2-D DP graph search.
Input: s l(q,r)
11 14 11 7 6 8 11 12 10
13 11 6 4 2 3 5 4 9
12 7 3 6 7 4 2 2 7
9 4 5 11 10 7 8 1 5
5 2 7 13 15 9 3 5 9
8 5 9 18 15 13 4 6 8
3 8 12 17 21 14 5 3 5
1 4 11 14 19 15 7 2 3
2 6 10 8 8 9 2 4 7
4 3 7 5 3 5 5 8 8
10 8 4 2 5 6 9 12 15
(a)
6 7 4 13 2 0 7 11 1 7 7 4 13 2 0 7 9 1 6 5 6 14 20 16 18 7 4 13 6 2 0 7 12 9 1 6 14 5 6 14 23 9 13
(b)
41 35 23 16 16 13 6 2 0 7 31 22 21 27 20 12 9 1 6
(c)
29 24 28 38 29 14 5 6 14 35 22 9 12 17 52 28 14 13 18 35 21 15 17 35 27 18 19 24 28 25 23 27 30 35 29 37
(d)
{Start (or seed) pixel.} {Local cost function for link between pixels q and r.}
38 28
29 18 12 8 6 7 4 13
Data Structures: L {List of active pixels sorted by total cost (initially empty).} N(q) {Neighborhood set of q (contains 8 neighbors of pixel).} e(q) {Boolean function indicating if q has been expanded/processed.} g(q) {Total cost function from seed point to q.} Output: p
18 14 14 18
32 31 32 40
{Pointers from each pixel indicating the minimum cost path.}
16 18
Algorithm: g(s)0; Ls; {Initialize active list with zero cost seed pixel.} while L do begin {While still points to expand:} qmin(L); {Remove minimum cost pixel q from active list.} e(q)TRUE; {Mark q as expanded (i.e., processed).} for each rN(q) such that not e(r) do begin gtmpg(q)+l(q,r); {Compute total cost to neighbor.} if rL and gtmp<g(r) then {Remove higher cost neighbors } rL; { from list.} if rL then begin {If neighbor not on list, } { assign neighbors total cost, } g(r)gtmp; p(r)q; { set (or reset) back pointer, } Lr; { and place on (or return to) } end { active list.} end end
(e)
45 38 28 18 14 14 18 16 18 41 29 18 12 8 6 7 4 13 35 23 16 16 13 6 2 0 7 31 22 21 27 20 12 9 1 6 29 24 28 38 29 14 5 6 14 35 29 37 53 35 22 9 12 17 33 37 46 59 49 28 14 13 18 34 38 49 53 54 35 21 15 17 36 42 47 39 35 27 18 19 24 40 39 40 33 28 25 23 27 30 50 43 35 31 32 31 32 39 45
Notice that since the active list is sorted, when a new, lower cumulative cost is computed for a pixel already on the list then that point must be removed from the list in order to be added back to the list with the new lower cost. Similar to adding a point to the sorted list, this operation is also performed in constant time. Figure 1 demonstrates the use of the 2-D DP graph search algorithm to create a minimum cumulative cost path map (with corresponding optimal path pointers). Figure 1(a) is the initial local cost map with the seed point circled. For simplicity of demonstration the local costs in this example are pixel based rather than link based and can be thought of as representing the gradient magnitude cost feature. Figure 1(b) shows a portion of the cumulative cost and pointer map after expanding the seed point (with a cumulative cost of zero). Notice how the diagonal local costs have been scaled by Euclidean distance (consistent with the gradient magnitude cost feature described previously). Though complicating the example, weighing by Euclidean distance is necessary to demonstrate that the cumulative costs to points currently on the active list can change if even lower cumulative costs are computed from as yet unexpanded neighbors. This is demonstrated in Figure 1(c) where two points
(f) Figure 1: (a) Initial local cost matrix. (b) Seed point (shaded) expanded. (c) 2 points (shaded) expanded. (d) 5 points (shaded) expanded. (e) 47 points expanded. (f) Finished total cost and path matrix with two of many paths (free points shaded) indicated.
have now been expanded--the seed point and the next lowest cumulative cost point on the active list. Notice how the points diagonal to the seed point have changed cumulative cost and direction pointers. The Euclidean weighting between the seed and diagonal points makes them more costly than non-diagonal paths. Figures 1(d), 1(e), and 1(f) show the cumulative cost/direction pointer map at various stages of completion. Note how the algorithm produces a wavefront of active points emanating from the initial start point, called the seed point, and that the wavefront grows out faster where there are lower costs. 3.3. Interactive Live-Wire Segmentation Tool Once the optimal path pointers are generated, a desired boundary segment can be chosen dynamically via a free point. Interactive movement of the free point by the mouse cursor causes the boundary to behave like a live-wire as it adapts to the new minimum cost path by following the optimal path pointers from the free point back
Figure 2: Image demonstrating how the live-wire segment adapts and snaps to an object boundary as the free point moves (via cursor movement). The path of the free point is shown in white. Live-wire segments from previous free point positions (t0, t1, and t2) are shown in green.
(a)
to the seed point. By constraining the seed point and free points to lie near a given edge, the user is able to interactively snap and wrap the live-wire boundary around the object of interest. Figure 2 demonstrates how a live-wire boundary segment adapts to changes in the free point (cursor position) by latching onto more and more of an object boundary. Specically, note the live-wire segments corresponding to user-specied free point positions at times t0, t1, and t2. Although Fig. 2 only shows live-wire segments for three discrete time instances, live-wire segments are actually updated dynamically and interactively (on-the-y) with each movement of the free point. When movement of the free point causes the boundary to digress from the desired object edge, interactive input of a new seed point prior to the point of departure reinitiates the 2-D DP boundary detection. This causes potential paths to be recomputed from the new seed point while effectively tieing off the boundary computed up to the new seed point. Note again that optimal paths are computed from the seed point to all points in the image (since the 2-D DP graph search produces a minimum cost spanning tree of the image [6]). Thus, by selecting a free point with the mouse cursor, the interactive live-wire tool is simply selecting an optimal boundary segment from a large collection of optimal paths. Since each pixel (or free point) denes only one optimal path to a seed point, a minimum of two seed points must be placed to ensure a closed object boundary. The path map from the rst seed point of every object is maintained during the course of an objects boundary denition to provide a closing boundary path from the free point. The closing boundary segment from the free point to the rst seed point expedites boundary closure. Placing seed points directly on an objects edge is often difcult and tedious. If a seed point is not localized to an object edge then spikes results on the segmented boundary at those seed points (since
(b) Static Cost Map
MG Cost MG Cost
Dynamic Cost Map
0
0
Gradient Magnitude
nG
0
0
Gradient Magnitude
nG
(c) (d) Figure 4: Comparison of live-wire (a) without and (b) with dynamic training. (a) Without training, the live-wire segment snaps to nearby strong edges. (b) With training, it favors edges with similar characteristics as those just learned. (c) The static gradient magnitude cost map shows that without training, high gradients are favored since they map to low costs. However, with training, the dynamic cost map (d) favors gradients similar to those sampled from the previous boundary segment.
the boundary is forced to pass through the seed points). To facilitate seed point placement, a cursor snap is available which forces the mouse pointer to the maximum gradient magnitude pixel within a user specied neighborhood. The neighborhood can be anywhere from 11 (resulting in no cursor snap) to 1515 (where the cursor can snap as much as 7 pixels in both x and y). Thus, as the mouse cursor is moved by the user, it snaps or jumps to a neighborhood pixel representing a good static edge point. 3.4. Path Cooling Generating closed boundaries around objects of interest can require as few as two seed points (for reasons given previously). Simple objects typically require two to ve seed points but complex objects may require many more. Even with cursor snap, manual placement of seed points can be tedious and often requires a large portion of the overall boundary denition time.
(a) (b) Figure 3: Comparison of live-wire without (a) and with (b) cooling. Withot cooling (a), all seed points must be placed manually on the object edge. With cooling (b), seed points are generated automatically as the live-wire segment freezes.
Automatic seed point generation relieves the user from precise manual placement of seed points by automatically selecting a pixel on the current active boundary segment to be a new seed point. Selection is based on path cooling which in turn relies on path coalescence. Though a single minimum cost path exists from each pixel to a given seed point, many paths coalesce and share portions of their optimal path with paths from other pixels. Due to Bellmans Principle of Optimality [3], if any two optimal paths from two distinct pixels share a common point or pixel, then the two paths are identical from pixel that back to the seed point. This is particularly noticeable if the seed point is placed near an object edge and the free point is moved away from the seed point but remains in the vicinity of the object edge. Though a new optimal path is selected and displayed every time the mouse cursor moves, the paths are typically identical near the seed point and object edges and only change local to the free point. As the free point moves farther and farther away from the seed point, the portion of the active live-wire boundary segment that does not change becomes longer. New seed points are generated at the end of a stable segment (i.e., that has not changed recently). Stability is measured by time (in milliseconds) on the active boundary and path coalescence (number of times the path has been redrawn from distinct free points). This measure of stability provides the live-wire segment with a sense of cooling. The longer a pixel is on a stable section of the live-wire boundary, the cooler it becomes until it eventually freezes and automatically produces a new seed point. Figure 3 illustrates the benet of path cooling. In Fig. 3(a), the user must place each seed point manually on the object boundary. However, with cooling (Fig. 3(b)), only the rst seed point (and last free point) need to be specied manually; the other seed points were generated automatically via cooling. 3.5. Interactive Dynamic Training On occasion, a section of the desired object boundary may have a weak gradient magnitude relative to a nearby strong gradient edge. Since the nearby strong edge has a relatively lower cost, the live-wire segment snaps to the strong edge rather than the desired weaker edge. This can be seen in Fig. 4(a). The desired boundary is the womans (Harriets) cheek. However, since part of it is so close to the high contrast shoulder of the man (Ozzie), the live-wire snaps to the shoulder. Training allows dynamic adaptation of the cost function based on a sample boundary segment. Training exploits an objects boundary segment that is already considered to be good and is performed dynamically as part of the boundary segmentation process. As a result, trained features are updated interactively as an object boundary is being dened. On-the-y training eliminates the need for a separate training phase and allows the trained feature cost functions to adapt within the object being segmented as well as between objects in the image. Fig. 4(b) demonstrates how a trained live-wire segment latches onto the edge that is similar to the previous training segment rather that the nearby stronger edge. To facilitate training and trained cost computation, a gradient magnitude feature map or image is precomputed by scaling the minimized gradient magnitude image, G', into an integer range of size nG (i.e., from 0 to nG - 1). The actual feature cost is determined by mapping these feature values through a look-up table which contains the scaled (weighted) cost for each value. Fig 4(c) illustrates edge cost based on gradient magnitude without training. Note that with training (Fig. 4(d)) edge cost plummets for gradients that are specic to the object of interests edges. Selection of a good boundary segment for training is made interactively using the live-wire tool. To allow training to adapt to slow (or smooth) changes in edge characteristics, the trained gradient magnitude cost function is based only on the most recent or
closest portion of the current dened object boundary. A training length, t, species how many of the most recent boundary pixels are used to generate the training statistics. A monotonically decreasing weight function (either linearly or Gaussian based) determines the contribution from each of the closest t pixels. This permits adaptive training with local dependence to prevent trained feature from being too subject to old edge characteristics. The closest pixel (i.e., the current active boundary segment endpoint) gets a weight of 1 and the point that is t pixels away, along the boundary from the current active endpoint, gets a minimal weight (which can be determined by the user). The training algorithm samples the precomputed feature maps along the closest t pixels of the edge segment and increments the feature histogram element by the corresponding pixel weight to generate a histogram for each feature involved in training. After sampling and smoothing, each feature histogram is then scaled and inverted (by subtracting the scaled histogram values from its maximum value) to create the feature cost map needed to convert feature values to trained cost functions. Since training is based on learned edge characteristics from the most recent portion of an objects boundary, training is most effective for those objects with edge properties that are relatively consistent along the object boundary (or, if changing, at least change smoothly enough for the training algorithm to adapt). In fact, training can be counter-productive for objects with sudden and/or dramatic changes in edge features. However, training can be turned on and off interactively throughout the denition of an object boundary so that it can be used (if needed) in a section of the boundary with similar edge characteristics and then turned off before a drastic change occurs. 3.6 Comparison with Snakes Due to the recent popularity of snakes and other active contours models and since the interactive boundary wrapping of the livewire may seem similar to the wiggling of snakes, we highlight what we feel are the similarities and their corresponding differences between snakes and Intelligent Scissors. Similarities (compare with corresponding differences below): 1. The gradient magnitude cost in Intelligent Scissors is similar to the edge energy functional used in snakes. 2. Both methods employ a smoothing term to minimize the effects of noise in the boundary. 3. Snakes and live-wire boundaries are both attracted towards strong edge features. 4. Both techniques attempt to nd globally optimal boundaries to try to overcome the effects of noise and edge dropout. 5. Snakes and Intelligent Scissors both require interaction as part of the boundary segmentation process. Differences (compare with corresponding similarities above): 1. The laplacian zero-crossing binary cost feature seems to have not been used previously in active contours models1 (or DP boundary tracking methods for that matter). 2. The active contour smoothing term is internal (i.e., based on the contours point positions) whereas the smoothing term for livewire boundaries is computed from external image gradient directions2(next page).
1. Kass et al. [8] did use a squared laplacian energy functional to show the relationship of scale-space continuation to the Marr-Hildreth edge detection theory. However, the squared laplacian does not represent a binary condition, nor could it since the variational calculus minimization used in [8] required that all functionals be differentiable.
3. Snakes are typically attracted to edge features only within the gravity of an edges gradient energy valley whereas the live-wire boundary can snap to strong edge features from arbitrary distances (since the 2-D DPs search window is the entire image). 4. Snakes are globally optimal over the entire contour whereas livewire boundaries are piece-wise optimal (i.e., optimal between seed points). We feel this creates a desirable balance between global optimality and local control. This piece-wise optimality also allows for path cooling and intra-object on-the-y training. 5. Finally, snakes rene (and interactively nudge by placing springs, etc.) a single rough boundary approximation where the live-wire tool interactively selects an optimal boundary segment from potentially all possible minimum cost paths. Interactive optimal 2-D path selection is what makes Intelligent Scissors work and is the key difference between Intelligent Scissors and all previous techniques. Snakes are interactively initialized with an approximate boundary contour (often requiring several manually placed points); this single contour is then iteratively adjusted in an attempt to minimize an energy functional. The livewire tool, on the other hand, is interactively initialized with just a single seed point and it then generates, at interactive speeds, all possible optimal paths from the seed point to every other point in the image, thus, allowing the user to interactively select the desired optimal boundary segment. As a result, Intelligent Scissors typically require less time and effort to segment an object than it takes to manually input an initial approximation to the object boundary. Actually, the live-wire tool is much more similar to previous stage-wise optimal boundary tracking approaches than it is to snakes, since Intelligent Scissors were developed as an interactive 2-D extension to previous optimal edge tracking methods rather than an improvement on active contours.
lution of the image. This may produce jaggies along object boundaries in a resulting composition. However, subpixel accuracy can be obtained by exploiting the signed output of the laplacian operator. That is, the position of the object edge can be estimated to subpixel accuracy by using a (linearly) weighted combination of the laplacian pixel values on either side of the zero-crossings. Since the live-wire boundary will not always correspond to a zero-crossing, jaggies can also be reduced by appropriate edge ltering, similar to anti-aliasing. Edge ltering is also desirable because real world images are acquired using nite image detectors and, as a result, pixels on or near an object boundary share information (i.e., color) from the object and the background. One approach to edge ltering is to perform a local post-smoothing of the image around the pasted object boundary. However, this does not account for the fact that the edge pixels of the cut object very likely contain some background information from the original image. This is most noticeable when an object is composited into a scene with a different background color than the objects original background color. A more general solution would determine how much of each edge pixel corresponds to the actual object color and weight them accordingly when combining into another image. 4.2. Spatial Frequency and Contrast Matching Once the object of interest has been segmented from the surrounding background we can scale it, rotate it, color it, or paste it onto another (destination) image. When pasting, it is desirable to perform image composition seamlessly in order to make it believable. That is, we should not be able to detect where the paste occurred. However, the source and the destination images will often have differing spatial frequencies or contrast due to differences in focus or lighting when the images were acquired. Thus, equivalencing of spatial frequencies and normalization of contrast is sometimes desirable in order to produce a convincing composition. Equivalencing of spatial frequencies is performed by matching the spectral content of the cut piece and the destination image in the vicinity where it is to be pasted. Convincing composition often requires the spectra of the object and the destination image to match. This is accomplished by low-pass ltering the spectrum with the higher frequency content to match that of the other. The spectrum with the higher frequency content is determined by parameter tting of a Butterworth low-pass lter (BLPF) to both spectra. Parameters corresponding to the spectrum with the lower frequency content are used to low-pass lter the spectrum of the other image. The BLPF B(u, v, d0, n) is given by
B
4. Image Composition with Intelligent Scissors
As mentioned, composition artists need an intelligent, interactive tool to facilitate image component boundary denition. Since Intelligent Scissors can quickly segment object from an image, it serves as a tool for cut and paste operations. After object boundaries have been extracted, object can be copied from the image and placed into a buffer (i.e., clipboard) for future processing and placement into another image, or the same image if desired. The cut object can be transformed--i.e., rotated, scaled, and translated, (RST)--before combination with the destination image. This is done using an interactive graphical tool with handles for RST control. The tool species a 2-D RST transformation matrix, M. The source image is then bilinearly interpolated through the matrix to paste the cut object into the destination image. Image composition often requires blending an object from one image into another image such that the cut-out object is not in the foreground. This requires the composition artist to slip the cutout object behind some scene components while leaving it in front of other components. This operation can again be performed using the live-wire tool to create a composition mask1. Scene components can be cut out of an image to create a mask such that any additions or changes to the scene will not affect masked pixels. 4.1. Edge Filtering As described, live-wire boundaries are limited by the pixel reso-
( u , v , d 0, n ) =
1
1
+
u
2
+ v2
2n
d0
where d0 is the distance of the cutoff frequency from the origin and n is the lter order. Equivalencing of spatial frequencies is performed by rst computing the fourier transforms, S(u, v) and I(u, v) of the source image S(x, y) and the destinati...