Coursehero >> California >> UC Irvine >> ICS 09

Course Hero has millions of student submitted documents similar to the one below including study guides, homework solutions, papers, exam answer keys and textbook solutions.

in Appeared IEEE ICDE 2009 Conference. Using Semantics for Speech Annotation of Images Chaitanya Desai Dmitri V. Kalashnikov Sharad Mehrotra Nalini Venkatasubramanian Computer Science Department University of California, Irvine I. INTRODUCTION Digital cameras and multimedia capture devices are becoming increasingly popular to take pictures. Annotating these pictures is important to support their browsing and retrieval. Fully automatic image annotation techniques typically rely entirely on visual properties of the image. The state of the art image annotation systems of this kind work well in detecting generic object classes: car, horse, motorcycle, airplane, etc. However, certain characteristics of the image are hard to capture using strictly the visual properties. These include location (Paris, California, San Francisco, etc), event (birthday, wedding, graduation ceremony, etc), people (John, Jane, brother, etc) and abstract qualities referring to objects in the image (beautiful, funny, sweet, etc) among others. The more conventional method of annotation that relies completely on human input has several limitations as well. Typing tags using the keypads of such devices can be cumbersome and error-prone. Secondly, delay in tagging may result in a loss of context in which the picture was taken (e.g., user may not remember the names of the people/structures in the image). This presents an opportunity for using speech as a modality to annotate images and/or other multimedia content. Most camera devices have a built-in microphone. In principle, some of the challenges associated with both, fully automatic annotation as well as manual tagging can be alleviated if the user were to use speech as a medium of annotation. Ideally, the user would take a picture and speak the desired tags into the device s microphone. A speech recognizer would transcribe the audio signal into text. The speech to text transcription can happen either on the device itself or be done on a remote machine. The transcribed text can be used as tags for the image, exactly as the user intended. One of the biggest bottlenecks facing such systems is the accuracy of the underlying speech recognizer. Even speaker dependent recognition systems can make mistakes in noisy environments. If the recognizer s output is considered as is for annotation, then poor recognition will lead to poor quality tags. Our work tries to address this issue by incorporating outside semantic knowledge to improve interpretation of the recognizer s output, as opposed to blindly believing what the recognizer suggests. To improve interpretation of speech output, we exploit the fact that most speech recognizers provide alternate hypotheses for each utterance. The main contribution of this paper is our This research was supported by NSF Awards 0331707 and 0331690, and DHS Award EMW-2007-FP-02535. approach for annotating images using speech as the input modality. The approach employs a probabilistic model for computing the joint probability of a given combination of tags using a Maximum Entropy solution. The extensive empirical evaluation demonstrates the advantage of the proposed solution, that leads to a signi cant improvement of quality of speech annotation. II. PROBLEM DEFINITION We consider a setting wherein the user intends to annotate an image with a sequence G = (g1 , g2 , . . . , gK ) of K ground truth tags. Each tag gi can be either a single word or a short phrase of multiple words, such as Niagara Falls, Golden Gate Bridge, and so on. Since a tag is typically a single word, we will use tag and word interchangeably. A. N-Best Lists To accomplish the annotation task, the user speaks out each of the words gi for i = 1, 2, . . . , K. These K words are then processed by a speech recognizer. We assume that the recognizer is trained to recognize a delimiter between each of these K utterances. The recognizer s task is to correctly recognize these words so that they can then be assigned as tags to the image. However, noisy environments and unrestricted vocabularies can increase the recognizer s uncertainty in its hypotheses. The recognizer might propose several alternatives for each utterance of a word. Thus, the output of the recognizer is a sequence L = (L1 , L2 , . . . , LK ) of K N -best lists for the K utterances. Each N -best list Li = (wi1 , wi2 , . . . , wiN ) consists of N words that correspond to the recognizer s alternatives for word gi . Observe that list Li might not contain the ground truth word gi . The words in a N -best lists Li are typically output in a ranked order. Thus, when the recognizer has to commit to a single word for each utterance, it would set N = 1 and output (w11 , w21 , . . . , wK1 ) as its answer. While wi1 has the highest chance of being the correct word, in practice it may be the incorrect option. This presents the need for an approach that can smartly disambiguate between the alternatives. B. Answer Quality We noted earlier that each Li may or may not contain the ground truth. Let us de ne a sequence as a K-dimensional vector W = (w1 , w2 , . . . , wK ), where each wi is either an element from list Li or is equal to null, where wi = null encodes the fact that the algorithm believes that the list Li does L1 w11 =pain w12 =Jane w13 =lane w14 =game L2 w21 =prose w22 =nose w23 =rose w24 =crows L3 w31 =garden w32 =harden w33 =jordan w34 =pardon L4 w41 = ower w42 =power w43 =shower w44 =tower L5 w51 =sad w52 =wad w53 =bad w54 =dad TABLE I SAMPLE N -BEST LISTS L = (L1 , L2 , L3 , L4 , L5 ). the ground truth tag g5 = red. Therefore the maximum achievable precision is 1 and recall is 4 . Suppose some 5 approach is applied to this case, and its answer is A = (Jane, rose, garden, power, null), that is, it picked power instead of ower and thus only Jane , rose , and garden tags are correct. Then P recision(A) = 3 and Recall(A) = 4 3 5. III. USING SEMANTICS FOR DENSITY ESTIMATION Here we show how we compute the score of a sequence W = (w1 , w2 , . . . , wK ) as the joint probability P (w1 , w2 , . . . , wK ) for an image to be annotated with tags w1 , w2 , . . . , wK using the approach of Maximum Entropy (ME). This probability is inferred based on how a collection of images has been annotated in the past. The ME approach reduces the problem of computing P (w1 , w2 , . . . , wK ) to a constrained optimization problem. It allows us to compute joint probability P (w1 , w2 , . . . , wK ) based on only the values of known correlations in data. The approach hinges on the information theoretic notion of entropy [6]. For a probability distribution P = (p1 , p2 , . . . , pn ), where pi = 1, the n entropy H(P ) is computed as H(P ) = i=1 pi log pi and measures the uncertainty associated with P . Entropy H(P ) reaches its minimal value of zero in the most certain case where pi = 1 for some i and pj = 0 for all j = i. It reaches its maximal value in the most uncertain uniform case where 1 pi = n for i = 1, 2, . . . , n. We will use a support-based method to decide whether the probability can be estimated directly from data. Speci cally, if K = 1, or if K 2 and n(w1 , w2 , . . . , wK ) k, where k is a positive integer value of support, then there is suf cient support to estimate the joint probability directly from data and P (w1 , w2 , . . . , wK ) is computed using a frequency based maximum likelihood estimate along with Lidstone s estimation that assumes a uniform prior on unseen sequences [2] [4]. In particular, a support-based estimate would be P (w1 , w2 , . . . , wK ) = n(w1 ,w2 ,...,wK )+ . We will refer to such P (w1 , w2 , . . . , wK ) NI + |V | as known probabilities. Cases of P (w1 , w2 , . . . , wK ) where K 2 but n(w1 , w2 , . . . , wK ) < k do not have suf cient support. They will be handled by the ME approach. We will refer to them as unknown probabilities. To compute P (w1 , w2 , . . . , wK ) the ME approach considers the power set S of set {w1 , w2 , . . . , wK }, that is, the set of all its subsets. For instance, the power set of {w1 , w2 , w3 } is {{}, {w1 }, {w2 }, {w1 , w2 }, {w2 , w3 }, {w1 , w2 , w3 }}. We can observe that for some of the subsets S S the probability P (S) will be known and for some it will be unknown. Let T be the truth set, i.e., the set of subsets for which P (S) is known: T = {S S : P (S) is known}. The values of P (S), where S T , will be used to de ne the constraints for the constrained optimization problem. To compute P (w1 , w2 , . . . , wK ) the algorithm considers atomic annotation descriptions, which are tuples of length K, where the i-th element can be only either wi or wi . Here wi means tag wi is present in annotations and wi means wi is absent from them. For instance, description (w1 , w2 , w3 ) refers not contain gi . Now we can de ne the quality of a sequence A = (w1 , w2 , . . . , wK ) by adapting the standard IR metrics of precision, recall, and F-measure [1]. Namely, if |A| = 0 then P recision(A) = Recall(A) = 0. If |A| > 0 then P recision(A) = |A G| and Recall(A) = |A G| = |A G| , |A| |G| K where |A G| is the number of wi such that wi = gi . The F-measure is computed as the harmonic mean of the precision and recall. Thus, our goal can be viewed as that of designing an algorithm that produces high quality answer for any given L. Having de ned the quality of an answer, we can make several observations. First, for a given L the best answer sequence is the sequence A = (w1 , w2 , . . . , wK ) such that wi = gi if gi Li and wi = null if gi Li . Another related observation is that there is a theoretic upper bound on the achievable quality of any sequence A for a given L. Speci cally, assume that only M out of K N -best lists contain the ground truth tags, where M K. Then the maximum reachable value of |A G| is M . Thus, if M = 0 then for any answer A it follows that P recision(A) = Recall(A) = 0. If M > 0 then the maximum reachable precision is M = 1 and M maximum recall is M which is less than 1 when M < K. K C. Overall Algorithm We will consider the set WL = {W } of all N K possible answer sequences given L. On each such sequence W WL a score S(W ) will be assigned based on the joint distribution of the tags in W . The algorithm chooses a sequence W as its nal answer by selecting among all W WL a sequence with the maximum overall score W = argmaxW WL S(W ) and then applying a null detection procedure to W to compute the nal answer. D. Notational Example As an example, suppose that the user takes a picture of her friends Jane in a garden full of roses, and provides utterances the with K = 5 words: G = (g1 = Jane, g2 = rose, g3 = garden, g4 = ower, g5 = red). Then, the corresponding set of ve N-best lists for N = 4 could be as illustrated in Table I. If the recognizer has to commit to a single word per utterance, its output would be (pain, prose, garden, f lower, sad). That is, only garden and ower would be chosen correctly. This motivates the need for an approach that can disambiguate between the different alternatives in the list. Theoretically, the best possible answer would be (Jane, rose, garden, f lower, null). The last word is null since list L5 does not contain Known probabilities: 000 ( w1 , w2 , w3 ) 100 ( w1 , w2 , w3 ) 110 ( w1 , w2 , w3 ) 101 (w1, w2 , w3 ) P({}) =1.0 P(w1), P(w2), P(w3) P(w1, w2) P(w1, w3) P(w2, w3) 111 ( w1 , w2 , w3 ) 010 ( w1 , w2 , w3 ) 011 ( w1 , w2 , w3 ) 001 ( w1 , w2 , w3 ) Probability we need to compute: P(w1, w2, w3) Fig. 1. Probability Space. Solving it will give us the desired P (w1 , w2 , . . . , wK ) which corresponds to x11 1 . The constrained optimization problem can be solved ef ciently by the method of Lagrange multipliers to obtain a system of optimality equations. Since the entropy function is concave, the optimization problem has a unique solution [5]. We employ the variant of the iterative scaling algorithm used by [4] to solve the resulting system. A. Correlations to all image annotations where tags w1 and w2 are present and w3 is absent. Each such description can be encoded with a help of a bit string b, where 1 corresponds to wi and 0 to wi . For instance (w1 , w2 , w3 ) can be encoded as b = 110. Let AS be the atom set for S, de ned as the set of all possible bit strings of size K such that for each b AS it holds that if wi S then b[i] = 1, for i = 1, 2, . . . , K. For instance for K = 3 and S = {w1 , w2 } set AS = {110, 111}, whereas for K = 3 and S = {w2 } set AS = {010, 011, 110, 111}. Let xb denote the probability to observe an image annotated with the tags that correspond to bit string b of length K. Figure 1 illustrates the probability space with respect to all xb for the case where K = 3. Then in the context of ME approach our goal of determining P (w1 , w2 , . . . , wK ) reduces to solving the following constrained optimization problem: Maximize Z = b B xb log xb subject to (1) b AS xb = P (S) for all S T and xb 0 for all b We can extend the notion of direct correlations to that of indirect correlations. Observe that even when two words may never have co-occurred together in any image, they could still be correlated to each other through other words. For instance, the words beach and ocean may be indirectly correlated through the word sand. Suppose we start with a base correlation graph G = (V, E) whose nodes are tags in the vocabulary V . An edge is created per each pair of nodes wi and wj and labelled with the value of c(wi , wj ). The base correlation matrix B = B1 of G is a V V 2 matrix with elements Bij = c(wi , wj ). Let Pij be the set of all paths of length two in graph G from wi to wj . Then the indirect correlation c2 (wi , wj ) of length two for wi and wj is de ned 2 as the sum of contribution of each path (x0 x1 x2 ) Pij , where the contribution of each path is computed as the product of base similarities on its edges: 2 c2 (wi , wj ) = 2 (x0 x1 x2 ) Pij i=1 c(xi 1 , xi ). (3) It can be shown that the corresponding similarity matrix B2 can be computed as B2 = B 2 . The idea can be extended further by considering ck (wi , wj ) and demonstrating that Bk = B k . A similarity matrix A, that takes into account indirect similarities for k = 1, 2, . . . , m can be computed in a manner similar to that of diffusion kernels [7]. For instance, in the spirit of exponential diffusion kernels, A can be computed m m 1 as A = k=0 k! k B k , or, as A = k=0 k B k . A and Bk for k = 1, 2, . . . , m are performed off line before processing of image annotations starts. Therefore very fast computation of A is not critical. B. Detecting Nulls This section discusses how A can be utilized for detecting null candidates. That is, detecting the situation that a given N-best list Li is unlikely to contain the ground truth tag gi . First, we extend the notion of a base correlation graph G to that of indirect correlation graph Gind . Like in G, the nodes of Gind are the tags wi V , but each edge (wi , wj ) is now labelled with the value of Aij . Let W = (w1 , w2 , . . . , wK ) be the sequence with the highest score among all the possible N K sequences for a given sequence of N-best lists L. If list Li L does not contain the ground truth tag gi , then wi = gi . We can observe that when such situations occur, it is likely that wi will not be strongly correlated with the rest of the tags in W . We can now design the null detection procedure. It takes W = (w1 , w2 , . . . , wK ) as input and analyzes each wi W . If A(wi , wj ) < for j = 1, 2, . . . , K, j = i, and a threshold value , then wi is considered to be isolated in Gind , in terms of correlations, from the rest of the tags. Such isolated tags are substituted with null values. IV. EXPERIMENTS Dataset:. Our set of images was obtained by crawling a popular image hosting website, namely Flickr. We start off In this section we de ne the notion of the correlation c(wi , wj ) between any pair of words wi and wj . This will allow us to create a method for detecting null cases. Let the correlation c(wi , wj ) between two words wi and wj be de ned as the Jaccard similarity: c(wi , wj ) = n(wi ,wj ) n(wi )+n(wj ) n(wi ,wj ) 0 if n(wi , wj ) > 0; (2) if n(wi , wj ) = 0. Here, n(wi , wj ), n(wi ), and n(wj ) are the number of images whose annotation include, respectively, both the tags wi and wj , tag wi , and tag wj . The value c(wi , wj ) is always in [0, 1] interval. by downloading 60000 Flickr images with their ground truth annotations. We randomly set aside 20% of the data for testing (will be called Dtest ) and 80% for training (Dtrain ). We will use portions of Dtest for testing. The size of the vocabulary is |V | = 18285. We randomly picked 102 images from Dtest and annotated them (generating the N-best lists) using a popular commercial off-the-shelf recognizer Dragon v.8. We will call this annotated set Dtest . The annotations were performed in a Low noise level. Low noise level corresponds to a quiet university lab environment. All non-English words were removed before using Dragon to create these N best lists. Approaches. We will compare the results of three approaches: Baseline is the output of the recognizer (Dragon v.8). ME is the output of the proposed approach. Upper Bound is the theoretic upper bound achievable, see Section II. It depends on how many N-best lists contain the ground truth tag. 1 0.9 0.8 0.7 F measure 0.6 0.5 0.4 0.3 0.2 Low Baseline ME Upper bound constantly better than the F-measure of Baseline by at least 15% for all K. In the subsequent discussion we will refer to Dtest data test . with the Low level of noise as just D 1 F measure Upper bound on F 0.95 0.9 F measure 0.85 0.8 0.75 0.7 0.65 1 2 3 N 4 5 6 Fig. 3. F vs. N (Size of N -Best Lists) Experiment 2. (Quality vs Size of N-Best Lists.) Figure 3 illustrate the F-measure as a function of the size of N -best list N on Dtest data. For a given N , the N-best lists are generated by taking the original N-best lists from Dtest data and keeping at most N rst elements in them. Increasing N presents a tradeoff. As N increases, the greater is the chance that the ground truth element would appear in the list. At the same time, ME algorithm is faced with more uncertainty as there are more options to disambiguate between. The results demonstrate that the potential bene t from the former outweighs the potential loss due to the latter, as Fmeasure increases with N . V. CONCLUSION: In this paper, we have postulated the problem of using discrete speech utterances to annotate an image as that of disambiguation across multiple N -best lists. Our solution is based on the Maximum Entropy approach and uses correlations between tags in an existing corpus of images to set up the constrains of the corresponding constrained optimization problem. Our experiments suggest that the proposed approach gives a signi cant improvement in quality as compared to an approach that considers the best answer suggested by a popular off-the-shelf recognizer. REFERENCES [1] R. Bayeza-Yates and B. Riberto-Neto. Modern Information Retrieval. Addison-Wesley, 1999. [2] D. Jurafsky and J. Martin. Speech and Language Processing. PrenticeHall, 2000. [3] C. Manning and H. Schutze. Foundations of Statistical Natural Language Processing. MIT Press, 1999. [4] V. Markl, P. J. Haas, M. Kutsch, N. Megiddo, U. Srivastava, and T. M. Tran. Consistent selectivity estimation via maximum entropy. VLDB J., 16(1):55 76, 2007. [5] S. D. Pietra, V. J. D. Pietra, and J. D. Lafferty. Inducing features of random elds. IEEE Trans. Pattern Anal. Mach. Intell., 19(4), 1997. [6] C. E. Shannon. The Mathematical Theory of Communication. University of Illinois Press, 1949. [7] J. Shawe-Taylor and N. Cristianni. Kernel Methods for Pattern Analysis. Cambridge University Press, 2004. Medium Noise level High Fig. 2. F-measure vs. Noise. Experiment 1. (Quality for Various Noise Levels.) We randomly picked 20 images from Dtest and re-annotated them (i.e. created N best lists for ground truth tags) using Dragon in two additional noise levels: Medium and High. Medium and High levels were produced by introducing white Gaussian noise through a speaker.1 Figure 2 shows the F-measure of the three approaches for the Low, Medium, and High noise levels on these 20 images. Since we had created Dtest in a Low noise level on 102 images, for a fair comparison, the points corresponding to Low noise levels in the plots are averages over these 20 images, as opposed to the full 102 images. As anticipated, higher noise levels negatively affect performance of all three approaches. The performance of ME dominates Baseline performance for all the different noise levels. When we considered images that are annotated with exactly K tags, we found the performance of ME is consistent across different values of K. For instance, for Low noise level and for K = 2, 3, . . ., the F-measure of ME was consistently within 12% of F-measure for the Upper Bound. In addition, it was 1 To give a sense of the level of noise, High was a little louder than the typical volume of TV in a living room.

Find millions of documents here - Study Guides, Homework Solutions, Papers, Exam Answer Keys and more. Course Hero has millions of course related materials that will enable you to learn better, faster and get an A in all your courses.
Below is a small sample set of documents: