M H M - B a s e d Word Alignment in Statistical Translation
Stephan Vogel Hermann Ney Christoph Tillmann L e h r s t u h l ffir I n f o r m a t i k V, R W T H A a c h e n D-52056 Aachen, Germany {vogel, n e y , t illmann}@inf ormat ik. rwth-aachen, de
Abstract
In this paper, we describe a new model for word alignment in statistical translation and present experimental results. The idea of the model is to make the alignment probabilities dependent on the differences in the alignment positions rather than on the absolute positions. To achieve this goal, the approach uses a first-order Hidden Markov model (HMM) for the word alignment problem as they are used successfully in speech recognition for the time alignment problem. The difference to the time alignment HMM is that there is no monotony constraint for the possible word orderings. We describe the details of the model and test the model on several bilingual corpora.
2
Review: Translation M o d e l
The goal is the translation of a text given in some language F into a target language E. For convenience, we choose for the following exposition as language pair French and English, i.e. we are given a French string f~ = fx ...fj...fJ, which is to be translated into an English string e / = el...ei...cl. Among all possible English strings, we will choose the one with the highest probability which is given by Bayes' decision rule:
a{ =
=
argmax{P,.(c{lAa)} q
a r g m a x {Pr(ejt) . l ' r ( f l e [ ) } el ~
1
Introduction
Pr(e{) is the language model of the target language, whereas Pr(fJle{) is the string translation model. The argmax operation denotes the search problem. In this paper, we address the problem of introducing structures into the probabilistic dependencies in order to model the string translation probability Pr(f~ le{).
3 Alignment Models
A key issne in modeling the string translation probability Pr(J'~le I) is the question of how we define the correspondence between the words of the English sentence and the words of the French sentence. In typical cases, we can assume a sort of pairwise dependence by considering all word pairs (fj, ei) for a given sentence pair I.-/1[~'J', We furelqlj' ther constrain this model by assigning each French word to exactly one English word. Models describing these types of dependencies are referred to as
In this paper, we address the problem of word alignments for a bilingual corpus. In the recent years, there have been a number of papers considering this or similar problems: (Brown et al., 1990), (Dagan et al., 1993), (Kay et al., 1993), (Fung et al., 1993). In our approach, we use a first-order Hidden Markov model (HMM) (aelinek, 1976), which is similar, but not identical to those used in speech recognition. The key component of this approach is to make the alignment probabilities dependent not on the absolute position of the word alignment, but on its relative position; i.e. we consider the differences in the index of the word positions rather than the index itself. The organization of the paper is as follows. After reviewing the statistical approach to machine translation, we first describe the conventional model (mixture model). We then present our first-order HMM approach in lull detail. Finally we present some experimental results and compare our model with the conventional model.
alignment models.
In this section, we describe two models for word alignrnent in detail: ,. a mixture-based alignment model, which was introduced in (Brown et al., 1990); an HMM-based alignment model. In this paper, we address the question of how to define specific models for the alignment probabilities. The notational convention will be as follows. We use the symbol Pr(.) to denote general
836
probability distributions with (nearly) no Sl)eeitic asSUml)tions. In contrast, for modcl-t)ased prol)-ability distributions, we use the generic symbol
v(.).
Alignment with Mixture Distri|mtion Here, we describe the mixture-based alignment model in a fornmlation which is different fronl the original formulation ill ( B r o w n el, a[., 1990). We will ,is(: this model as reference tbr the IIMMbased alignments to lie 1)resented later. The model is based on a decomposition of the joint probability [br ,l'~ into a product over the probabilities for each word J):
3.1 a j=l wheFe~ fo[' norll-la]iz;i,t i o n 17(~/SOllS~ the 8elltC][ce
For unilbrm alignment probabilities, it can be shown (Brown et al., 1990), that there is only one optinnnn and therefore the I,',M algorithm (Baum, 1!)72) always tinds the global optimum. For mixture alignment model with nonunilbrm alignment probabilities (subsequently referred to as IBM2 model), there ~tre to() many alignrnent parameters Pill j, I) to be estimated for smMl c o l pora. Therefore, a specific model tbr tile Mignment in:obabilities is used: p ( i l j , 1) = l
r(i-j~-)
.I
(~)
Ei':l
"( it --" J J-)
length probability p(J] l) has been included. The next step now is to assutne a sort O['l,airwise interact, ion between tim French word fj an(l each, F,nglish word ci, i = 1, ...l. These dep('ndencies are captured in the lbrm of a rnixtnre distritmtion:
1
This model assumes that the position distance relative to the diagonal line of the (j, i) plane is the dominating factor (see Fig. 1). 'lb train this model, we use the ,naximutn likelihood criterion in the so-called ulaximmn al)proximation, i.e. the likelihood criterion covers only tile most lik(-.ly align: inch, rather than the set of all alignm(,nts:
d
P,'(f(I,:I) ~ II ~"IUHO, ~)v(J} I,:~)]
j=l
(a)
p(J)le{)
=
~_.p(i, fjlc I)
i=1
I
In training, this criterion amounts to a sequence of iterations, each of which consists of two steps:
* posilion alignmcnl: (riven the model parameters, deLerlniim the mosL likely position align]lient.
= ~_~p(ilj, l).p(fjle~)
i=1
Putting everything together, we have the following mixture-based ntodel:
J l
r,'(fi!l~I) = p(JIO ' H ~_~ [~,(ilJ, l). ~,(j)led] (1)
j = l i=t
with the following ingredients: sentence length prob~d)ility: P(Jll); mixture alignment probability: p ( i l j , I); translation probM)ility: p(f[e). Assuming a tmifornl ~flignment prol)ability
1
paramc, lcr cstimalion: Given the position alignment, i.e. goiug along the alignment paths for all sentence pairs, perform maxitnulu likelihood estimation of the model parameters; for model-De(' distributions, these estimates result in relative frequencies.
.p(ilj, 1) = 7
l)ue to the natnre of tile nfixture tnod(:l, there is no interaction between d j a c e n t word positions. Theretbre, the optimal position i for each position j can be determined in(lependently of the neighbouring positions. Thus l.he resulting training procedure is straightforward.
a.2
we arrive at the lh'st model proposed t)y (Brown et al., 1990). This model will be referred to as IB M 1 model. To train the translation probabilities p(J'fc), we use a bilingual (;orpus consisting of sentence pairs
Alignment with HMM
[:/';4"1 : ', . , s Using the ,,laxin,ul , likelihood criterion, we ol)tain the following iterative
L a
equation (Brown et al., 1990):
/)(fie) = ~ $'
will,
A(f,e) = ~
2 ~5(f,J).~) }~ a(e,e~.~)
We now propose all HMM-based alignment model. '['he motivation is that typicMly we have a strong localization effect in aligning the words in parallel texts (for language pairs fi:om ]ndoeuropean languages): the words are not distrilmted arbitrarily over the senteuce ])ositions, but tend to form clusters. Fig. 1 illustrates this effect for the language pair G e r m a n - 15'nglish. Each word of the German sentence is assigned to a word of the English sentence. The alignments have a strong tendency to preserve the local neighborhood when going from the one langnage to the other language. In mm,y cases, although not al~ ways, there is an even stronger restriction: the differeuce in the position index is smMler than 3.
837
DAYS BOTH ON
EIGHT
+ + + + + + + + +j~ + + + + + + + + + ~J ~+ +
+++++++/++.-.
+ + + + + + +/+ + + + + + + + + + ~x~ + + + + +
+ + + + +/+ D + + + + +
AT IT MAKE CAN WE IF THINK
I
where, in addition, we have assmned that tile translation probability del)ends only oil aj and not oil aj-:l. Putting everything together, we have the ibllowing llMM-based model:
a
++ + + ~ + + + _~ + + +~ + + +jg + + +~ +++
g + + ++
+ + + + + + + + + + + + + + + ++ ++++ + + + + + + + + + + + + + +
+ ++ + + + +
Pr(f:i'le{) = ~ I-I [p(ajlaj - ' , l).p(Y)lea,)] af J=, with the following ingredients: IlMM alignment probability:
p ( a j l a j _ l , I);
(4)
WELL
p(i]i', I) or
z
aa
Figure 1: Word alignment for a G e r m a n - English sentence pair. To describe these word-by-word aligmnents, we introduce the mapping j ---+ aj, which assigns a word fj in position j to a word el in position { = aj. The concept of these alignments is similar to the ones introduced by (Brown et al., 1990), but we wilt use another type of dependence in the probability distributions. Looking at such alignments produced by a hmnan expert, it is evident that the mathematical model should try to capture the strong dependence of aj on the previous aligmnent. Therefore the probability of alignment aj for position j should have a dependence on the previous alignment aj _ 1 :
translation probabflity: p(f]e). In addition, we assume that the t{MM alignment probabilities p(i[i', [) depend only on the jump width (i - i'). Using a set of non-negative parameters { s ( i - i')}, we can write the IIMM alignment probabilities in the form: 4i-
p(ili', i) = E '
1=1
i') s(1 - i')
(5)
This form ensures that for each word position i', i' = 1, ..., I, the ItMM alignment probabilities satisfy the normMization constraint. Note the similarity between Equations (2) and (5). The mixtm;e model can be interpreted as a zeroth-order model in contrast to the first-order tlMM model. As with the IBM2 model, we use again the maximum approximation:
J
Pr(fiSle~) "~ max]--[
a' /
.ll. j,,,
[p(asl<*j-1, z)p(fjl<~,)]
task of finding the
(6)
j=l
p(ajiaj_l,i)
,
In this case,
the
optimal
where we have inchided the conditioning on the total length [ of the English sentence for normalization reasons. A sinfilar approach has been chosen by (Da.gan et al., 1993). Thus the problem formulation is similar to that of the time alignment problem in speech recognition, where the so-called IIidden Markov models have been successfully used for a long time (Jelinek, 1976). Using the same basic principles, we can rewrite the probability by introducing the 'hidden' alignments af := al...aj...aa for a sentence pair If,a; e{]:
alignment is more involved than in the case of the mixture model (lBM2). Thereibre, we have to resort to dynainic programming for which we have the following typical reeursion formula:
Q(i, j) = p(fj lel) ,nvax [p(ili', 1). Q(i', j - 1)]
i =l,.,,I
Here, Q(i, j) is a sort of partial probability as in time alignment for speech recognition (Jelinek,
197@.
4 Experimental Results
4.1 T h e T a s k a n d t h e C o r p u s The models were tested on several tasks: the Avalanche Bulletins published by the Swiss Federal Institute for Snow and Avalanche Research (SHSAR) in Davos, Switzerland and made awtilable by the Eup "q I ropean Corpus Initiative (I,CI/MCI, 1994); the Verbmobil Corpus consisting of spontaneously spoken dialogs in the domain of appointment scheduling (Wahlster, 1993);
Pr(f~al es) = ~_,Vr(fal, aT[ eI't,
a7
,1
= ~ 1-IP"(k,"stfT-',"{ -*,e/)
a I j=l So far there has been no basic restriction of the approach. We now assume a first-order dependence on the alignments aj only:
Vr(fj,aslf{ a -~, J-* , e l ) I
838
,, the E u T r a n s C,orpus which contains tyl)ical phrases from the tourists and t.ravel docnain. (EuTrans, 1996). 'l'able ] gives the details on the size of tit<; corp o r a a,ud t;]t<'it' vocal>ulary. It shottld I>e noted that in a.ll thes(; three ca.ses the ratio el' vocal)t,]ary size a.ml numl)er of running words is not very faw)rable.
Tall)le, I: (,orpol :L
(,o~pt s AvalancJte
] A[ [ r a i l s
l,angua.ge
Frolt ch
(~('~
l
lall
Spanish I,;nglish
( l e 11 a n
Voc. Size 1993 2265 --1:77@2008
15888 t 63(}
Words 62849 ,]4805
is the' globa.l o p t i m i z a t i o n criterion, the tables also show the perplexities of the translation probabilities and of the alignment probabilities. T h e last line in Table 2 gives the perplexity measures wh(m a.lJplying the rtlaxilnun| a p p r o x i m a t i o n and COml>uting the perph'~xity in t;]lis a p p r o x i m a t i o n . These values are equal to the ones after initializing the IBM2 and HMM models, as they should be. From Ta,ble 3, we can see. that the m i x t u r e alignment gives slightly better perplexity values for the translation l)roba.1)ilities, whereas the I I M M model produces a smaller perplexity for the alignment l>rohal)ilities. In the calculatiot, of the, perplexities, th<' seld;en(;e length probal)ility was not in= eluded. Tahle 2: IBM I: Translation, a,ligmnent and total pert)h'~xil.y as a. fimction of' the iteration. Iteration
0 1
Verlmlobil
150279
English
25,] 27
dO 17 2`]/13
For several years 1)et;weeu 83 and !)2, the Avalanche Bulletins are awdlabte for I>oth Getntan and I!'ren(;]l. T h e following is a tyl)ical sen-t<;nce t>air fS;onl the <;or:IreS: Bei zu('.rst recht holnm, Sl)~i.tev tM'eren 'l'eml)eraJ, uren sind vou Samsta.g his 1)ienstag t n o f gett auf <l<'~t; All>ennor(ls<'.ite un</ am All>en-. ha.uptkanml oberhalb 2000 m 60 his 80 cm Neuschnee gel'aJlen. l)ar des temp&'atures d' abord dlevdes, puis plus basses, 60 h 8(1 cm de neige sent tombs de samedi h. m a r d i matin sur le versant herd el; la eft're des Alpes au-dessus de 2000 l[1. An exa,nq)le fi'om the Vet%mobil corpus is given in Figure 1. 4.2 Training and ILesults
2
9
Tra,nslatiotl. 99.36 3.72 2.67
t.87 1.86
Alignrnent 20.07
20.07
20.07 20.07
20.07
Total 1994.00 7/1.57 53.62 37.55 37.36 77.!)5
10 Max.
3.88
20.07
'l'able 3:'1 rans] ~+tion, aligmn en t and totaJ perplexity as a function of the itcra.tion for the IBM2 (A) and the IIMM model (13) Iter.
A 0 l 2
Tratmlat;i(m 3.883.17
3.25 3.22
A ligniN.elJ t 20.07 10.82
10.15 10.10
'l'otal
l,;ach of the three COrlJora. were ttsed to train 1)oth alignnmnt models, the mixture-I>ased alignment model in Eq.(1) and the llMM-base<l a.lignntent mod('l in Eq.(d). ltere, we will consider the exp<'.rimenta.l tesl;s on tit<'. Avalanche corpus in more detail. T h e traii, ing procedure consiste(l of the following steps: , Initialization training: IBMI model trahted for t0 iterations of the i';M algorithm. ,, l{,efinement traiuiug: T h e translation pcoba1)ilities Dotn the initialization training wet'(; use+d to initialize both the IBM2 model and the I I M M-based nligntnent mo<t<'+l IBM2 ilnum IIMM imum Model: 5 iteratious using Lit(" m a x a.I)proximatiolt (Eq+(3)) Model: 5 iterations usiug l l e max-. al)l)roximation (Fq.(6))
3 ,]
1~ 5 0
3.20 3.18
3.88
] 0.06 10.05
20.07
1 3
4 5
3.37 3.46 ;{./17
"Ld6
3.`]5
7.99 6.17 5.90 5.85 5.8,]
77.95 34.27 33.03 32.48 32.18 32.00 77.95 26.98 2t.36 20.48 20.2/1 20.18
'l'h(, resulting perl>h:'~xity (inverse g<~olu(;l.ric avera,ge of the likelihoods) for the dilferent lno(lels ave given iu tim Tal>[es 2 and 3 for the Awdanehe <:<)rims. In adclitiou t;o the total i>erl>lexity, whi<'.h
Anoth<2r inl;crc:sting question is whether the IIMM alignntent model helps in finding good and sharply fo('usscd word+to-word (-orres]Jondences. As an (;xamf,1o, Table 4 gives a COmlm+rison of the translatioJ~ probabilities p(fl e) bctweett the mixture and the IIMM alignnw+nt model For the (,e, u +l word Alpensiidhang. T h e counts of the words a.re given in brackets. The, re is virLually no ,:lilfc~rc~nce between the translation l.al>les for the two nn)dels (1BM2 and IIMM). But+ itt general, the tl M M model seems to giw'. slightly better resuits in the cases of (;, ttna t COml+olmd words like Alpcus'iidha'n,(I vcrsant sud des Alpes which require ['u,tction words in the trattslation.
839
Table 4: Alpens/idhang. IBM1 Alpes (684) des (1968) le (1419) sud (416) sur (769) versant (431) Alpes (684) sud (41.6) versant (431) Alpes (684) des (1968) sud (416) versant (431) 0.171 0.035 0.039 0.427 0.040 0.284 0.276 0.371 0.356 0.284 0.028 0.354 0.333
IBM2
The Verbmobil Corpus consists of spontaneously spoken dialogs in the domain of appointment scheduling. The assumption that every word in the source language is aligned to a word in the target language breaks down for many sentence pairs, resulting in poor alignment. This in turn affects the quality of the translation probabilities. Several extensions to the current IIMM based model could be used to tackle these problems: * The results presented here did not use the concept of the empty word. For the HMMbased model this, however, requires a secondorder rather than a first-order model. . We could allow for multi-word phrases in both languages. In addition to the absolute or relative alignment positions, the alignment probabilities can be assumed to depend on part of speech tags or on the words themselves. (confer model 4 in (Brown et al., 1990)).
HMM
This is a result of the smoother position alignments produced by the HMM model. A pronounced example is given in Figure 2. 'She problem of the absolute position alignment can he demonstrated at the positions (a) and (c): both Schneebretlgefahr und Schneeverfrachtungen have a high probability on neige. The IBM2 models chooses the position near the diagonal, as this is the one with the higher probability. Again, Schneebrettgefahr generates de which explains the wrong alignment near the diagonal in (c). However, this strength of the HMM model can also be a weakness as in the case of est developpe ist ... entstanden (see (b) in Figure 2. The required two large jumps are correctly found by the mixture model, but not by the HMM model. These cases suggest an extention to the HMM model. In general, there are only a small number of big jumps in the position alignments in a given sentence pair. Therefore a model could be useful that distinguishes between local and big jumps. The models have also been tested on the Verbmobil Translation Corpus as well as on a small Corpus used in the EuTrans project. The sentences in the EuTrans corpus are in general short phrases with simple grammatical structures. However, the training corpus is very small and the produced alignments are generally of poor quality. There is no marked difference...