11-LangModel

11-LangModel - Language Model CS273 - Data and Knowledge...

Info iconThis preview shows pages 1–9. Sign up to view the full content.

View Full Document Right Arrow Icon
Language Model S273 ata and Knowledge Bases CS273 - Data and Knowledge Bases Xifeng Yan Computer Science niversity of California at Santa Barbara University of California at Santa Barbara
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Department of Computer Science [Readings] Study of Smoothing Methods for Language Models Applied to 1. A Study of Smoothing Methods for Language Models Applied to Ad Hoc Information Retrieval. ChengXiang Zhai, John D. Lafferty, SIGIR 2001 Data and Knowledge Bases | University of California at Santa Barbara 2
Background image of page 2
Department of Computer Science Generative Models: P(D|Q,R) and P(Q|D,R) Basic idea Compute O(R=1|Q,D) using Bayes’ rule ) 1 ( ) 1 | , ( ) , | 1 ( ) , | 1 ( R P R D Q P D Q R P D Q R O Ignored for ranking D Define P(Q,D|R) pecial cases ) 0 ( ) 0 | , ( ) , | 0 ( R P R D Q P D Q R P Special cases Document “generation”: P(Q,D|R)=P(D|Q,R)P(Q|R) (not covered in this lecture) Query “generation”: P(Q,D|R)=P(Q|D,R)P(D|R) lides adapted from ChengXiang Zhai Data and Knowledge Bases | University of California at Santa Barbara 3 Slides adapted from ChengXiang Zhai
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Department of Computer Science Document Generation ) 0 | , ( ) 1 | , ( ) , | 0 ( ) , | 1 ( R D Q P R D Q P D Q R P D Q R P ) 1 , | ( ) 0 | ( ) 0 , | ( ) 1 | ( ) 1 , | ( R Q D P R Q P R Q D P R Q P R Q D P Model of relevant docs for Q ) 0 , | ( R Q D P Model of non-relevant docs for Q Data and Knowledge Bases | University of California at Santa Barbara 4
Background image of page 4
Department of Computer Science The Simplest Language Model (Unigram Model) Generate a piece of text by generating each word DEPENDENTLY INDEPENDENTLY Thus, p(w 1 w 2 ... w n )=p(w 1 )p(w 2 )…p(w n ) Parameters: {p(w i )} p(w 1 )+…+p(w N )=1 (N is voc. size) Essentially a multinomial distribution over words A piece of text can be regarded as a sample rawn according to this word distribution drawn according to this word distribution Data and Knowledge Bases | University of California at Santa Barbara 5
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Department of Computer Science Text Generation with Unigram LM (Unigram) Language Model p(w| ) Document Sampling text 0.2 mining 0.1 socation 0 01 Text mining assocation 0.01 clustering 0.02 food 0.00001 Topic 1: Text mining paper n d | | food 0.25 opic 2: ood nutrition i w c i w c w c w P d P n 1 ) ( ) ( ),. .. ( 1 1 ) | ( ) | ( nutrition 0.1 healthy 0.05 diet 0.02 Topic 2: Health Food nutrition paper Data and Knowledge Bases | University of California at Santa Barbara 6
Background image of page 6
Department of Computer Science Estimation of Unigram LM (Unigram) Language Model ocument Estimation p(w| )=? Document text 10 mining 5 association 3 database 3 gorithm 2 text ? mining ? assocation ? database ? 10/100 5/100 3/100 3/100 algorithm 2 query 1 efficient 1 query ? 1/100 A “text mining paper” Data and Knowledge Bases | University of California at Santa Barbara 7 (total #words=100)
Background image of page 7

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Department of Computer Science Maximum Likelihood Estimate Data: a document d with counts c(w 1 ), …, c(w N ), and length |d| Model: multinomial (unigram) with parameters {p(w i )} Likelihood: p(d| ) Maximum likelihood estimator: =argmax p(d| ) | | d N N log ) ( ) | ( log ) | ( ) ( , ) ( )... ( ) | ( 1 ) ( 1 ) ( 1
Background image of page 8
Image of page 9
This is the end of the preview. Sign up to access the rest of the document.

Page1 / 24

11-LangModel - Language Model CS273 - Data and Knowledge...

This preview shows document pages 1 - 9. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online