9, Class March 30th New Style Web Searching
Quality
Varies widely Relevance: not enough Value of page = high quality + relevance
1
Quality Ranking
Goal: order the answer to a query in decreasing order of value
Query independent: assign an intrinsic value to a document independently of the query. E.g., length, vocabulary, publication date, number of citations (in-degree), who publishes it, etc. Query dependent: value is determined with respect to a particular query. E.g. dot product Mixed
Connectivity Analysis
Mine hyperlink info on the Web Assumption:
Links often connect related pages Links between pages is a recommendation
E.g., co-citation matrix
Graph:
Pages - nodes Link - directed arcs (u,v) Page u contains hyperlink to v
2
Web Search - "Traditional"
Web Query
List of URLs Vector Based Model
word count weights
Web Search: "New Style"
Web is seen as a graph Links (as established by <A HREF= "url"> text </A>) are important Two independent but similar approaches:
PageRank: the basis for Google by Brin and Page (Stanford students) HITS: Kleinberg (MIT, IBM, and Cornell)
3
Web Search - "New style"
Web List of URLs Page Rank
Query
word count weights+type hit
Vector Based Model
List of modifiedU RLs
Page Rank: Back links/examples
4
Page Rank (2) - Example
PageRank
PageRank
Adds an objective measure of citation importance to backlinks that reference a site Can be described as model of user behavior
Assume "random given surfer" a web page at random and keeps clicking on links User never hits back and eventually gets bored and requests another random page Probability that the surfer will visit a page is PageRank (PR) Probability that the surfer will request a random page is (1-d)
5
PageRank (cont.)
PageRank is defined as follows:
PageRank
We assume page A has pages T1...Tn which point to it (i.e., are citations). The parameter d is a damping factor which can be set between 0 and 1 (typically 0.85). Also, C(A) is defined as the number of links going out of page A. The PageRank of a page A is given as follows: PR(A) = (1-d) + d (PR(T1)/C(T1) + ... + PR(Tn)/C(Tn)) PageRanks form a probability distribution over web pages, so the sum of all web pages' PageRanks will be one.
PageRank (cont.)
PageRank
Pages have a high PageRank if they have a lot of sites pointing to them Pages can also have a high P...