as Web Network: A Case Study
Networked Life CIS 112 Spring 2010 Prof. Michael Kearns
The Web as Network
C o ns id e rth e we b a s a ne two rk
ve rtic e s :ind ivid ua l(h tm l) p a g e s e d g e s :h y p e rlinks b e twe e np a g e s willvie wa s b o th a directedandundirectedgraph connectedcomponents degreedistributions etc.
Whatisthestructureofthisnetwork?
Whatdoesitsayaboutthepeoplebuildingandusingit? Whatarethealgorithmicconsequences?
websearch communityidentification pageandlinkgeneration visitationstatistics
Graph Structure in the Web
[Broder et al. paper]
R e p o rto nth e re s ults o ftwo m a s s ive we b c ra wls Exe c ute d b y Alta Vis ta inMa ya nd O c to b e r1 9 9 9 De ta ils o fth e c ra wls :
May99crawl: Oct99crawl:
a uto m a te d s c rip tfo llo wing h y p e rlinks (UR Ls ) fro m p a g e s fo und la rg e s e to fs ta rting p o ints c o lle c te d o ve rtim e c ra wlim p le m e nte d a s breadthfirstsearch havetodealwithwebspam,infinitepaths,timeouts,duplicates,etc. 200millionpages,1.5billionlinks 271millionpages,2.1billionlinks
Unaudited,selfreportedSep03stats:
3majorsearchenginesclaim>3billionpagesindexed
Five Easy Pieces
Auth o rs d id two kind s o fb re a d th firs ts e a rc h : Theythenidentifyfivedifferentregionsoftheweb:
stronglyconnectedcomponent(SCC): componentIN:
canreachanypageinSCCfromanyotherindirectedfashion canreachanypageinSCCindirectedfashion,butnotreverse canbereachedfromanypageinSCC,butnotreverse weaklyconnectedtoalloftheabove,butcannotreachSCCorbereachedfromSCCindirected fashion(e.g.pointedtobyIN)
ig no ring linkd ire c tio n weakconnectivity onlyfollowingforwardlinksstrongconnectivity
componentOUT:
componentTENDRILS:
SCC+IN+OUT+TENDRILSformweaklyconnectedcomponent(WCC) everythingelseiscalledDISC(disconnectedfromtheabove) hereisavisualizationofthisstructure
Size of the Five
S C C :~5 6 Mp a g e s ,~2 8 % IN:~4 3 Mp a g e s ,~2 1 % O UT :~4 3 Mp a g e s ,~2 1 % T ENDR ILS :~4 4 Mp a g e s ,~2 2 % DIS C :~1 7 Mp a g e s ,~8 % WC C >9 1 % o fth e we b th e giantcomponent Oneinterpretationofthepieces:
SCC:theheartoftheweb IN:newersitesnotyetdiscoveredandlinkedto OUT:insularpageslikecorporatewebsites
Diameter Measurements
Dire c te d wo rs tc a s e d ia m e te ro fth e S C C : Dire c te d wo rs tc a s e d ia m e te ro fIN S C C O UT : Over75%ofthetime,thereisnodirectedpathbetweenarandomstartand finishpageintheWCC AverageundirecteddistanceintheWCCis7 webisasmallworldwhenweignoredirection Moral:
otherwisethepictureismorecomplex whenthereisadirectedpath,averagelengthis16 a tle a s t5 0 3 a tle a s t2 8
Degree Distributions
T h e y a re ,o fc o urs e , heavytailed P o we rla wd is trib utio no f componentsize Undirectedconnectivityofwebnotreliantonconnectors
whathappensasweremovehighdegreevertices? consistentwiththeErdosRenyimodel
Here is a 2005 update on all this stuff.
Exploiting Web Structure: Google and PageRank
The PageRank Algorithm
Le ts d e fine a m e a s ure o fp a g e im p o rta nc e we willc a llth e rank Notation:foranypagep,let Idea:importantpagesdistributeimportanceovertheirforwardlinks Sowemighttrydefining
R(p):=sumofR(q)/N(q)overallpagesqp candefineiterativealgorithmforcomputingtheR(p) (ifitconverges,solutionhasaneigenvectorinterpretation) problem:cyclesaccumulaterankbutneverdistributeit N(p)bethenumberofforwardlinks(pagesppointsto) R(p)bethe(tobedefined)rankofp
Thefix:
LetsplaywiththePageRankcalculator
R(p):=[sumofR(q)/N(q)overallpagesqp]+E(p) E(p)issomeexternalorexogenousmeasureofimportance sometechnicaldetailsomittedhere(e.g.normalization)
The Random Surfer Model
Le ts s up p o s e th a tE(p ) s um s to 1 (no rm a lize d ) T h e nth e re s ulting P a g e R a nks o lutio nR (p ) will R (p ) is th e stationarydistributionofthefollowingprocess:
startingfromsomerandompage,justkeepfollowingrandomlinks ifstuckinaloop,jumptoarandompagedrawnaccordingtoE(p) sosurferperiodicallygetsboredandjumpstoanewpage E(p)canthusbepersonalizedforeachsurfer a ls o b e no rm a lize d c a nb e inte rp re te d a s a p ro b a b ility d is trib utio n
PageRankone(important)componentGooglessearchtech
dontforgetthewords! informationretrievalandstatisticallanguageprocessing
Looking Ahead: Left Side vs. Right Side
SofarwearediscussingthelefthandsearchresultsonGoogle Righthandorsponsoredsearch:paidadvertisementsinaformalmarket Sametwotypesofsearch/resultsonYahoo!,MSN, Commonperception:
Wewillspendalectureonthesemarketslaterintheterm a.k.aorganicsearch;
Butbothsidesaresubjecttogaming(strategicbehavior)
organicresultsareobjective,basedoncontent,importance,etc. sponsoredresultsaresubjectiveadvertisements organic:invisibletermsinthehtml,linkfarmsandwebspam,reverseengineering sponsored:biddingbehavior,jamming optimizationofeachsidehasitsownindustry:SEOandSEM organic:typosquatting sponsored:clickfraud
andperhapstooutrightfraud Morelater