Course Hero - We put you ahead of the curve!

You have requested the below document.

Coursehero >> New York >> Cornell >> CS 614

Systems Publish-Subscribe Ken Birman Many slides from Aseem Bajaj (2004) Publish-Subscribe Often called event notification Producer publishes messages Consumer expresses interest by subscribing to various topics Platform accepts messages ( events ), finds matching subscribers, does upcall to deliver Styles of pub-sub Multicast TIBCO or older Isis news platform Web Services as a callback infrastructure Amazon at one point ran all communication over a pub-sub bus, but found this unstable People like location-independent addressing For routing or filtering in data centers Often done in hardware, for load-balancing or to deliver requests to the right server Origins Zwaenepoel/Cheriton (V system) Mention idea in connection with group communication ISIS Project Process groups & group communication ISIS Toolkit, 1987 included news system The Information Bus An Architecture for Extensible Distributed Systems, 1993 Tibco Background (cont.) Gryphon Project, IBM Matching Events in Content-based Subscription System, 1999 Enterprise Middleware Design of Wide Area Event Service, 1998 Mesh based Content Routing using XML, 2001 Siena Project, Univ of Colorado XML Event Routing Cayuga Event Filter Cornell (Gehrke, Demers) Issues Design of the name space Implementation of matching, routing, low-level protocols Choice of information spaces Complexity of subscriptions Performance Can application retrieve past history ? Does the pub-sub system offer any kind of real semantics? What kinds of failures can arise? How are they handled? Application-visible programming model Information Bus Suggests that publish subscribe is a powerful model for distributed systems Introduces a framework around the information bus: types, classes, objects, services Shows how to use such a bus to build distributed applications Introduces Anonymous Communication & Subject Based Addressing Content-based Subscription System Assumes publish-subscribe infrastructure But rather than limiting matching to topics goes further and allows queries against the actual content of messages Problem becomes one of matching at high speeds TIBCO paper: Benefits of pub-sub Decouples publisher from subscriber Easy to add new kinds of subscribers or upgrade components hence allows continuous operations No system downtime for upgrades or maintenance Start with something simple, then build up over time A kind of plug-and-play integration of new components Dynamic System Evolution Extensible Distributed Systems: Principles Minimal Core Semantics Communication system makes least possible assumptions about the application ( weak semantics ) Objects support queries about meta-information like type, attribute names & types, operation signatures Introduction of classes at runtime supported by TDL, a small interpreted language Subject Based Addressing. Messages sent and received by subject rather than identities. Self-Describing Objects Dynamic Classing Anonymous Communication This turns out to pose serious problems for This turns out to pose serious problems for some applications. is that some applications . IssueWhy? some publishers may actually have a specific subscriber in mind and yet there is no way to sense the error case where the subscriber isn t actually running or the bus drops the message Anonymous Communication Subject Based Addressing Publisher produces content without knowing the consumer, labels the content with hierarchically structured subject like news.equity.YHOO Consumer accepts content based on the Content Subscription can be wild carded Subscriber can be introduced anytime, starts consuming Publisher can be introduced anytime, start publishing System evolution Architecture Types are like interfaces Classes implement types Objects are instances of classes Service Objects Encapsulate & control access to system resources e.g. database system, print service Cannot be transferred to nodes other than where they reside, invoked from their location using some kind of RPC Bus Architecture Implementation Details Local Area Networks Each node has a daemon running Applications register, place subscriptions on daemon Ethernet broadcasts (unreliable) Daemon gets all messages on Ethernet, forwards to applications based on subscriptions Application Level Information Routers Routers receive messages by placing subscriptions Pass on messages to other routers that then get re-published on another bus . Messages only republished on buses that have subscriptions for that subject Wide Area Networks Reliability How does this happen? During load surges, network or receiver nodes can become lossy, triggering a further surge of retransmit When all is working normally (nobody crashes and requests and retransmissions. Effective goodput network is reasonably reliable) rate collapses to perhaps 5%. Especially common Message deliveredusing Ethernet broadcastonce to subscriber exactly (a broadcast storm ) After 90 Either sender-receiver seconds,or long-term network error no crash TIBCO gives up (assumes receiver has crashed) and stops trying. Sender sees partition indication. Order maintained for same sender, not multiple Message delivered to subscriber at most once When network becomes severely overloaded Big users observe strange phenomenon: 90 seconds of complete shutdown of TIBCO, during which all messages vanish, then restart of services and this repeats. Asserted semantics Guaranteed Message Delivery Message stored before sending Publisher retransmits unless acknowledged Message delivered to subscriber at least once Question: Does TIBCO achieve these? Dynamic Discovery & Remote Method Invocation (Who s out there?) Dynamic Discovery (I am) RMI Brokerage Trading Floor Brokerage Trading Floor Introduce Keyword Generator Subscribes and accepts stories Publishes keywords as property objects Monitors interprets & displays the property objects Latency Sun SPARCstation 2s with 24MB RAM, Sun IPXs with 48MB RAM Lightly loaded 10Mbps Ethernet 15 nodes: 1 publisher, 14 consumers 1 subject Latency vs. message Size *99% confidence intervals in dashed lines Throughput Message volume vs. message Size 1 publisher 14 consumers 1 subject Batch Processing Parameter on Delays small messages gathers them together Improves throughput Throughput Byte volume vs. message Size 1 publisher 14 consumers 1 subject Batch processing parameter on Throughput Byte volume vs. Message Size 1 publisher Publishes on 10,000 subjects 14 consumers Consumer subscribe to all subjects Batching processing parameter on Information Bus - Discussion Are semantics strong enough? Recalls Linda: great idea but fuzzy around edges Here, issues revolve around handling of overloads and receiver crashes Also impossible to implement a history mechanism with strong semantics But paper argues strongly that end-to-end philosophy supports their model Content filtering A rich area Gryphon really looks at two issues Content filtering on a single node Routing infrastructure Most modern systems don t bother with the routing infrastructure and do content filtering on a server messages flow through it Has obvious capacity limits The Matching Problem Instead of a topic subscription, user poses a query over the messages Our task: combine parts of subscription to reduce the number of tests for each event Cayuga system here at Cornell takes this idea way beyond what Gryphon offered Matching Algorithm Analyze subscriptions sub := pr1 ^ pr2 ^ pr3 Conjunction of elementary predicates pri = testi(e) -> resi e.g. (city=LA) and (temprature < 40) pr1 = test1( ) -> LA pr2 = test2( ) -> < test1 = examine attribute city test2 = examine attribute temperature 40 Matching Algorithm Preprocess to make matching tree Each non-leaf node is a test Each edge from test node is a possible result Each leaf node is a subscription Pre-process each of the subscriptions and combine the information to prepare the tree On receiving events, follow the sequence of test nodes and edges till a leaf node is reached Gryphon internals Gryphon implements the Java Messaging Service publish/subscribe specification. allows arbitrary filters based loosely on SQL where clause semantics. More recent systems such as Cayuga have focused on the XML query language Matching Trees Note that they really only handle equality queries. Key feature, result 1 from test 1 leads to exactly one place Matching Tree with *-edges The important formulae V is the number of possible values an attribute can take K is the number of attributes S is the set of subscriptions C(S) is the expected cost Optimizations Collapse a chain of * edges (60% gain) Example: collapse B to A Assumption: non-* edges evaluated before *-edge Idea is to use information about traversal to skip over tests including *-edges that are implied Example: For any event <1,2,3,8,2> consider successors of node C <a1=1,a2=2,a3=3> Statically pre-compute successor nodes H:<a1=1,a2=2,a3=*> G:<a1=1,a2=*,a3=3> D:<a1=*,a2=2,a3=3> E:<a1=*,a2=*,a3=3> F:<a1=*,a2=2,a3=*> Since D doesn t exist, consider it s successors Optimizations Optimizations More aggressive static analysis (20% gain) Separate sub-trees for attributes that rarely have don t care in subscriptions Performance Pentium 100MHz, Java based prototype Attributes vary in popularity, follow Zipf s distribution Tests for 30 attributes with 3 possible values Distribution always got 100 matches per event Performance Space (thousands of cells) 1000 s of subscriptions Operations per Event Space per Event = Edges + Successor nodes Latency: 4ms for 25,000 subscriptions Operations per Event Content based subscription Discussion Do applications actually want to query message contents in this manner? Gryphon goes to great lengths to optimize its routing structure. But in practice would this feature be useful? Could a user launch an unintended denial of service attack by posing a very complex query?

Find millions of documents here - Study Guides, Homework Solutions, Papers, Exam Answer Keys and more. Course Hero has millions of course related materials that will enable you to learn better, faster and get an A in all your courses.
Below is a small sample set of documents:

vsync.ppt
Path: Cornell >> CS >> 614 Fall, 2008

Epidemic.ppt
Path: Cornell >> CS >> 614 Fall, 2008
EGRW00-face.pdf
Path: Cornell >> CS >> 667 Fall, 2008
SG08-shhair.pdf
Path: Cornell >> CS >> 667 Fall, 2008
PlantSoil 225, 63-72, 2000 Lehmann.pdf
Path: Cornell >> CSS >> 472 Spring, 2008
PlantSoil 225, 63-72, 2000 Lehmann.pdf
Path: Cornell >> CSS >> 672 Fall, 2008
Lehmann 2007 Scitizen 709.pdf
Path: Cornell >> CSS >> 472 Spring, 2008
Lehmann 2007 Scitizen 709.pdf
Path: Cornell >> CSS >> 672 Fall, 2008
PlantSoil 286, 287-299, 2006 Soethe.pdf
Path: Cornell >> CSS >> 472 Spring, 2008
PlantSoil 286, 287-299, 2006 Soethe.pdf
Path: Cornell >> CSS >> 672 Fall, 2008
SoilSciSocAmJ 70, 1708-1718, 2006 Kinyangi.pdf
Path: Cornell >> CSS >> 472 Spring, 2008
SoilSciSocAmJ 70, 1708-1718, 2006 Kinyangi.pdf
Path: Cornell >> CSS >> 672 Fall, 2008
QuartRevBiol 79, 313-314, Lehmann 2004.pdf
Path: Cornell >> CSS >> 472 Spring, 2008
QuartRevBiol 79, 313-314, Lehmann 2004.pdf
Path: Cornell >> CSS >> 672 Fall, 2008
NatureGeoscience 1, 238-242, 2008 Lehmann.pdf
Path: Cornell >> CSS >> 472 Spring, 2008
NatureGeoscience 1, 238-242, 2008 Lehmann.pdf
Path: Cornell >> CSS >> 672 Fall, 2008
compostairemissions.pdf
Path: Cornell >> CSS >> 674 Fall, 2008
ColdCompostreport.pdf
Path: Cornell >> CSS >> 674 Fall, 2008
NEDairybeddingcosts.pdf
Path: Cornell >> CSS >> 674 Fall, 2008
Lupinski.pdf
Path: Cornell >> CSS >> 674 Fall, 2008
milorganite.pdf
Path: Cornell >> CSS >> 674 Fall, 2008
beddingfinalreport.pdf
Path: Cornell >> CSS >> 674 Fall, 2008
carcass2.ppt
Path: Cornell >> CSS >> 674 Fall, 2008
PackWast.pdf
Path: Cornell >> CSS >> 674 Fall, 2008
WhatIs.pdf
Path: Cornell >> CSS >> 674 Fall, 2008
soiltests.pdf
Path: Cornell >> CSS >> 674 Fall, 2008
aifs.pdf
Path: Cornell >> CSS >> 674 Fall, 2008
Searchhandout.pdf
Path: Cornell >> CSS >> 674 Fall, 2008
updatejul2008.pdf
Path: Cornell >> CSS >> 674 Fall, 2008
compostsign.pdf
Path: Cornell >> CSS >> 674 Fall, 2008
esp2.pdf
Path: Cornell >> DUTCH >> 300 Spring, 2008
ep52.pdf
Path: Cornell >> DUTCH >> 300 Spring, 2008
esp2005-19.pdf
Path: Cornell >> DUTCH >> 300 Spring, 2008
esp4.pdf
Path: Cornell >> DUTCH >> 300 Spring, 2008
esp18.pdf
Path: Cornell >> DUTCH >> 300 Spring, 2008
esp2005-18.pdf
Path: Cornell >> DUTCH >> 300 Spring, 2008
Liuetal:96SP.pdf
Path: Cornell >> ECE >> 493 Fall, 2008
Liuetal:96SP.pdf
Path: Cornell >> ECE >> 537 Fall, 2008
MaranoMattaWillettTong04SPL.pdf
Path: Cornell >> ECE >> 493 Fall, 2008
MaranoMattaWillettTong04SPL.pdf
Path: Cornell >> ECE >> 537 Fall, 2008
MaranoMattaWillettTong06SP.pdf
Path: Cornell >> ECE >> 493 Fall, 2008
MaranoMattaWillettTong06SP.pdf
Path: Cornell >> ECE >> 537 Fall, 2008
SungEtal07JSAC.pdf
Path: Cornell >> ECE >> 493 Fall, 2008
SungEtal07JSAC.pdf
Path: Cornell >> ECE >> 537 Fall, 2008
TongZhao:99SP.pdf
Path: Cornell >> ECE >> 493 Fall, 2008
TongZhao:99SP.pdf
Path: Cornell >> ECE >> 537 Fall, 2008
DongTongSadler07SP.pdf
Path: Cornell >> ECE >> 493 Fall, 2008
DongTongSadler07SP.pdf
Path: Cornell >> ECE >> 537 Fall, 2008
noterev.pdf
Path: Cornell >> ECON >> 324 Fall, 2008
tsyrennikov.pap.pdf
Path: Cornell >> ECON >> 324 Fall, 2008
dp142.pdf
Path: Cornell >> ECON >> 324 Fall, 2008
OLRcae.pdf
Path: Cornell >> ECON >> 324 Fall, 2008
Izmalkov_Enaure_abstract.pdf
Path: Cornell >> ECON >> 703 Fall, 2008
manjonglee.pap.pdf
Path: Cornell >> ECON >> 703 Fall, 2008
08-01.pdf
Path: Cornell >> ECON >> 703 Fall, 2008
04-05.pdf
Path: Cornell >> ECON >> 703 Fall, 2008
AddedWorker.PDF
Path: Cornell >> ECON >> 703 Fall, 2008
Long.pdf
Path: Cornell >> ECON >> 703 Fall, 2008
04-01.pdf
Path: Cornell >> ECON >> 703 Fall, 2008
PFG00.pdf
Path: Cornell >> SINHA >> 121 Fall, 2008
KJP02.pdf
Path: Cornell >> SINHA >> 121 Fall, 2008
McIntosh et al 2002.pdf
Path: Cornell >> ENTOM >> 264 Fall, 2008
freyetal.pdf
Path: Cornell >> FILM >> 495 Fall, 2008
minimum_principle.pdf
Path: Cornell >> FILM >> 495 Fall, 2008
102.pdf
Path: Cornell >> FILM >> 495 Fall, 2008
81.pdf
Path: Cornell >> FILM >> 495 Fall, 2008
stop_liquid.pdf
Path: Cornell >> FILM >> 495 Fall, 2008
perception_mcgraw.pdf
Path: Cornell >> FILM >> 495 Fall, 2008
edelman-reply.pdf
Path: Cornell >> FILM >> 495 Fall, 2008
pirenne's shadow san vincenzo 4.pdf
Path: Cornell >> GREEK >> 102 Fall, 2008