and Design Evaluation of a Wide-Area Event Notication Service
ANTONIO CARZANIGA University of Colorado at Boulder DAVID S. ROSENBLUM University of California, Irvine and ALEXANDER L. WOLF University of Colorado at Boulder
The components of a loosely coupled system are typically designed to operate by generating and responding to asynchronous events. An event notication service is an application-independent infrastructure that supports the construction of event-based systems, whereby generators of events publish event notications to the infrastructure and consumers of events subscribe with the infrastructure to receive relevant notications. The two primary services that should be provided to components by the infrastructure are notication selection (i.e., determining which notications match which subscriptions) and notication delivery (i.e., routing matching notications from publishers to subscribers). Numerous event notication services have been developed for local-area networks, generally based on a centralized server to select and deliver event notications. Therefore, they suffer from an inherent inability to scale to wide-area networks, such as the Internet, where the number and physical distribution of the services clients can quickly overwhelm a centralized solution. The critical challenge in the setting of a wide-area network is to maximize the expressiveness in the selection mechanism without sacricing scalability in the delivery mechanism. This paper presents SIENA, an event notication service that we have designed and implemented to exhibit both expressiveness and scalability. We describe the services interface to applications, the algorithms used by networks of servers to select and deliver event notications, and the strategies used Effort sponsored by the Defense Advanced Research Projects Agency, and Air Force Research Laboratory, Air Force Materiel Command, USAF, under agreement numbers F30602-94-C-0253, F3060297-2-0021, F30602-98-2-0163, F30602-99-C-0174, F30602-00-2-0608, and N66001-00-8945; by the Air Force Ofce of Scientic Research, Air Force Materiel Command, USAF, under grant number F49620-98-1-0061; and by the National Science Foundation under Grant Number CCR-9701973. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the ofcial policies or endorsements, either expressed or implied, of the Defense Advanced Research Projects Agency, Air Force Research Laboratory, or the U.S. Government. Authors addresses: A. Carzaniga and A. L. Wolf, Department of Computer Science, University of Colorado at Boulder, 430 UCB, Boulder, CO 80309-0430; email: {carzanig,alw}@cs.colorado.edu; D. S. Rosenblum, Department of Information and Computer Science, University of California, Irvine, Irvine, CA 92697-3425; email: [e-mail]. Permission to make digital/hard copy of part or all of this work for personal or classroom use is granted without fee provided that the copies are not made or distributed for prot or commercial advantage, the copyright notice, the title of the publication, and its date appear, and notice is given that copying is by permission of the ACM, Inc. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specic permission and/or a fee. C 2001 ACM 0734-2071/01/08000332 $5.00
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001, Pages 332383.
Wide-Area Event Notication Service
333
to optimize performance. We also present results of simulation studies that examine the scalability and performance of the service. Categories and Subject Descriptors: C.2.1 [Network Architecture and Design]: Distributed Networks; Network Communication; Network Topology; Store and Forward Networks; C.2.2 [Network Protocols]: Applications; Routing Protocols; C.2.4 [Distributed Systems]: Client/server Distributed applications; C.2.5 [Local and Wide-Area Networks]: Internet; C.2.6 [Internetworking]: Routers; C.4 [Performance of Systems]: Design Studies; Modeling Techniques; I.6.3 [Simulation and Modeling]: Applications; I.6.4 [Simulation and Modeling]: Model Validation and Analysis; I.6.8 [Simulation and Modeling]: Types of SimulationDiscrete event General Terms: Algorithms, Experimentation, Performance Additional Key Words and Phrases: Event notication, publish/subscribe, content-based addressing and routing
1. INTRODUCTION The asynchrony, heterogeneity, and inherent loose coupling that characterize applications in a wide-area network promote event interaction as a natural design abstraction for a growing class of software systems. An emerging building block for such systems is an infrastructure called an event notication service [Rosenblum and Wolf 1997]. We envision a ubiquitous event notication service accessible from every site on a wide-area network and suitable for supporting highly distributed applications requiring component interactions ranging in granularity from ne to coarse. Conceptually, the service is implemented as a network of servers that provide access points to clients. Clients use the access points to advertise information about events and subsequently to publish multiple notications of the kind previously advertised. Thus, an advertisement expresses the clients intent to publish a particular kind of notication. They also use the access points to subscribe for notications of interest. The service uses the access points to then notify clients by delivering any notications of interest. Clearly, an event notication service complements other general-purpose middleware services, such as point-to-point and multicast communication mechanisms, by offering a many-to-many communication and integration facility. The event notication service can carry out a selection process to determine which of the published notications are of interest to which of its clients, routing and delivering notications only to those clients that are interested. In addition to serving clients interests, the selection process also can be used by the event notication service to optimize communication within the network. The information that drives the selection process originates with clients. More specically, the event notication service may be asked to apply a lter to the contents of event notications, such that it will deliver only notications that contain certain specied data values. The selection process may also be asked to look for patterns of multiple events, such that it will deliver only sets of notications associated with that pattern of event occurrences (where each individual event occurrence is matched by a lter). Given that the primary purpose of an event notication service is to support notication selection and delivery, the challenge we face in a wide-area setting
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
334
A. Carzaniga et al.
is maximizing expressiveness in the selection mechanism without sacricing scalability in the delivery mechanism [Carzaniga et al. 2000a]. Expressiveness refers to the ability of the event notication service to provide a powerful data model with which to capture information about events, to express lters and patterns on notications of interest, and to use that data model as the basis for optimizing notication delivery. In terms of scalability, we are referring not simply to the number of event generators, the number of event notications, and the number of notication recipients, but also to the need to discard many of the assumptions made for local-area networks, such as low latency, abundant bandwidth, homogeneous platforms, continuous and reliable connectivity, and centralized control. We recognize that there are other important attributes of an event notication service besides expressiveness and scalability, including security, reliability, and fault tolerance, but we do not address them in this paper. Intuitively, a simple event notication service that provides no selection mechanism can be reduced to a multicast routing and transport mechanism for which there are numerous scalable implementations. However, once the service provides a selection mechanism, then the overall efciency of the service and its routing of notications are affected by the power of the language used to construct notications and to express lters and patterns. As the power of the language increases, so does the complexity of the processing. Thus, in practice, scalability and expressiveness are two conicting goals that must be traded off. This paper presents SIENA, an event notication service that we have designed and implemented to maximize both expressiveness and scalability. In Section 3 we describe the services formally dened application programming interface, which is an extension of the familiar publish/subscribe protocol [Birman 1993]. Several candidate server topologies and protocols are presented in Section 4. We then describe in Section 5 the routing algorithms used by the service to deliver event notications to clients; these algorithms are designed specically for distributed networks of event servers. This is followed by a description of strategies for optimizing the performance of the notication selection process. Supported in part by the results of simulation studies, we present an analysis of the scalability of our design choices in Section 6. In our simulation studies, we focus on two alternative service architectures, one based on a hierarchical topology, and the other based on a peer-to-peer topology. In particular, we study how these two architectures perform under various classes of applications in which the densities and distributions of clients differ. We conclude in Sections 7 and 8 with a discussion of related work and a brief indication of our future plans. 2. FRAMING THE PROBLEM AND ITS SOLUTION As discussed in Section 1, an event notication service implements two key activities, notication selection and notication delivery. A naive approach to realizing these activities is to employ a central server where all subscriptions are recorded, where all notications are initially targeted, where notications
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
335
Fig. 1. Distributed event notication service.
are evaluated against all subscriptions, and from where notications are sent out to all relevant subscribers. This solution is logically very simple, but is impractical in the face of scale. Clearly, the service instead must be architected as a distributed system in which activities are spread across the network, hopefully exploiting some sort of locality, and hopefully exhibiting a reasonable growth in complexity. In its most general form, a distributed event notication service is composed of interconnected servers, each one serving some subset of the clients of the service, as shown in Figure 1. (Some use the terms proxy and broker instead of the term server.) The clients are of two kinds: objects of interest, which are the generators of events, and interested parties, which are the consumers of event notications. Of course, a client can act as both an object of interest and an interested party. Both kinds of clients interact with a locally accessible server, which functions as an access point to the networkwide service. In practice, the service becomes a wide-area network of pattern matchers and routers, overlaid on top of some other wide-area communication facility, such as the Internet. One reasonable allocation of such servers might be to place a server at each administrative domain within the low-level, wide-area communication network. Creating a network of servers to provide a distributed service of any sort gives rise to three critical design issues: Interconnection topology: In what conguration should the servers be connected? Routing algorithm: What information should be communicated between the servers to allow the correct and efcient delivery of messages? Processing strategy: Where in the network, and according to what heuristics, should message data be processed in order to optimize message trafc? These three design issues have been studied extensively for many years and in many contexts. Our challenge is to nd a solution in the particular domain of wide-area event notication, leveraging previous results (both positive and negative) wherever possible.
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
336
A. Carzaniga et al.
In terms of interconnection topology, there are essentially two broad classes from which to choose: a hierarchy and a general graph. Existing distributed event notication services, such as JEDI [Cugola et al. 1998] and TIBCOs product TIB/Rendezvous, adopt a hierarchical topology. However, our analysis (presented in Section 6) shows that such a topology can exhibit signicant performance problems. In SIENA we have adopted the general graph, which in common terms means that the servers are organized in a peer-to-peer relationship, as we detail in Section 4. A hybrid of the two structureswhether a hierarchy of peers, or peers of hierarchiesis also a topology to consider, but requires a priori knowledge of the inherent structure of the services applications in order to make a proper subdivision among peers and hierarchies. Having such knowledge would violate the notion that the service is general purpose. Our desire for the event notication service to be general purpose also complicates the routing problem for the service. In particular, we assume that objects of interest have no knowledge of interested parties. Therefore, event notications cannot be addressed and routed in the same, relatively simple manner as, for example, an electronic mail message. Moreover, we cannot assume any particular locality of objects of interest and interested parties, which is a fact that bears a strong relationship to the server topology issue. At best we can only try to take advantage of any locality or structure in the message trafc as it emerges. We demonstrate below that advertisements and subscriptions serve as the basis for this. Given these considerations, solving the routing problem can be seen as a choice among three alternatives. Common to the three alternatives is the need to broadcast some piece of information to all the servers in the network, where the broadcast is required by the lack of a priori knowledge of locality. The rst alternative broadcasts notications, which implies that notication matching is performed at each local server based on the subscriptions received at that server. This alternative suffers from the drawback that all notications are delivered to all local servers, whether or not they are serving any parties interested in the notications. The second and third alternatives try to take advantage of emergent locality and structure. In particular, the second alternative involves a broadcast of subscriptions. A shortest-path algorithm is used to route notications back to only the local servers of interested parties. Under the third alternative, advertisements are broadcast, and subscriptions are then used to establish paths, akin to virtual circuits, by which notications are routed to only the local servers of interested parties. Of course, both these alternatives suffer from the cost of having to store either all subscriptions or all advertisements at all servers. The drivers that trade off among the three alternatives are fairly straightforward to identify, but in the design of a general-purpose service, any choice will be suboptimal for some situation, as we discuss in Section 5. Fortunately, we can improve the situation considerably by being intelligent about how we allocate the notication-matching tasks within the network. This is the design issue that concerns the processing strategy. We observe, that in practice, many parties are interested in similar events. Put another
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
337
way, it is likely that distinct subscriptions dene partially, or even completely, overlapping sets of notications. A similar observation can be made about objects of interest and their advertisements. We therefore sketch how this observation leads to a processing strategy for subscriptions and assume a corresponding strategy exists for advertisements; Section 5 presents a detailed discussion of these strategies. Based on our observation about the likely relationship among subscriptions, the strategy works as follows. When a subscription reaches a server (either from a client or from another server), the server propagates that subscription only if it denes new selectable notications that are not in the set of selectable notications dened by any previously propagated subscription. Three benets accrue from this approach. First, we reduce network costs by pruning the propagation of new subscriptions. Second, we reduce the storage requirements for servers. Third, by reducing the number of subscriptions held at each server, we reduce the computational resources needed to match notications at that server. We use a similar strategy for propagation of advertisements. Up to this point in the discussion we have treated notications, subscriptions, and advertisements in rather abstract terms. We now make these concepts somewhat more concrete as a basis for the material presented in the next several sections. As mentioned in the introduction, the information associated with an event is represented by a data structure called a notication. We refer to the data model or encoding schema of notications as the event notication model or simply event model. Most existing event notication services adopt a simple recordlike structure for notications, while some more recent frameworks dene an object-oriented model (e.g., the Java Distributed Event Specication [Sun Microsystems 1998] and the CORBA Notication Service [Object Management Group 1998b]). Closely related to the event model is the subscription language, which denes the form of the expressions associated with subscriptions. Two aspects of the subscription language are crucial to the issue of expressiveness. Scope of the subscription predicates: This aspect is concerned with the visibility that subscriptions have into the contents of a notication. For a recordlike notication structure, visibility determines which elds can be used in specifying a subscription. Power of the subscription predicates: This aspect is concerned with the sophistication of operators that can be used in forming subscription predicates. The predicates apply both to any possible ltering of individual notications as well as to any possible formation of patterns of multiple notications. The dual of the subscription language is the advertisement language, which shares the issues of scope and power, but from the perspective of an object of interest, rather than an interested party. One consequence of this difference in perspective is that interested parties may subscribe for patterns of multiple notications, whereas objects of interest advertise only individual notications. The following sections elaborate on these basic concepts and our approach to achieving expressiveness and scalability.
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
338
A. Carzaniga et al.
Table I. Interface of SIENA publish(notication n) subscribe(string identity, pattern expression) unsubscribe(string identity, pattern expression) advertise(string identity, lter expression) unadvertise(string identity, lter expression)
3. API AND SEMANTICS At a minimum, an event notication service exports two functions that together dene what is usually referred to as the publish/subscribe protocol. Interested parties specify the events in which they are interested by means of the function subscribe. Objects of interest publish notications via the function publish. SIENA extends the publish/subscribe protocol with an additional interface function called advertise, which an object of interest uses to advertise the notications it publishes. SIENA also adds the functions unsubscribe and unadvertise. Subscriptions can be matched repeatedly until they are canceled by a call to unsubscribe. Advertisements remain in effect until they are canceled by a call to unadvertise. Table I shows the interface functions of SIENA. The expression given to subscribe and unsubscribe is a pattern, while the expression given to advertise and unadvertise is a lter; we discuss patterns and lters in greater detail below. The parameter identity species the identity of the object of interest or interested party. Objects of interest and interested parties must identify themselves to SIENA when they advertise or subscribe, respectively, so that they can later cancel their own advertisements or subscriptions. The only requirement that SIENA imposes on identiers is that they be unique. We note that SIENA maintains a mapping between the identities of interested parties and their handlers. A handler species the means by which an interested party receives notications, either through callbacks or through messages sent via a communication protocol such as HTTP or SMTP. Separating these two concepts at the level of clients allows for the possibility of redirecting and/or temporarily suspending the ow of notications from objects of interest to interested parties, and supports the mobility of clients. Detailed discussion of handlers is beyond the scope of this paper. 3.1 Notications An event notication (or simply a notication) is a set of typed attributes. For example, the notication displayed in Figure 2 represents a stock price change event. Each individual attribute has a type, a name, and a value, but the notication as a whole is purely a structural value derived from its attributes. Attribute names are simply character strings. The attribute types belong to a predened set of primitive types commonly found in programming languages and database query languages, and for which a xed set of operators is dened.
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
339
Fig. 2. Example of a notication.
The justication for choosing this typing scheme is scalability. In other systems, such as one nds for example in the Java Distributed Event Specication [Sun Microsystems 1998] and CORBA Notication Service [Object Management Group 1998b], a notication is a value of some named, explicit notication type. This implies a global authority for managing and verifying the type space, something which is clearly not feasible at an Internet scale. On the other hand, we dene a restricted set of attribute types from which to construct (arbitrary) notications. By having this well-dened set, we can perform efcient routing based on the content of notications. As we discuss in Section 7, content-based routing has distinct advantages over the alternative schemes of channel- and subject-based routing. 3.2 Filters An event lter, or simply a lter, selects event notications by specifying a set of attributes and constraints on the values of those attributes. Each attribute constraint is a tuple specifying a type, a name, a binary predicate operator, and a value for an attribute. The operators provided by SIENA include all the common equality and ordering relations (=, =, <, >, etc.) for all of its types; substring (), prex (> ), and sufx ( <) operators for strings; and an operator any that matches any value. An attribute = (type , name , value ) matches an attribute constraint = (type , name , operator , value ) if and only if type = type name = name operator (value , value ). We say an attribute satises or matches an attribute constraint with the notation . When matches , we also say that covers . Figure 3 shows a lter that matches price increases for stock DIS on stock exchange NYSE. When a lter is used in a subscription, multiple constraints for the same attribute are interpreted as a conjunction; all such constraints must be matched. Thus, we say that a notication n matches a lter f , or equivalently that f N covers n (n S f for short): n
N S
f f : n :
A lter can have two or more attribute constraints with the same name, in which case the matching rule applies to all of them. Also, the notication may contain other attributes that have no correspondents in the lter. Table II gives N some examples that illustrate the semantics of S . The second example is not a match because the notication is missing a value for attribute level. The third
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
340
A. Carzaniga et al.
Fig. 3. Example of an event lter.
Table II. Examples of Notication string what = alarm time date = 02:40:03 string what = alarm time date = 02:40:03 string what = alarm integer level = 10 string what = alarm integer level = 5
N S N S
N S
Subscription string what = alarm string what = alarm integer level > 3 string what = alarm integer level > 3 integer level < 7 string what = alarm integer level > 3 integer level < 7
N S
N S
example is not a match because the constraints specied for attribute level in the subscription are not matched by the value for level in the notication. 3.3 Patterns While a lter is matched against a single notication based on the notications attribute values, a pattern is matched against one or more notications based on both their attribute values and on the combination they form. At its most generic, a pattern might correlate events according to any relation. For example, the programmer of a stock market analysis tool might be interested in receiving price change notications for the stock of one company only if the price of a related stock has changed by a certain amount. Rich languages and logics exist that allow one to express event patterns [Mansouri-Samani and Sloman 1997]. In SIENA we do not attempt to provide a complete pattern language. Our goal is rather to study pattern operators that can be exploited to optimize the selection of notications within the event notication service. Here, we restrict a pattern to be syntactically a sequence of lters, f 1 f 2 f n , that is matched by a temporally ordered sequence of notications, each one matching the corresponding lter. An example of a pattern is shown in Figure 4, which matches an increase in the price of stock MSFT followed by a subsequent increase in the price of stock NSCP. In general, we observe that more sophisticated forms of patterns can always be split into a set of simple subscriptions and then matched externally to SIENA (i.e., at the access point of the interested party), although this is likely to induce extra network trafc. We say that a pattern is simple when it is composed of a single lter, and similarly we say that a subscription is simple when it requests a simple pattern.
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
341
Fig. 4. Example of an event pattern.
There are many possible semantics for the lter sequence operator. In the interests of scalability, we have opted for the simplest possible semantics, which ignores out-of-order matches of notications due to network latency (see Section 3.8). To understand the semantics we chose, consider the pattern A B (read A followed by B), which we assume to be submitted as a subscription at time j t0 . We represent notications that match A as Ai , meaning the notication was generated at time ti by the object of interest and was matched at time t j by the server responsible for matching the pattern (and similarly for notications j Bi matched to B). Consider the following sequence of notications (shown in match order):
1 4 6 8 B4 A2 A3 B2 A5 B6 A7 B8 3 1 5 7
According to the semantics we chose, the server matching A B uses the rst j m Ai it matches followed by the rst Bk it matches to form the rst match of the pattern, such that i < k and j < m. It then uses the next A it matches followed by the next B it matches to form the second match of the pattern, and so on. 6 Hence, the rst match of the pattern would be the sequence A2 B6 , and the 3 7 8 1 second match would be the sequence A7 B8 . The matcher receives B4 rst but discards it because it has not yet matched an A. The rst A it matches is A2 , so it 3 m ignores all subsequent As until it matches a Bk where k > 3. Thus it ignores A3 1 4 and A5 because it is waiting for a B; it also ignores B2 because it was generated 5 6 before A2 . Hence B6 is the rst B that can be matched with A2 . Once this whole 3 3 sequence has been matched, the matching of the pattern begins anew with the 6 next A following B6 , which is A7 . The second match of the pattern is completed 7 8 with B8 . 3.4 Advertisements
N We have seen how the covering relation S denes the semantics of lters in subscriptions. We now dene the semantics of advertisements by dening a similar relation N . The motivation for advertisements is to inform the event A notication service about which kind of notications will be generated by which objects of interest, so that it can best direct the propagation of subscriptions. The idea is, that while a subscription denes the set of interesting notications for an interested party, an advertisement denes the set of notications potentially generated by an object of interest. Therefore, the advertisement is relevant to the subscription only if these two sets of notications have a nonempty intersection. ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
342
A. Carzaniga et al.
Table III. Examples of Notication string what = alarm string what = alarm time date = 02:40:03 string what = login string username = carzanig string what = logout string username = carzanig
N A N A
Advertisement string what = alarm string what = login string username = any string what = alarm string what = login string username = any string what = alarm string what = login string username = any string what = alarm string what = login string username = any
N A
N A
N A
The relation
N A
denes the set of notications covered by an advertisement: n
N A
a n : a :
This expression says that an advertisement covers a notication if and only if it covers each individual attribute in the notication. Note that this is the dual of subscriptions, which dene the minimal set of attributes that a notication must contain. In contrast to subscriptions, when a lter is used as an advertisement, multiple constraints for the same attribute are interpreted as a disjunction rather than as a conjunction; only one of the constraints need be satised. Table III shows some examples of the relation N . The second examA ple is not a match because the attribute date of the notication is not dened in the advertisement. The fourth example is not a match because the value of attribute what in the notication does not match any of the constraints dened for what in the advertisement. 3.5 Two Variants of the Semantics of SIENA We have studied two alternative semantics for SIENA, a subscription-based semantics and an advertisement-based semantics. Under the subscription-based semantics, the semantics of subscriptions is N dened solely by the relation S and its extensions to patterns. Advertisements are not enforced on noticationsthey may be used for optimization purposes, or they can be ignored completely by the implementation of the service. Thus, a notication n is delivered to an interested party X if and only if N X submitted at least one subscription s such that n S s. Under the advertisement-based semantics, both advertisements and subscriptions are used. In particular, a notication n published by object Y is delivered to interested party X if and only if Y advertised a lter a that covers n (i.e., such that n N a) and X registered a subscription s that covers n (i.e., A N such that n S s).
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
343
Under both semantics, a notication is delivered at most once to any interested party. 3.6 Other Important Covering Relations So far we have dened a number of relations that express the semantics of subscriptions and advertisements: : attribute matches attribute constraint ; N n S f : notication n matches lter f , where f is interpreted as a subscription lter; n N a: notication n matches lter a, where a is interpreted as an adverA tisement lter; From these, other relations can be derived: f 2 S f 1 : lter f 1 covers lter f 2 , where f 1 and f 2 are interpreted as subS scriptions. Formally, f2
S S
f 1 n : n
N S
f2 n
N S
f1
which means that f 1 denes a superset of the notications dened by f 2 . a2 A a1 : lter a1 covers lter a2 , where a1 and a2 are interpreted as adverA tisements. Formally a2
A A a1
n : n
N A
a2 n
N A
a1
which means that a1 denes a superset of the notications dened by a2 . f S a: lter a covers lter f , where a is interpreted as an advertisement A and f is interpreted as a subscription. Formally, f
S Aa
n : n
N A
an
N S
f
which means that a denes a set of notications that has a nonempty intersection with the set dened by f . The relations S and A can also dene the equality relation between lters S A with its intuitive meaning: f1 = f2 f1 We now use the relations tions and unadvertisements.
S S
f2 f2
A A
f 1.
and
to dene the semantics of unsubscrip-
3.7 Unsubscriptions and Unadvertisements Unsubscriptions and unadvertisements serve to cancel previous subscriptions and advertisements, respectively. Given a simple unsubscription unsubscribe(X , f ), where X is the identity of an interested party and f is a lter, the event notication service cancels all simple subscriptions subscribe(X , g ) submitted by the same interested party X with a subscription lter g covered by f (i.e., such that g S f ). This semantics is extended easily to S patterns. An unsubscription for a pattern P = f 1 f 2 f k cancels all previous subscriptions S = g 1 g 2 g k such that g 1 S f 1 g 2 S f 2 . . . g k S f k . In S S S
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
344
A. Carzaniga et al.
an analogous way, unadvertisements cancel previous advertisements that are covered according to the relation A . A Note that an unsubscription (unadvertisement) either cancels previous subscriptions (advertisements) or else has no effect. It cannot impose further constraints onto existing subscriptions. For example, subscribing with a lter [price > 100] and then unsubscribing with [price > 200] does not result in creation of a reduced subscription, [price > 100, price 200]. Rather, the unsubscription simply has no effect, since it does not cover the subscription. Note also that all subscriptions covered by an unsubscription are canceled by that unsubscription. Thus, when an interested party initially subscribes with a specic lter (say, [change > 10]), then subscribes with a more generic one (say, [change > 0]), and then nally unsubscribes with a lter that covers the more generic subscription (say, [change > 0]), the effect is to cancel all the previous subscriptions, not to revert to the more specic one [change > 10]. 3.8 Timing Issues The semantics of SIENA depends on the order in which SIENA receives and processes requests (subscriptions, notications, etc.). For instance, in the subscription-based semantics, a subscription s is effective after it is processed and until an unsubscription u that cancels s is processed. In the most general case, a service request R, say a subscription, is generated at time R g , received at time Rr , and completely processed at time R p (with R g Rr R p ). SIENA guarantees the correct interpretation of R immediately after R p . Notice that the external delay R g Rr is caused by external communication mechanisms and is by no means controllable by SIENA. The processing delay R p R g is instead directly caused by computations and possibly by other communication delays internal to SIENA. SIENAs semantics is that of a best-effort service. This means that the implementation of SIENA must not introduce unnecessary delays in its processing, but it is not required to prevent race conditions induced by either the external delay or the processing delay. Clients of SIENA must be resilient to such race conditions; for instance, they must allow for the possibility of receiving a notication for a canceled subscription. SIENA associates a timestamp with each notication to indicate when it was published.1 This allows the service to detect and account for the effects of latency on the matching of patterns, which means that within certain limits the actual order of notications can be recognized. 4. ARCHITECTURES: SERVER TOPOLOGIES AND PROTOCOLS The previous section describes the protocol by which clients (i.e., objects of interest and interested parties) communicate with the servers that act as the clients access points to the event notication service. As mentioned in Section 2, the
1 With
the advent of accurate network time protocols and the existence of the satellite-based Global Positioning System (GPS), it is reasonable to assume the existence of a global clock for creation of these timestamps, and it is hence reasonable for all but the most time-sensitive applications to rely on these timestamps.
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
345
Fig. 5. Hierarchical client/server architecture.
servers themselves communicate in order to cooperatively distribute the selection and delivery tasks across a wide-area network. The servers must therefore be arranged into an interconnection topology and make use of a server/server communication protocol. Together, the topology and protocol dene what we refer to as an architecture for the event notication service. The architecture is assumed to be implemented on top of a lower-level network infrastructure. In particular, a topological connection between two servers does not necessarily imply a permanent or direct physical connection between those servers, such as TCP/IP. Moreover, the server/server protocol might make use of any one of a number of network protocols, such as HTTP or SMTP, through standard encoding and/or tunneling techniques. All we assume at this point in the discussion is that a given server can communicate with some number of other specic servers by exchanging messages. This is the same assumption we make about the communication between clients and servers. In this section we consider three basic architectures: hierarchical client/server, acyclic peer-to-peer, and general peer-to-peer. We also consider some hybrid architectures. Because it is not scalable, the degenerate case of a centralized architecture having a single server is ignored in this paper. 4.1 Hierarchical Client/Server Architecture A natural way of connecting event servers is according to a hierarchical topology, as illustrated in Figure 5. In this topology, pairs of connected servers interact in an asymmetric client/server relationship. Hence, we use a directed graph to represent the topology of this architecture, and we refer to this architecture as a hierarchical client/server architecture (or simply a hierarchical architecture). A server can have any number of incoming connections from other client servers, but only one outgoing connection to its own master server. A server that has no master server of its own is referred to as a root.
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
346
A. Carzaniga et al.
Fig. 6. Acyclic peer-to-peer server architecture.
The hierarchical architecture is a straightforward extension of a centralized architecture. It only requires that the basic central server be modied to propagate any information that it receives (i.e., subscriptions, etc.) on to its master server. In fact, the server/server protocol we use within the hierarchical architecture is exactly the same as the protocol described in Section 3 for communication between the servers and the external clients of the event notication service. Thus, in terms of communication, a server is not distinguished from objects of interest or interested parties. In practice, this means that a server will receive subscriptions, advertisements, and notications from its client servers, and will send only notications back to those client servers. As we demonstrate in Section 6.2.2.3, the main problem exhibited by the hierarchical architecture is the potential overloading of servers high in the hierarchy. Moreover, every server acts as a critical point of failure for the whole network. In fact, a failure in one server disconnects all the subnets reachable from its master server and all the client subnets from each other. 4.2 Acyclic Peer-to-Peer Architecture In the acyclic peer-to-peer architecture, servers communicate with each other symmetrically as peers, adopting a protocol that allows a bidirectional ow of subscriptions, advertisements, and notications. Hence we use an undirected graph to represent the topology of this architecture. (As always, the external clients of the service use the standard client/server protocol described in Section 3.) The conguration of the connections among servers in this architecture is restricted so that the topology forms an acyclic undirected graph. Figure 6 shows an acyclic peer-to-peer architecture of servers. The communication between servers is represented by thick undirected lines, while the communication between clients and servers is represented by dashed arrows. It is important that the procedures adopted to congure the connections among servers maintain the property of acyclicity, since routing algorithms might rely on the property to assume, for instance, that any two servers are
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
347
Fig. 7. General peer-to-peer server architecture.
connected with at most one path. However, ensuring this can be difcult and/or costly in a wide-area service in which administration is decentralized and autonomous. As in the hierarchical architecture, the lack of redundancy in the topology constitutes a limitation in assuring connectivity, since a failure in one server S isolates all the subnets reachable from those servers directly connected to S. 4.3 General Peer-to-Peer Architecture Removing the constraint of acyclicity from the acyclic peer-to-peer architecture, we obtain the general peer-to-peer architecture. Like the acyclic peer-to-peer architecture, this architecture allows bidirectional communication between two servers, but the topology can form a general undirected graph, possibly having multiple paths between servers. An example is shown in Figure 7. The advantage of the general peer-to-peer architecture over the previous two architectures is that it requires less coordination and offers more exibility in the conguration of connections among servers. Moreover, allowing redundant connections makes it more robust with respect to failures of single servers. The drawback of having redundant connections is that special algorithms must be implemented to avoid cycles and to choose the best paths. Typically, messages will carry a time-to-live counter, and routes will be established according to minimal spanning trees. Consequently, the server/server protocol adopted in the general peer-to-peer architecture must accommodate this extra information. 4.4 Hybrid Architectures A wide-area, large-scale, decentralized service such as SIENA poses different requirements at different levels of administration. In other words, we must account for intermediate levels between the local area and the wide area. We can potentially take advantage of these intermediate levels to gain some efciencies
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
348
A. Carzaniga et al.
Fig. 8. Hierarchical/general hybrid server architectures.
by considering the use of different architectures at different levels of network granularity. For example, in the case of a multinational corporation, it might be reasonable to assume a high degree of control and coordination in the administration of the cluster of subnets of the corporations intranet. The administrators of this intranet might very well be able to design and manage the whole network of event servers deployed on their subnets, and thus it might be a good idea to adopt a hierarchical architecture within the intranet. Of course, the intranet would connect to other networks outside of the inuence of the administrators. Thus, what could arise is a general peer-to-peer architecture at the global level, serving to interconnect different corporate intranets, each having a hierarchical architecture. This is illustrated in Figure 8. In other cases, we might want to invert the structure, as illustrated in Figure 9. For example, suppose that some clusters of subnets carry a high degree of event-service message trafc, and for some specic applications or perhaps for security reasons, only a small fraction of that trafc is visible outside the cluster. In this case, for efciency reasons a general peer-to-peer architecture might be preferable within the clusters, while the high-level architecture could be acyclic peer-to-peer. For every cluster, there would be a gateway server that should be able to lter the messages used for the protocol inside the cluster, and adapt them to the protocol used between clusters. For example, if a protocol is used locally within a cluster to discover minimal spanning trees, then the messages associated with that protocol should not be propagated outside the cluster. Hybrid architectures such as these are somewhat more complicated than the three basic architectures. Nevertheless, they offer the opportunity to tailor the server/server topologies and protocols in such a way that localities can be exploited. 5. ROUTING ALGORITHMS AND PROCESSING STRATEGIES Once a topology of servers is dened, the servers must establish appropriate routing paths to ensure that notications published by an object of interest are
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
349
Fig. 9. General/acyclic hybrid server architectures.
correctly delivered to all the interested parties that subscribed for them. In general, we observe that notications must meet subscriptions somewhere in the network so that the notications can be selected according to the subscriptions and then dispatched to the subscribers. This common principle can be realized according to a spectrum of possible routing algorithms. One possibility is to maintain subscriptions at their access point and to broadcast notications throughout the whole network; when a notication meets and matches a subscription, the subscriber is immediately notied locally. However, since we expect the number of notications to far exceed the number of subscriptions or advertisements, this strategy appears to offer the least possible efciency, and so we consider it no further for SIENA. 5.1 Routing Strategies in SIENA To devise more efcient routing algorithms, we employ principles found in IP multicast routing protocols [Deering and Cheriton 1990]. Similar to these protocols, the main idea behind the routing strategy of SIENA is to send a notication only toward event servers that have clients that are interested in that notication, possibly using the shortest path. The same principle applies to patterns of notications as well. More specically, we formulate two generic principles that become requirements for our routing algorithms: downstream replication: A notication should be routed in one copy as far as possible and should be replicated only downstream, that is, as close as possible to the parties that are interested in it. This principle is illustrated in Figure 10. upstream evaluation: Filters are applied, and patterns are assembled upstream, that is, as close as possible to the sources of (patterns of ) notications. This principle is illustrated in Figure 11.
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
350
A. Carzaniga et al.
Fig. 10. Downstream notication replication.
Fig. 11. Upstream lter and pattern evaluation.
These principles are implemented by two classes of routing algorithms, the rst of which involves broadcasting subscriptions and the second of which involves broadcasting advertisements: subscription forwarding: In an implementation that does not use advertisements, the routing paths for notications are set by subscriptions, which are propagated throughout the network so as to form a tree that connects the subscribers to all the servers in the network. When an object publishes a notication that matches that subscription, the notication is routed toward the subscriber following the reverse path put in place by the subscription. advertisement forwarding: In an implementation that uses advertisements, it is safe to send a subscription only toward those objects of interest that intend to generate notications that are relevant to that subscription. Thus, advertisements set the paths for subscriptions, which in turn set the paths for notications. Every advertisement is propagated throughout the network, thereby forming a tree that reaches every server. When a server receives a subscription, it propagates the subscription in reverse, along the paths to all advertisers that submitted relevant advertisements, thereby
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
351
activating those paths. Notications are then forwarded only through the activated paths. In the process of forwarding subscriptions, SIENA exploits commonalities among the subscriptions. In particular, SIENA prunes the propagation trees by propagating along only those paths that have not been covered by previous requests. The derived covering relation S is used to determine whether a S new subscription is covered by a previous one that has already been forwarded. Advertisements are treated similarly using the relation A . And although not A discussed in detail here, unsubscriptions and unadvertisements are handled in a similar way as well. Subscription-forwarding algorithms realize a subscription-based semantics, while advertisement-forwarding algorithms realize an advertisement-based semantics. As we show in Section 5.3, advertisement-forwarding algorithms are needed in order to implement the upstream evaluation principle for event patterns. In addition to the principles introduced above, we have also devised several other strategies that lead to optimizations in resource usage. These are discussed elsewhere [Carzaniga 1998; Carzaniga et al. 1999]. 5.2 Putting Algorithms and Topologies Together We now describe in detail how subscription-forwarding and advertisementforwarding algorithms are implemented within the hierarchical and peer-topeer architectures. In particular, we describe the principal data structures maintained by servers and the main algorithms that process the various requests coming from clients or other servers. Here we consider only simple subscriptions; Section 5.3 deals with patterns. At a high level and in general terms, the algorithms for the acyclic and peerto-peer architectures attempt to reduce communication, storage, and computation costs by pruning spanning trees over a network of SIENA servers. More specically, the subscription-forwarding algorithms operate by broadcasting subscriptions along spanning trees rooted at interested parties. When a server receives a new subscription, it can terminate the further propagation of that subscription if it has already propagated a more general subscription that covers the new one. In this way servers prune spanning trees along which new subscriptions are propagated. The advertisement-forwarding algorithms operate in an analogous fashion by pruning the spanning trees rooted in objects of interest. Computation of spanning trees in a network is a solved problem [Dalal and Metcalfe 1978], and therefore we do not discuss their construction. Instead, our focus is on the details of pruning. The algorithms for the hierarchical architectures are simpler, because subscriptions and advertisements are not propagated along spanning trees, but are merely propagated along unique paths to the root of the hierarchy. 5.2.1 The Filters Poset. In order to keep track of previous requests, their relationships, where they came from, and where they have been forwarded, event servers maintain a data structure that is common to the different algorithms
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
352
A. Carzaniga et al.
Fig. 12. Example of a poset of simple subscriptions. Arrows represent the immediate relation
S S.
and topologies. This data structure represents a partially ordered set (poset) of lters. The partial order is dened by the covering relations S for subscription S lters, and A for advertisement lters. We denote with PS a poset dened by A S A S , and denote with P A a poset dened by A . Figure 12 shows an example of a poset of subscriptions. Note that S and A are transitive relations, while the diagram and its S A representation in memory store immediate relationships only. In a poset PS , ordered according to S , a lter f 1 is an immediate predecessor of another lter S f 2 and f 2 is an immediate successor of f 1 if and only if f 1 S f 2 and there is no S other lter f 3 in PS such that f 1 S f 3 S f 2 . The top-level lters, which we S S refer to as roots, are those that have no successors in the poset. Inserting a new lter f into a poset, three different cases apply that are of special interest for the forwarding algorithms: f is added as a root lter; f exists already in the poset; or f is inserted somewhere in the poset with a nonempty set of successors. As we detail below, only root lters produce network trafc, due to the propagation of subscriptions (or advertisements). Thus the shape of a subscription (or advertisement) poset roughly reects the degree of opportunity presented to our processing strategies. In particular, a poset that extends vertically indicates that subscriptions are very much interdependent and that there are just a few subscriptions summarizing all the other ones. Conversely, a poset that extends horizontally indicates that there are few similarities among subscriptions and that there are thus few opportunities to reduce network trafc. 5.2.2 Hierarchical Client/Server Architecture. A hierarchical server maintains its subscriptions in a poset PS . Each subscription s in PS has an associated set called subscribers(s) containing the identities of the subscribers of that lter. Every server also has a variable master, possibly null, containing the identity of its master server.
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
353
5.2.2.1 Subscriptions. Upon receiving a simple subscription subscribe(X , f ), a server E walks through its subscription poset PS , starting from each root subscription, looking for a lter f that covers the new lter f and that contains X in its subscribers set: f S f X subscribers( f ). If the server nds such a S subscription f in PS , it simply terminates the search without any effect. This happens when the same interested party (X ) has already subscribed for a more generic lter ( f ). In case the server does not nd such a subscription, the search process terminates producing two possibly empty sets f and f , representing the immediate successors and the immediate predecessors of f , respectively. If f = f = { f }, i.e., if lter f already exists in PS , then the server simply inserts X in subscribers( f ). Otherwise, f is inserted in PS between f and f , and X is inserted in its subscribers set. Only if f = , i.e., only if f is inserted as a root subscription, does the server then forward the same subscription to its master server. In particular, if master is not null, the server (E) sends a subscription subscribe(E, f ) to master. If f = , the server removes X from the sets of subscribers of all the subscriptions covered by f . This is done by recursively walking breadth rst through the poset PS starting from the subscriptions in f . The recursion is stopped whenever X is found in a subscription (and removed). Note that in this process some subscriptions might be left with no associated interested parties; such subscriptions are removed from PS . We illustrate the processing of subscriptions in the hierarchical architecture with the scenario depicted in Figure 13. Figure 13(a) depicts a hierarchical server (1) that has two clients (a and b) and a master server (2). The server receives and processes a subscription [airline = UA] from client a. The right side of the gure shows the subscription poset PS of server 1. The new subscription is inserted as a root subscription, so server 1 forwards it to its master server (2). In Figure 13(b), server 1 receives another subscription [airline = UA, dest = DEN] from client b. Since this new subscription is already covered by the previously forwarded subscription (it is not made a root subscription in PS ), server 1 does not forward it to its master. In Figure 13(c), server 1 processes another subscription [airline = any] from client a. This is a root subscription, so it is forwarded to server 2. In this case, server 1 eliminates a from the subscribers of all the subscriptions covered by the new one. In particular, it removes a from the rst subscription [airline = UA]; because there are no other subscribers for that subscription, the subscription itself is also removed. 5.2.2.2 Notications. When a server receives a notication n, it walks through its subscriptions poset PS breadth rst looking for all the subscriptions matching n. In particular, the server initializes a queue Q with its root N subscriptions. Then, the server iterates through each element s in Q. If n S s, the server appends to Q all the immediate predecessors of s that have not yet N been visited. Otherwise, if n S s, the server removes s from the queue. When this process terminates, Q contains all the subscriptions that cover n. The server then sends a copy of n to each subscriber of the subscriptions in
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
354
A. Carzaniga et al.
(a)
(b)
(c)
Fig. 13. Example scenario using a hierarchical client/server architecture (subscription).
Q. Independently of the matching of subscriptions, if the server has a master server and the master server was not the sender of n, then the server also sends a copy of n to its master server. 5.2.2.3 Unsubscription. Unsubscriptions cancel previous subscriptions, but they are not exactly the inverse of subscriptions. They are slightly more complex to handle and sometimes more expensive in terms of communication. One reason is that a single unsubscription might cancel more than one previous subscription. The other reason is that an unsubscription might cancel one or more root subscriptions, which in turn might uncover other more specic subscriptions (which in turn become new root subscriptions). In this case, the server
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
355
Fig. 14. Example scenario using a hierarchical client/server architecture (unsubscription).
must forward the unsubscription to its master server, but it must also forward the new root subscriptions as well. More specically, when a server receives an unsubscription unsubscribe(X , f ), it removes X from the subscribers set of all the subscriptions in PS that are covered by f . The algorithm used by the server in this case is a simple variation of the algorithm that computes the set of matching subscriptions for a notication. The only difference is that the relation S is used to ll the S N queue instead of S . As a consequence of removing X , some subscriptions might remain with an empty set of subscribers. Let S X be the set of such subscriptions, and let S r X (S r S X ) be the set of those that are also root subscriptions in PS . The server X computes S r as the union of all the immediate predecessors of each subscription X in S r . With all this, the server X (1) removes all the subscriptions in S X from PS , (2) forwards the unsubscription for f to its master server, and (3) sends all the subscriptions in S r to its master server. X Figure 14 continues the scenario from Figure 13(c). Server 1 receives an unsubscription for [airline = any] from client a. As a consequence, it removes a from the subscribers of subscription [airline = any], which is the only subscription from a covered by the unsubscription (in this case, the two lters coincide). The subscription contains no more subscribers, so the server removes it. But since it was also a root subscription, the server forwards the unsubscription to its master along with the new root subscription, [airline = UA, dest = DEN]. 5.2.2.4 Advertisements. The advertisement-forwarding technique does not apply to the hierarchical architecture. Although it would be possible to propagate advertisements from a server to its master, this would be useless, since the master server would never respond by sending back subscriptions. In fact, a hierarchical server considers all its clients as normal clients (i.e., outside the event notication service), so it would not forward subscriptions to them. In practice, advertisements and unadvertisements are silently dropped.
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
356
A. Carzaniga et al.
5.2.3 Peer-to-Peer Architectures with Subscription Forwarding. In peer-topeer architectures, each server maintains a set neighbors containing the identiers of the peer servers to which the server is connected. A peer-to-peer server also maintains its subscriptions in a poset PS that is an extension of the subscription poset of a hierarchical server. As in the hierarchical server, a peerto-peer server associates a set subscribers(s) with each subscription s, and it associates an additional set with s called forwards(s), which contains the subset of neighbors to which s has been forwarded. 5.2.3.1 General vs. Acyclic Architectures. A subscription or notication is propagated from its origin to its destination following a minimal spanning tree. In an acyclic peer-to-peer architecture the path that connects any two servers (if it exists) is unique, and any such spanning tree coincides with the whole network of servers. Thus, when propagating a message m, say a subscription, a server simply sends it to all of its neighbors excluding the sender of m. Any server that propagates m is considered to be a sender of m, but the origin of a message is the (unique) event notication service access point to which the message is originally posted. In a general peer-to-peer architecture, two servers might be connected by two or more different paths. So when a server receives a message that must be forwarded throughout the network of servers, the rst server must make sure to forward it only through the links of the minimal spanning tree rooted in the origin of that message. This is similar to the well-known problem of broadcasting information over a packet-switched network. In order to simplify the description of the algorithms, we focus only on acyclic peer-to-peer architectures; algorithms for the general peer-to-peer architectures can be found elsewhere [Carzaniga 1998]. 5.2.3.2 Peer Connection Setup. A server E1 connects to a server E2 by sending a peer connect(E1 ) request to E2 . E2 can either accept or refuse the connection. In case E 2 accepts E1 as a peer, E2 sends a conrmation message back to E 1 so that both servers add each others address to their neighbors set. Then the accepting server E2 forwards every root subscription in its subscriptions poset PS to the requesting server E1 , adding E1 to the corresponding forwards set. Servers can also be dynamically disconnected with a peer disconnect(E1 ) request. When a server E2 receives a peer disconnect(E1 ), it removes E1 from its neighbors set, unsubscribes E1 for all its root subscriptions, and nally removes E1 from all its forwards sets. 5.2.3.3 Subscriptions. The algorithm by which a peer-to-peer server processes subscriptions is an extension of the algorithm of the hierarchical server. When a server receives a subscription subscribe(X , f ), it searches its subscriptions poset PS for either (1) a subscription f that covers f and has X among its subscribers: f S f S X subscribers( f ). In this case, the search terminates with no effect; or (2) a subscription f that is equal to f and does not have X among its subscribers: f S f f S f . Here the server adds X to subscribers( f ); or S S
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
357
(3) two possibly empty sets f and f , representing the immediate successors and the immediate predecessors of f respectively. Here the server inserts f as a new subscription between f and f , and adds X to subscribers( f ). In cases 2 and 3, the server also removes X from all the subscriptions in PS that are covered by f , and then removes from PS those subscriptions that have no other subscribers. This procedure differs from the corresponding procedure of the hierarchical server in how the peer-to-peer server forwards the subscription to its neighbors. Formally, given a subscription f in PS , let forwards( f ) be dened as follows: forwards( f ) = neighbors NST( f )
f PS f
S S
forwards( f )
f
(1)
In other words, f is forwarded to all neighbors of the server except those not downstream from the server along any spanning tree rooted at an original subscriber of f (the second term in the formula), and those to which subscriptions f covering f have been forwarded already by this server (the last term in the formula). The second term in the formula (whose functor stands for Not on any Spanning Tree) accounts for the fact that there may be multiple paths connecting a subscriber to potential publishers, and that therefore the propagation of a subscription f must follow only the computed spanning trees rooted at the original subscribers of f . Viewing a spanning tree rooted at f as a directed graph, we may refer to paths traveling away from f as going downstream with the edges, and those traveling toward f as going upstream against the edges. In practice, the propagation process excludes those neighbors that are not downstream from the server of interest along any spanning tree rooted at a subscriber of f . NST( f ) is trivially computed for the topology of the acyclic architecture, since every spanning tree in the topology coincides with the whole topology itself. For the topology of the general architecture its computation is more complicated; however, the necessary techniques, such as link-state or distance-vector routing algorithms, are well-known and widely deployed. An alternative approach to propagating subscriptions is to use Dalal and Metcalfes broadcasting algorithm [Dalal and Metcalfe 1978]. The last term in the formula represents an important optimization that the server makes in the situation where more generic subscriptions have been propagated already to some neighbors. We illustrate the processing of subscriptions in the acyclic peer-to-peer with architecture the scenario depicted in Figure 15. Figure 15(a) shows a fragment of a peer-to-peer event notication service. In this example, server 1 is connected to servers 2, 3, and 4. Server 1 also has a local client a. Server 3 sends a subscription [airline = any] to server 1. The poset shown on the right side of the gure represents the subscription poset PS of server 1. As shown in the gure, the new subscription is inserted as a root subscription in PS and then forwarded to servers 2 and 4 but not to server 3, which is in the NST set of the subscription. In this gure and the following ones, for each subscription in PS , subscribers are denoted with an outgoing arrow from the subscription, while
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
358
A. Carzaniga et al.
(a)
(b)
(c)
Fig. 15. Example scenario using an acyclic peer-to-peer architecture.
forwards are denoted with an incoming arrow. Intuitively, arrows indicate the direction of notications. Figure 15(b) shows the effect of a second subscription [airline = UA, orig = DEN] sent to server 1 by server 2. This subscription is inserted in PS as an immediate predecessor of the previous (root) subscription and is forwarded to server 3, which is the only neighbor that is not in the forwards set for the covering subscription [airline = any]. In Figure 15(c), client a subscribes for [airline = any]. In this case the subscription is found in PS , and a is simply added to its subscribers set. Because the
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
359
NST set for that subscription is now empty, the subscription is then forwarded to server 3. Every time the server forwards a subscription f to a neighbor server E2 , it adds E2 to the forwards set of f and consequently removes E2 from the forwards sets of all the subscriptions covered by f . In the example, the server removes 3 from the forwards set of subscription [airline = UA, orig = DEN]. 5.2.3.4 Unsubscriptions. An unsubscription has the effect of removing a subscriber from a number of subscriptions in PS . More specically, when a server E receives an unsubscription unsubscribe(X , f ), it removes X from the subscribers set of every subscription covered by f . As a consequence of these cancellations, some subscriptions might remain with an empty subscribers set; such subscriptions are removed from PS . The removal of X from some subscriptions might also affect the NST set of those subscriptions. In particular, removing a subscriber for a subscription means removing its distribution spanning tree, which in turn might add some neighbor servers to NST for those paths that are not on the spanning tree of any other subscriber (see Eq. (1)). In order to reduce the forwards set of those subscriptions according to Eq. (1), the server forwards the corresponding unsubscriptions to every neighbor server added to NST. The reduced forwards sets of some subscriptions might affect the forwards sets of other covered subscriptions. This effect is produced by the last term of Eq. (1) for the covered subscriptions. Intuitively, this means that after unsubscribing for some more generic subscriptions it might be necessary to (re)forward some more specic subscriptions whose propagation was blocked by the existence of the more generic subscriptions. We illustrate the processing of unsubscriptions in the acyclic peer-to-peer architecture with the scenario depicted in Figure 16. Figure 16(a) depicts the subscriptions poset of server 1 from Figure 15(c) after it has received some subscriptions from local clients and neighbor servers. This is the state of server 1 just before it receives an unsubscription lter [airline = any] from client a. As a rst step in processing the unsubscription from client a, server 1 removes the subscriber (i.e., a) from all the subscriptions covered by the unsubscription lter [airline = any]. Figure 16(b) shows the subscriptions poset PS in this state. Two (root) subscriptions are affected: subscription [airline = UA] changes its NST set (which is initially empty) to include server 3, while subscription [airline = AZ] remains with an empty subscribers set. As a consequence, server 1 forwards the rst unsubscription [airline = UA] to the neighbor server added to the NST set (i.e., 3) and forwards the second unsubscription [airline = AZ] to all the previous forwards 2, 3, and 4. Eventually, server 1 processes the immediate predecessors of the canceled subscriptions, since their forwards might have changed as a consequence of the previous unsubscriptions. Figure 16(c) shows the state of the subscription poset at this time. The subscription [airline = UA, price < 500] must be forwarded to server 3 because its (only) successor has not been forwarded to server 3 (see Eq. (1)). Subscription [airline = UA, dest = DEN] does not need to be propagated because all the neighbor servers have received either one of its successors. Subscription [airline = AZ, price < 800] has now become a root subscription and
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
360
A. Carzaniga et al.
(a)
(b)
(c)
Fig. 16. Unsubscriptions in an acyclic peer-to-peer architecture.
thus must be forwarded to every neighbor server except those in its NST (i.e., server 3). 5.2.3.5 Notications. The algorithm for peer-to-peer architectures processes notications exactly like the one that operates on the hierarchical architecture. So, a subscription n is forwarded to every subscriber of s for every s that covers n. 5.2.4 Advertisement Forwarding. With the subscription-forwarding algorithm presented in the previous sections, we have described almost everything needed to implement an advertisement-forwarding algorithm. In fact, we can exploit the duality between subscriptions and advertisements to transpose
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
361
the subscription-forwarding algorithm to advertisement forwarding. To some extent, if we read the description of the subscription-forwarding algorithm, replacing the terms regarding subscriptions with the corresponding terms regarding advertisements, and replacing the terms regarding notications with the corresponding terms regarding subscriptions, we obtain an almost exact description of the advertisement-forwarding algorithm. The main difference with respect to the subscription-forwarding structure is that there are actually two interacting computations; one realizes the forwarding of advertisements while the other realizes the forwarding of subscriptions. Both computations have similar data structures and similar algorithms, equivalent to the ones described above. In particular, the server has a poset of advertisements PA , ordered according to the relation A , as well as a poset of A subscriptions PS . In PA , each advertisement a has an associated set of identities advertisers(a) and another set of identities forwards(a). These two computations interact in the sense that advertisement forwarding constrains subscription forwarding. For instance, in maintaining PS , when processing a subscription s, the server does not use the global set neighbors, but instead uses a subset neighborss neighbors that is specic to s. neighborss is dened as the set of advertisers listed in PA for all the advertisements covering s. Formally: neighborss =
aTA : s
S A
advertisers(a) neighbors
a
Note that one effect of this constraint is that new advertisements and unadvertisements are viewed by the subscription-forwarding computation as new peer connections or dropped peer connections, respectively. Thus, if the server receives a new advertisement that covers a set of subscriptions s1 , s2 , . . . , sk , then the server reacts by forwarding s1 , s2 , . . . , sk immediately to the sender of the advertisement. 5.3 Matching Patterns So far we have seen how simple subscriptions and simple notications are handled by event servers. A major additional functionality provided by SIENA is the matching of patterns of notications. This functionality is implemented with distributed monitoring following the upstream evaluation principle set forth in Section 5.1. To match patterns, servers assemble sequences of notications from smaller subsequences or from single notications. Thanks to advertisements, every server knows which notications and which subpatterns may be sent from each of its neighbors, which is why this technique requires an advertisement-based semantics. In addition to the notications and patterns available from its neighbors, a server might further use patterns from previous subscriptions that the server itself is already set up to recognize. We use the term pattern factoring to refer to the process by which the server breaks a compound subscription into smaller compound and simple subscriptions. After a subscription has been factored into its elementary components, the
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
362
A. Carzaniga et al.
Table IV. Example of a Table of Available Patterns Pattern a1 a2 string alarm = failed-login integer attempts > 0 string le = any string operation = le-change Providers 3 2,3
server attempts to group those factors into compound subscriptions to forward to some of its neighbors. This process is called pattern delegation. 5.3.1 Available Patterns Table. Every server maintains a table TP of available patterns. This table is simply the advertisements poset PA that, in addition to the usual advertisements, contains also those patterns that the server has already processed. Each pattern p in TP has an associated set of identities providers( p) that contains all the peer servers from which p is available. Table IV shows an example of a table of available patterns. The table says that notications matching lter a1 notications that signal a failed login with an integer attribute named attemptsare available from server 2, and that notications matching lter a2 le modication noticationsare available from servers 2 and 3. 5.3.2 Pattern Factoring. Let us suppose a server E receives a compound subscription subscribe(X , s), where s = f 1 f 2 . . . f k . Now, the server scans s trying to match each f i with a pattern pi , or trying to match a sequence of lters f i f i+1 . . . f i+ki with a single compound pattern pi...i+ki using patterns p that are contained in TP . For example, assuming the table of available patterns shown in Table IV, suppose server 1 receives a subscription s = f g h for a sequence of two failed login alarms with one and two attempts respectively ( f = [alarm = failed-login, attempts = 1], and g = [alarm = failed-login, attempts = 2]), followed by a le modication event on le /etc/passwd (h = [le = /etc/passwd, operation = lechange]). In response to s, the server factors s, matching the three lters of s with the sequence of available patterns a1 a1 a2 . Table V shows the subscription and the factoring computed by the server. Because the only operator in SIENA for combining subpatterns is the sequence operator, the output of the factoring process is always a sequence. 5.3.3 Pattern Delegation. Once a compound subscription is divided into available parts, the server must (1) send out the necessary subscriptions to collect the required subpatterns and (2) set up a monitor that will receive all the notications matching the subpatterns and will observe and distribute the occurrence of the whole pattern. In deciding which subscriptions to send out, the server tries to reassemble the elementary factors in groups that can be delegated to other servers, thereby implementing the upstream evaluation principle. The selection of subpatterns that are eligible for delegation follows some intuitive criteria. For example, only contiguous subpatterns available from a
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
Table V. Example of a Factored Compound Subscription Requested string alarm = failed-login integer attempts = 1 string alarm = failed-login integer attempts = 2 string le = /etc/passwd string operation = le-change Available string alarm = failed-login integer attempts > 0 string alarm = failed-login integer attempts > 0 string le = any string operation = le-change (a1 ) (a1 ) (a2 )
363
single source can be grouped and delegated to that source. A complete discussion of these criteria is presented elsewhere [Carzaniga 1998]. In the example of Table V, server 1 would group the rst two lters a1 a1 and delegate the subpattern dened by the corresponding two subscriptions ( f g ) to server 2. Thus, it would send a subscription subscribe(E, s1 ) with pattern s1 = string alarm = failed-login string alarm = failed-login integer attempts = 1 integer attempts = 2
to server 2, and would send the remaining lter h using a simple subscription subscribe(E, s2 ) with s2 = string le = /etc/passwd string operation = le-change
to servers 2 and 3. Server 1 will then start up a monitor that recognizes the sequence (s1 = f g ) (s2 = h). Figure 17 depicts an example that corresponds to the tables and subscriptions discussed above. In particular, server 1 delegates f g to server 2, subscribes for h, and monitors ( f g ) h. The diagram also shows how server 2 handles the delegated subscription. Assuming that f is available from server 5 and g is available from server 4, server 2 sends the two corresponding subscriptions to 4 and 5 and then starts up a monitor for f g . 6. EVALUATION Substantiating claims of scalability for an event notication service is a difcult challenge. In particular, how does one demonstrate its ability to scale, when fully doing so would require the deployment of an implementation of the service to thousands of computers across the world? Conceding to pragmatics, our approach is to build an argument based on (1) reasoning qualitatively about the rationale for the expressiveness of the notication selection mechanism, (2) performing simulation studies to determine the relative performance of the various architectures under certain hypothetical usage scenarios, and (3) constructing a prototype implementation of the service as a proof of concept. This section presents each of these three elements of the argument. Our conclusion is, that while more study is required to fully validate the design, the early evidence strongly suggests that we have achieved our goal of developing an event notication service suitable for use at the scale of a wide-area network.
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
364
A. Carzaniga et al.
Fig. 17. Pattern monitoring and delegation.
6.1 Rationale for Chosen Expressiveness The interface to the SIENA event notication service is a tailored application of the basic publish/subscribe protocol. Certain factors affecting the scalability of our design (such as network latency and data structure size) are intrinsic to the service and its use, and hence are beyond our control. The key factors we can control are the denitions of notications, lters, and patterns, and the complexity of computing the covering relations. Our chosen level of expressiveness in SIENA represents a compromise, at which notication structure, attribute types, and attribute operators approximate those of the well-understood and widely used database query language SQL. The covering relations are well behaved and predictable in the sense that they exhibit an arguably reasonable computational complexity deriving from the expressiveness of lters: assuming a brute-force and unoptimized algorithm, the complexity of determining whether a given subscription and a given N notication are related by S is O(n + m), where n is the number of attribute constraints in the subscription lter and m is the number of attributes in the N notication. The complexity of computing S reects the computation of an intersection between the attribute values in a notication and constraints on those values appearing in a subscription. The complexity of each individual comparison is O(1) for all the predened types we have included in SIENA. The only exception is the string type, but efcient comparison algorithms are well known. The complexities of computing S , A , and S are all O(nm), where n and S A A m represent the number of attribute constraints appearing in the respective subscription and/or advertisement lters. This complexity represents a comparison between each attribute constraint in one lter and any corresponding attribute constraints in the other lter. Checking a covering relation between lters amounts to a universal quantication. But given our choice of types and operators, comparing a pair of attribute constraints can be reduced to evaluating an appropriate predicate on the two constant values of the constraints, with a complexity O(1). For example, to see if [x > k1 ] covers [x > k2 ] we can simply verify that k2 k1 .
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
365
We also restrict the expressiveness of patterns in SIENA in the interests of efciency. Patterns, as we discuss in Section 3.3, are a simple sequence of lters. The computational complexity of matching a pattern is O(l (n + m)), where l is the length of the pattern. This means that it is linear in the number of lters, N whose covering relation S has complexity O(n + m). Our conclusion from this analysis is that the covering relations exhibit a complexity that is quite reasonable for a scalable event notication service. In fact, the factors n and m are, in practice, likely to be relatively small (typically less than 10), making the computations negligible compared to the network costs they are attempting to reduce. This is all achieved with an expressiveness that approximates SQL. 6.2 Simulation Studies There are many questions that one could ask about a wide-area event notication service. In our initial simulation studies, we have concentrated on the particular question of scalability with respect to the architectures and algorithms described in the previous sections. 6.2.1 Simulation Framework. The simulation framework we use consists of two parts: (1) a conguration of servers and clients mapped onto the sites of a wide-area network and (2) an assignment of application behaviors to objects of interest and interested parties. A site represents a subnet and its possibly many hosts, where the cost of communication between hosts within a site is assumed to be zero. The conguration of servers reects the choice of the event notication service architecture, while the application behaviors involve the basic service requests of advertise/unadvertise, subscribe/unsubscribe, and publish. The primary measurement of interest is an abstract quantity we refer to as cost. We assign a relative cost to each site-to-site communication in the network and then calculate the effect on this cost of varying a number of simulation parameters. In other words, we evaluate the architectures and algorithms in terms of the communication induced by the application behaviors, since we are interested in characterizing the degree to which each architecture/algorithm combination can or cannot absorb increased communication costs in the face of increasing application demands. 6.2.1.1 Network Conguration. Figure 18 shows the layered structure of a network conguration in our simulation framework. At the bottom level is a model of a wide-area network topology. This model denes sites (depicted as cubes) and links (depicted as heavy lines between cubes). To develop realistic network topologies that account for important properties such as latency, and approximate the relative costs of real wide-area networks, we use a publicly available generator of random network topologies2 that implements the Transit-Stub model [Zegura et al. 1996]. (A discussion of this and other models
2 Georgia
Tech Internet Topology Models (GT-ITM).
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
366
A. Carzaniga et al.
Fig. 18. Layers in a network conguration.
for generating network topologies can be found elsewhere [Zegura et al. 1997].) The results we present here were performed on networks of 100 sites. We make several simplifying assumptions about sites and links at this level. First, we assume that the costs of computation at sites and communication through links are linear functions of load. Second, links have constant latency. Third, sites and links have innite capacity. Note that one consequence of these assumptions is that our model does not account for the effect of congestion. At the top level of the network conguration is a model of an event notication service topology. This model denes the servers (depicted as shaded ovals) and clients (depicted as white ovals), with the interconnection among servers resulting from a choice of architecture (depicted as heavy lines between servers), the assignment of a client to a server (depicted as a dotted arrow), and the mapping of clients and servers onto sites in a wide-area network (depicted as dashed lines from ovals to cubes). As a simplication, the simulations we present here involve only homogeneous architectures, and not the hybrid architectures that are also possible (see Section 4.4). Moreover, each client represents only either an object of interest or an interested party, although in general it is possible for a client to be both. Finally, we allocate one server per site; we congure every server to connect to the servers residing at its neighbor sites in the network topology; and we congure every client to use a server at its (local) site. In other words, we assume that the locations and interconnections among servers are an image of the underlying network topology. This assumption signicantly reduces the parameter space in the simulation. Nonetheless, this is a reasonable assumption, since it reects the structure of domains that characterize the Internet [Clark 1989]. In addition to simulating the various multiserver architectures, we simulate a single-server, centralized architecture, which serves as a baseline for our comparisons; where the centralized architecture performs as well as or better than the others, it should be chosen simply because of its simplicity. Of course, the
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
367
centralized architecture requires no forwarding algorithm, since it comprises a single server. 6.2.1.2 Application Behavior. The behavior of an application using the event notication service involves the collective behaviors of its objects of interest and interested parties. These individual behaviors are specied as sequences of service requests. In particular, an object of interest executes m sequences
n times
advertise, publish, publish, . . . , unadvertise and within each cycle publishes n notications. In addition, an average delay between publish requests, t, can be specied (with the delays generated according to a Poisson distribution). Similarly, an interested party executes p sequences
q times
subscribe, recv notif, recv notif, . . . , unsubscribe where recv notif represents the operation of waiting for and then receiving a notication. 6.2.1.3 Scenario Generation. Input to our simulation tool is generated in a two-step process. In the rst step a (random) network topology is generated, as discussed above. In the second step the generated topology is combined with a scenario parameter le that species both the event notication service topology and the application behavior. 6.2.2 Results. The space of studies made possible by our simulation framework is quite extensive. Here we explore a portion of that space, focusing on usage scenarios that distinguish four basic architecture/algorithm combinations: centralized, hierarchical client/server with subscription forwarding, acyclic peer-to-peer with subscription forwarding, and acyclic peer-to-peer with advertisement forwarding.3 Furthermore, we are only simulating the objects of interest and interested parties associated with one particular kind of event. Because the main purpose of the simulations presented here is to highlight the relative behaviors of the architectures and algorithms, we choose not to complicate the experiments by simulating additional kinds of events (with their associated objects of interest and interested parties). While it is conceivable that this choice could affect the results in some way, our intuition tells us that our conclusions about the relative behaviors would remain the same. To reveal the scaling properties of the architecture/algorithm combinations, our approach is to keep the behaviors of objects of interest and interested parties constant while varying the number of objects of interest from 1 to 1000 and the number of interested parties from 1 to 10,000. In all cases, the number of network sites is 100. Referring to the characterization given in Section 6.2.1, we
3A
more extensive set of data plots is presented elsewhere [Carzaniga 1998; Carzaniga et al. 1999].
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
368
A. Carzaniga et al.
used parameter values of m = 10 and n = 10 for objects of interest, indicating 10 sequences of 10 iterations each, and an average interpublication delay t in the interval 20002500. The behavior parameters for interested parties was p = 10 and q = 10, indicating 10 iterations of 10 received notications each. We ran our scenarios with articially low ratios of publications-peradvertisement and notications-per-subscription in order to produce conservative simulation results. Applications ultimately benet from delivery of notications, so advertisements and subscriptions can be considered a necessary overhead to obtain that benet. Such low ratios then serve to exaggerate this overhead. In real applications, we would expect the service to deliver a much higher volume of notications (with correspondingly lower overhead) than is represented by these ratios. The results we present are all shown as plots whose data points represent the average of 10 simulation runs for the same parameter values. Except for the plots of the acyclic peer-to-peer architecture with advertisement forwarding (whose behavior is unstable and as yet inexplicable), the variance within each set of 10 simulation runs was negligible, and therefore we choose not to include error bars in the plots. For the majority of the plots, the horizontal axis gives the number of interested parties in a logarithmic scale ranging from 1 to 10,000, while the vertical axis gives a linear measure of cost. As mentioned above, the cost values are derived from an assignment of relative costs for communicating over network links. Therefore, the absolute value of a data points cost is meaningless, but its relative value gives a useful characterization. In the plots below we use the following aliases for the event notication service architecture/algorithm combinations: ce = centralized, hs = hierarchical client/server with subscription forwarding, as = acyclic peer-to-peer with subscription forwarding, and aa = acyclic peer-to-peer with advertisement forwarding. 6.2.2.1 Total Cost. A basic metric for the event notication service is the total cost of providing the service. The total cost is calculated by summing the costs of all site-to-site message trafc. The total cost captures an important aspect of scalability by revealing how communication cost is impacted by increases to the load presented to the service. Figure 19 compares the total costs incurred by the three distributed architectures with 1, 10, 100, and 1000 objects of interest; we omit the curves for the centralized architecture because its total cost far outweighs that of the distributed architectures, exhibiting exponential blowup starting at around 1000 interested parties in each plot. There are several interesting observations we can make about these plots. First, when there are more than 100 interested parties, the total cost is essentially constant, meaning that there is a point beyond which there is no additional cost incurred in delivering notications to additional interested parties. We call this the saturation point, since there is high likelihood that there is an object of interest at every site, and thus an object of interest near every interested party. (Recall that all objects of interest are publishing the same notications.)
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
369
Fig. 19. Comparison of total costs incurred by distributed architectures.
Second, all the architectures scale sublinearly when the number of interested parties is below the saturation point. When below this point, it is likely that an object of interest and an interested party are not at the same site. Therefore, it is also likely that the cost to deliver a notication is nonzero. However, as the number of interested parties grows, the likelihood increases that each new interested party will be located at a site at which there is already an interested party. The marginal cost of each additional interested party decreases up to the saturation point, where the cost becomes zero. This sublinear scaling character of the architectures can be discerned more easily in the plots of Figure 20, which present the portion of the data of Figure 19 below the saturation point using a linear scale in the horizontal axis. Third, as the number of objects of interest increases, the hierarchical client/server architecture with subscription forwarding performs worse by an increasingly large constant factor as compared to the acyclic peer-to-peer architecture with subscription forwarding. This can be attributed to the fact that, while the acyclic peer-to-peer architecture is penalized by its broadcast of subscriptions, the hierarchical client/server architecture, which propagates notications only toward the root of the hierarchy, is forced to do so whether or not interested parties exist on the other side of the root of the network. This generates potentially signicant amount of trafc in unnecessary notications. Finally, the acyclic peer-to-peer architecture with advertisement forwarding displays a strikingly, and as yet inexplicable, unstable cost prole for low densities of interested parties. On the other hand, its costs essentially follow those of the acyclic peer-to-peer architecture with subscription forwarding once the saturation point is passed. This effect becomes more evident as the number of objects of interest increases. We can attribute this to our conservative choice of
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
370
A. Carzaniga et al.
Fig. 20. Comparison of total costs below the saturation point.
behavior for objects of interest in these studies. In particular, an object of interest unadvertises and readvertises quite frequently, compared to the number of publications it generates at each iteration. Overall, the acyclic peer-to-peer architecture with subscription forwarding appears to scale well and predictably under all circumstances, and thus is likely to represent a good choice to cover a wide variety of scenarios. 6.2.2.2 Cost Per Service Request. An event notication service is efcient if it can amortize the cost of satisfying newer client requests over the cost of satisfying previous client requests. This is another manifestation of the network effect. The average per-service cost is calculated by dividing the total cost, as introduced above, by the total number of client requests. A low value for this ratio indicates low overhead. Recall that for these studies we congure the network so that clients are connected to servers at their local sites, and therefore the client-to-server communication cost is treated as zero. The per-service cost thus purely reects the choice of architecture/algorithm combination. We can see several interesting things in the results of the data analysis presented in Figure 21. First, the centralized architecture is unreasonable in essentially all scenarios as compared to the other architectures. Second, advertisement forwarding again shows itself unstable for high numbers of objects of interest until the saturation point in interested parties is reached. Third, for low numbers of objects of interest and low numbers of interested parties, the costs are dominated by message-passing costs internal to SIENA; since there are relatively few notications generated in the network, there are few parties interested in receiving those notications, and there is a signicant internal cost incurred in setting up routing paths from objects of interest to interested
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
371
Fig. 21. Comparison of per-service costs.
parties. The hierarchical client/server architecture with subscription forwarding does well in this situation because subscriptions are forwarded only toward the root server, resulting in lower setup costs. However, as the number of objects of interest or the number of interested parties increases, its advantage quickly disappears, recovering only beyond the saturation point for interested parties. Finally, the acyclic peer-to-peer architecture with subscription forwarding does extremely well when there is a high number of objects of interest, independent of the number of interested parties. This effect is explained in the next series of plots. 6.2.2.3 Cost Per Subscription and Per Notication. Based on the results of studying the total and per-service cost incurred by each of the four architecture/algorithm combinations, the hierarchical client/server architecture and acyclic peer-to-peer architecture, both with subscription forwarding, appear to be the two most promising choices. However, they are clearly distinguished if we examine which kind of service request each one favors for its optimizations. The average per-subscription cost is calculated by dividing the total cost of all subscription-related messages by the number of subscriptions processed. The graph of Figure 22 shows the per-subscription cost incurred by the hierarchical client/server architecture and acyclic peer-to-peer architecture with a single object of interest. In trying to understand the different cost drivers of the two architectures, we simulated several scenarios with a single object of interest while varying only the behavioral parameters. In these cases, we observed no signicant variation in cost. However, additional simulations, in which we varied the density of interested parties, highlight the difference between the two architectures. The results of these simulations are presented in Figure 22 and
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
372
A. Carzaniga et al.
Fig. 22. Comparison of per-subscription costs for hierarchical client/server and acyclic peer-to-peer architectures under varying numbers of interested parties.
Fig. 23. Comparison of per-notication costs for hierarchical client/server and acyclic peer-to-peer architectures under varying numbers of interested parties.
reveal that the costs are primarily dependent on the density of interested parties. In particular, the per-subscription cost is evidently higher for the acyclic peer-to-peer architecture than the hierarchical client/server for low densities of interested parties, while both architectures benet from increasing densities of interested parties. The main difference is in the way each architecture forwards subscriptions. In the acyclic peer-to-peer architecture, a subscription must be propagated throughout the network; in a network of N sites, a subscription goes through O(N ) hops, and therefore the cost is O(N ). The hierarchical client/server architecture, on the other hand, requires that a subscription be forwarded only upward toward the root server; in this case the number of hops and hence the cost are both O(log N ). The acyclic peer-to-peer architecture recoups its greater setup costs for subscriptions by reducing the average cost of notications. Figure 23 compares the per-notication costs incurred by the acyclic peer-to-peer architecture and the hierarchical client/server architecture with a single object of interest. In the particular scenario of Figure 23, the difference between the per-notication costs in the two architectures is constant with respect to the number of interested parties. The same difference is clearly visible from the global per-service costs shown in Figure 21. We observe that this constant bracket depends on the number of ignored notications. In many of the scenarios we simulated, the total number of notications produced by objects of interest exceeds the number of notications
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
Wide-Area Event Notication Service
373
Fig. 24. Comparison of per-notication costs for hierarchical client/server and acyclic peer-to-peer architectures under varying numbers of publications in two degenerate cases.
consumed by interested parties. For example, in the scenario represented by the results presented here, a total of 100,000 notications are produced, while each interested party consumes 100 notications before terminating, leaving 99,900 ignored notications. The cost of ignored notications is clearly shown by the degenerate scenarios of Figure 24, in which per-notication costs are plotted against the number of notications published. The average per-notication cost is calculated by dividing the total cost of all notication-related messages by the number of notications processed. The rst scenario has one object of interest that emits a varying number of notications and no interested parties at all. Here the hierarchical client/server architecture incurs a constant cost due to the fact that every notication must be propagated toward the root of the hierarchy, whereas the acyclic peer-to-peer architecture incurs no cost at all, since every notication remains local to its access server. The second scenario has one interested party that consumes exactly one notication and then terminates. Again, in the hierarchical client/server architecture, the per-notication cost is constant, while the acyclic peer-to-peer architecture incurs an initial cost for the rst notication that is subsequently amortized by the zero cost of the subsequent ignored notications. 6.2.2.4 Worst-Case Per-Site Cost. A critical issue in a communication network is how it behaves in the face of congestion. In the simulation model presented here, however, we cannot directly simulate congestion (see Section 6.2.1). Nevertheless, we can ask a related question that gives us an indication of which architecture would approach a given level of congestion soonest. The metric we use is the worst-case per-site cost. It is calculated, for each scenario, by averaging the cost of communication incurred by each site over the 10 simulations of that scenario, and then by computing the maximum over those average per-site costs. The plots of Figure 25 show the maximum cost incurred by a site in the hierarchical client/server and acyclic peer-to-peer architectures under the same scenarios of Figure 19 and Figure 20. The hierarchical client/server architecture exhibits slightly lower worst-case per-site cost under low densities of objects of interest. For high densities of objects of interests, and therefore under high volumes of notications, the hierarchical client/server architecture incurs a
ACM Transactions on Computer Systems, Vol. 19, No. 3, August 2001.
374
A. Carzaniga et al.
Fig. 25. Comparison of worst-case per-site costs.
much greater maximum per-site cost than the acyclic peer-to-peer architecture. In other words, a high volume of notications is likely to cause congestion to be reached sooner in the hierarchical client/server architecture than in the acyclic peer-to-peer architecture. How do we explain this difference in behavior? We can see that it is not attributable to a simple overloading of the root server in the hierarchical client/server architecture. This, perhaps counterintuitive, observation is based on the fact that the acyclic peer-to-peer architecture gains advantage only when, for a given object of interest, there is no interested party on the opposite side of the network, and hence the trafc remains within a localized neighborhood of the object of interest. This situation rarely arises un...