of Applications MultiAgent Learning in ECommerce and Autonomic Computing
Jeff Kephart IBM Research
[e-mail]
December 14, 2002
Two broad application areas
Ecommerce
Largescale competitive MAS Billions of economically motivated agents http://www.research.ibm.com/infoecon
Buying and selling information goods and services Adaptive, and coupled directly and indirectly (through markets)
Autonomic computing
Largescale cooperative or competitive MAS Selfmanaging computing systems http://www.research.ibm.com/autonomic IEEE Computer, January 2003
Selfconfiguring, Selfhealing, Selfoptimizing, Selfprotecting
The Future Information Economy
$
RoboBob RoboBob Bob e-utility e-utility 2
$
e-News e-News
Bidding Service 1
$
Bidding Service 2
$
NYSE
$
RoboXYZ RoboXYZ XYZ
$ $
NY Times
eBay e-Marketplace Service Match Match Maker 1 UDDI cXML cXML ebXML
Service Service Match e-Textiles Maker e-Marketplace 2 2 BBB $ $ Reputation $ RoboButtons BizRate BizRate
$
$
$
Textiles Ontology Ontology Translator Translator
Buttons 'R Us
$
ASP 2 2
e-utility $ 1 Translator $ B2B $ B2B Protocol Protocol Matcher Matcher
$
ASP 1 ASP
Billions of interacting, adaptive agents. What emergent behaviors will arise?
Experiment
Bidding Agents vs. Humans
Continuous Double Auction
100
Market History
Trade Ask Bid
Price
75 50 25 0 0 2 4 6 8 10
Time
CDA is common in financial markets Extensive prior literature All-human experiments (Vernon Smith) All-agent experiments (SFI DA, Gode-Sunder, Cliff, TAC)
Watson Experimental Economics Lab
Copyright New York Times
Limit Prices
Submit
Seller GUI (CDA)
Ask Queue
Bid Queue
Bidding Agent Architecture
Agent GUI
Auctioneer
Auction Info Orders
Message Handler Brain
Wakeup? Compute Order Place Order
Bookkeepe r
Market state Agent state
Bidding Agent
AgentHuman experiments
Human subjects
Setup
recruited from local colleges and IBM Research given interactive instructions and test paid in proportion to surplus 6 Humans, 6 Agents 6 Buyers, 6 Sellers Each agent shares limit prices with a human 9 to 16 3minute periods Limit prices change every 35 periods
Experiment
Experiment #6: Fast GD vs. Humans
Summary of experimental results
Agents won by substantial margins in all experiments ~20% more surplus than novice humans ~57% more surplus than experienced humans Agents and humans interact with one another Not two decoupled markets ~3050% of trades are agenthuman Market efficiency improves with number of agents Humans fare better when there are more agents
Agents can supplant humans as economic decision makers
Allagent experiments
Simulator
Market parameters
Discretetime; stochastic asynchronous dynamics. Ran mixtures of several strategies and variants 10 buyers, 10 sellers 10 units each. Fixed limit prices (chosen randomly) 100 expts, 5 trading pds/expt, 300 time steps/pd. Homogeneous (0 A vs. 20 B) OneinMany Tests (1 A vs. 19 B) Balanced Team Tests (10 A vs. 10 B)
Experimental comparisons
Expt. 2: 1 A vs. 19 B
Differential Efficiency
ZIP and MGD invade ZI, Kaplan & GD
But don't invade one another
Kaplan can invade all strategies All strategies invade ZI ZI doesn't invade any
Expt. 3: 10 A vs. 10 B
Differential Surplus (out of 2612 total)
ZI beats Kaplan 100-0! Other strategies beat Kaplan, but by smaller margin GDX > MGD > ZIP > GD > ZI > Kaplan
Evolutionary dynamics
CDA game
What happens when agents gradually switch to more successful strategies? No strategy is dominant. This is a useful view for Mechanism design Agent design
Dynamic pricing game
Shopbots and pricebots
Amazon.com
MY Q
Borders.com
pricebots (various pricing algorithms)
2
1
2
2
1
1
users/user agents (various buyer strategies and valuations)
Dynamic Pricing
Metapayoff matrix
4 MY
.0337
4 DF
.0225
4 GT
.0109
1 MY Nash
.0337
.0690
.0185
.0361
.0387
.0159
1 DF Seller optimal 1 GT
.0136
.0387
.0134
.0169
.0226
.0129
.0119
.0536
.0129
Evolutionary dynamics
Dynamic pricing game
Computation
5 Agents
20 Agents
Dynamic pricing game
Shopbots and pricebots
Time series: cyclical price war
A cure for myopia
Compute future discounted profit, Q:
Compute pricing policy, R, that maximizes Q:
Use Q-learning, analysis to derive 2 distinct solutions
Price response curves
Symmetric and Asymmetric solutions
Higher efficiency, but unstable!
Sequential Learning
Limit Cycle
Autonomic Computing
Selfmanaging computing systems
Administration of individual systems is increasingly difficult
100s of configuration, tuning parameters for DB2, WebSphere
Heterogeneous systems are becoming increasingly connected
Integration becoming ever more difficult
Architects can't intricately plan interactions among components
Increasingly dynamic; more frequently with unanticipated components
More of the burden must be assumed at run time
But human system administrators can't assume the burden; already
6:1 cost ratio between storage admin and storage 40% outages due to operator error
We need selfmanaging computing systems
Behavior specified by sys admins via highlevel policies System and its components figure out how to carry out policies
Evolving towards Selfmanagement
Today Self configure Selfheal Self optimize Selfprotect
Corporate data centers are multivendor, multiplatform. Installing, configuring, integrating systems is time consuming, errorprone.
The Autonomic Future
Automated configuration of components, systems according to highlevel policies; rest of system adjusts automatically. Seamless, like adding new cell to body or new individual to population.
Problem determination in large, Automated detection, diagnosis, and complex systems can take a repair of localized software/hardware team of programmers weeks problems. WebSphere, DB2 have hundreds of nonlinear tuning parameters; many new ones with each release. Components and systems will continually seek opportunities to improve their own performance and efficiency.
Manual detection and recovery Automated defense against malicious from attacks and cascading attacks or cascading failures; use early failures. warning to anticipate and prevent systemwide failures.
Autonomic element
structure
Managed Manager
Fundamental element(s)
Autonomic atom of the architecture
Database, storage system, etc.
Sensors Analyze
Effectors Plan
Managed Element
Responsible for:
Plus one autonomic manager
Monitor Knowledge Sensors
Execute
Providing its service Managing its own behavior in accordance with policies Interacting with other autonomic elements
Effectors
An Autonomic Element
Autonomic elements
interaction
Relationships
Dynamic, ephemeral Formed by agreement
Full spectrum
May be negotiated Peertopeer Hierarchical
Subject to policies
AC System and Infrastructure
Sentinel Arbiter Reputation Authority Planner Registry Network
Aggregator
Server Registry
Server Broker
Arbiter
Sentinel
Event Broker Negotiator Correlator
Database Storage
Communication Naming
Monitor
Provisioner
Monitor
Autonomic Platform Services
Lifecycle Security Policy Booststrapper Human Interface
Negotiator
Server
Workload Manager
Location
Multiagent Learning scenarios
Designing system behavior Interacting feedback loops Negotiation and resource allocation Problem determination
Control & harness emergent behavior
Understand, control, and exploit emergent behavior in autonomic systems.
How do self*, stability, etc. depend on
How to invert this relationship to achieve system goals?
Behaviors and goals of the agents Pattern and type of interactions among agents External influences and demands on system
Server 2
Interacting control/optimization loops
Transaction Requests
Server 1 Increase demand
File System
DB Service
Increase service Feedback control & optimization of single autonomic elements Done for 12 variables What happens when feedback loops interact?
Storage Service 2
Storage Service 1
Server 2
Interacting control/optimization loops
Transaction Requests
Server 1 Increase demand
File System
DB Service
Capacity limit reached: Get more storage
X
Storage Service 2
Storage Service 1
Server 2
Interacting control/optimization loops
Transaction Requests
Server 1 Demand not being met: Find alternate supplier
File System
DB Service
Getting more storage
X
Storage Service 2
Storage Service 1
Server 2
Interacting control/optimization loops
Transaction Requests
Server 1 Sorry; already found an alternative
X
File System
DB Service
Ready to give you that extra service
Storage Service 2
Storage Service 1
Negotiation and resource allocation
Server 2 Server 1 Transaction Requests
Policies: utility functions
Request( QueryService, Queries = 800/sec, Type = 2, RT = 5 sec) Request( QueryService, Queries = 400/sec, Type = 5, RT = 3 sec) Counterpropose( QueryService, Queries = 320/sec, Type = 5, RT = 4 sec)
Compute costs, benefits from business contract, propagate them down. Forms of negotiation: Bilateral Multilateral Auction Supply chain Competitive/coop Learning During a negotiation Strategy evolution Collective behavior?
File System 1
DB 1
Request( LogicalVolume, Size = 12 Gbytes, Reads = 500/sec, Writes = 500/sec)
Request( TableSpace, Size = 3 GBytes, Reads = 2000/sec, Writes = 100/sec)
Counterpropose( TableSpace, Size = 3 GBytes, Reads = 1600/sec, Writes = 100/sec)
Storage 2
Storage 1
Problem Determination
Construct adaptive statistical models of large networked systems
Learn about interelement dependencies (within locale) Determine model structure, parameters
e.g. Bayes Net
App server Web server
Database server
Monitor logs, use model to
Challenge
Detect potential problems Set up monitors as needed Diagnose problems Shifting topology
LWS
LAS
LDBS Router1
EPP2
Analysis & Control
Probing Workstation
Console: analysis&control
Closing remarks
Ecommerce: competitive, gigaagent MA...