Structure Efcient Learning in Factored-state MDPs
Alexander L. Strehl, Carlos Diuk and Michael L. Littman
RL3 Laboratory Department of Computer Science Rutgers University Piscataway, NJ USA {strehl,cdiuk,mlittman}@cs.rutgers.edu
Abstract
We consider the problem of reinforcement learning in factored-state MDPs in the setting in which learning is conducted in one long trial with no resets allowed. We show how to extend existing efcient algorithms that learn the conditional probability tables of dynamic Bayesian networks (DBNs) given their structure to the case in which DBN structure is not known in advance. Our method learns the DBN structures as part of the reinforcement-learning process and provably provides an efcient learning algorithm when combined with factored Rmax.
Introduction
In the standard Markov Decision Process (MDP) formalization of the reinforcement-learning (RL) problem (Sutton & Barto 1998), a decision maker interacts with an environment consisting of nite state and action spaces. Learning in environments with extremely large state spaces is challenging if not infeasible without some form of generalization. Exploiting the underlying structure of a problem can effect generalization and has long been recognized as an important aspect in representing sequential decision tasks (Boutilier, Dean, & Hanks 1999). A factored-state MDP is one whose states are represented as a vector of distinct components or features. Dynamic Bayesian networks (DBNs) and decision trees are two popular formalisms for succinctly representing the statetransition dynamics of factored-state MDPs, rather than enumerating such dynamics state by state (Guestrin, Patrascu, & Schuurmans 2002). We adopt these powerful formalisms. Algorithms for provably experience-efcient exploration of MDPs have been generalized to factored-state MDPs specied by DBNs. Factored E 3 (Kearns & Koller 1999) and Factored Rmax (Guestrin, Patrascu, & Schuurmans 2002) are known to behave near optimally, with high probability, in all but a polynomial number of timesteps. Unfortunately, these algorithms require as input a complete and correct DBN structure specication (apart from the CPT parameters), which describes the exact structural dependencies among state variables.
Copyright c 2007, American Association for Articial Intelligence (www.aaai.org). All rights reserved.
There has been recent interest in learning the underlying structure of a problem from data, especially within the RL framework. Degris, Sigaud, & Wuillemin (2006) introduce a model-based algorithm that incrementally builds a decisiontree representation of state transitions. Their method, while successful in challenging benchmark domains, incorporates a very simplistic exploration technique (-greedy) that is known to miss important opportunities for exploration. In addition, it has no formal performance guarantees. Abbeel, Koller, & Ng (2006) show that factor graphs, a generalization of DBNs, can be learned in polynomial time and sample complexities. Their method, which is not based on maximum likelihood estimation, assumes access to i.i.d. samples during separate training and testing phases. While our approach is inspired by theirs, we address the RL problem, which requires dealing with highly dependent (non i.i.d.) inputs, temporal considerations, and the explorationexploitation tradeoff. Our paper makes two important contributions. First, we integrate structure learning with focused exploration into a complete RL algorithm. Second, we prove that our algorithm uses its experience efciently enough to provide formal guarantees on the quality of its online behavior. We deal almost solely with the problem of maximizing experience, rather than computational, efciency. Using supervised learning terminology as a metaphor, we seek to minimize sample rather than computational complexity. Our algorithm relies on access to an MDP planner (value iteration in our experiments), which is often very costly in terms of computation. For a more practical implementation, faster approximate planners could be employed. (For examples, see the paper by Degris, Sigaud, & Wuillemin, 2006.) In the next section, we discuss and formally describe MDPs and factored-state MDPs. Next, we introduce two common structuresdynamic Bayesian networks and decision treesfor succinct representation of the transition dynamics in factored-state MDPs. We formulate the general Structure-Learning Problem as an online learning-theory problem that involves prediction of the probability an output of 1 will be observed for a specied input. We provide the Basic Structure-Learning Algorithm as a concrete solution to a simple instance of this problem. We prove that, with high probability, the number of errors (refusals to predict) made by the algorithm is small (polynomial in the size
of the problem representation). In the following section, we return to the reinforcement-learning problem and introduce our new algorithm, SLF-Rmax. The algorithm uses the solution to the online learning-theory problem just described. We prove that the SLF-Rmax algorithm acts according to a near-optimal policy on all but a small number of timesteps, with high probability. This bound is inherently linked, through reduction, with the corresponding error bound of the online learning algorithm. We briey discuss how the Basic Structure-Learning Algorithm can be extended and incorporated into SLF-Rmax to efciently solve the general structure-learning RL problem using either the DBN or decision-tree models. Next, we perform an empirical comparison of SLF-Rmax with Rmax and Factored Rmax. These algorithms differ according to how much prior knowledge they haveRmax neither uses nor requires underlying structure; SLF-Rmax is told there is structure but must nd it itself, and Factored Rmax requires complete knowledge of the underlying structure. The performance of the algorithms reects the amount of knowledge available in that the more background provided to the algorithm, the faster it learns.
from the set D(Xi ). In other words, each state can be written in the form x = x(1), . . . , x(n) , where x(i) D(Xi ). The goal of factored representations is to succinctly represent large state spaces. The number of states of a factoredstate MDP M is exponential in the number n of state variables. To simplify the presentation, we assume the reward function is known and does not need to be learned. We also assume that each factor is binary valued (D(Xi ) = {0, 1}). All of our results have straightforward extensions to the case of an unknown reward function and multi-valued factors. Now, we make a mild independence assumption (that can be relaxed in some settings). Assumption 1 Let s, s be two states of a factored-state MDP M , and a an action. The transition distribution function satises the following conditional independence condition: T (s |s, a) = Pi (s (i)|s, a), (1)
i
where Pi (|s, a) is a discrete probability distribution over D(Xi ) for each factor Xi and state-action pair (s, a). Said another way, the DBNs have no synchronic arcs. This assumption ensures that the values of each state variable after a transition are determined independently of each other, conditioned on the previous state and action. The learning algorithms we consider are allowed to interact with the environment only through one long trajectory of (state, action, reward, next-state) tuples, governed by the system dynamics above. The transition function is not provided to the algorithm and must be learned from scratch.
Background
This section introduces the Markov Decision Process (MDP) notation used throughout the paper (Sutton & Barto 1998). Let PS denote the set of all probability distributions over the set S. Denition 1 A nite MDP M is a ve tuple S, A, T, R, , where S is a nite set called the state space, A is a nite set called the action space, T : S A PS is the transition function, R : S A PR is the reward function, and 0 < 1 is a discount factor on the summed sequence of rewards. We call the elements of S and A states and actions, respectively, and dene S = |S| and A = |A|. We use T (s |s, a) to denote the transition probability of state s in the distribution T (s, a) and R(s, a) to denote the expected value of the distribution R(s, a). A policy is any strategy for choosing actions. A stationary policy is one that produces an action based on only the current state, ignoring the rest of the agents history. We assume, without loss of generality, that rewards all lie in the interval [0, 1]. For any policy , let VM (s) (Q (s, a)) M denote the discounted, innite-horizon value (action-value) function for in M (which may be omitted from the notation) from state s. Specically, for any state s and policy , let rt denote the tth reward received after following in M starting from state s. Then, VM (s) = E[ t=0 t rt |s, ]. The optimal policy is denoted and has value functions VM (s) and Q (s, a). Note that a policy cannot have a value M greater than 1/(1 ).
Different Models
Factored-state MDPs are mainly useful when there are restrictions on the transition function that allows it to be represented by a data structure with reduced size. The corresponding goal of a learning algorithm is to achieve a learning rate that is polynomial in the representation size. Two different representations, DBNs and decision trees, are discussed in this section. As an example of the different expressive powers of these representations, we refer the reader to the taxi domain (Dietterich 2000), a grid world in which a taxi has to pick up a passenger from one of four designated locations and drop it off at a different one. The state space can be factored into 3 state variables: taxi position, passenger location and passenger destination. The dynamic Bayesian network (DBN) framework is one model commonly used to describe the structure of factoredstate MDPs (Boutilier, Dean, & Hanks 1999). This model restricts the set of factors that may inuence the value of a specied factor after a specied action. For example, it is powerful enough to capture the following relationship in the taxi domain: the factor that indicates the passenger location depends only on itself (and not, say, the destination) after a move forward command. Several learning methods have been developed for factored-state MDPs that require the DBNs structures themselves (but not the CPTs) as input (Kearns & Koller 1999; Guestrin, Patrascu, & Schuurmans 2002). Our new algorithm operates when provided
Factored-state MDPs
Denition 2 A factored-state MDP is an MDP where the states are represented as vectors of n components X = {X1 , X2 , . . . , Xn }. Each component Xi (called a state variable or state factor) may be one of nitely many values
only an upper bound on the max degree of the DBNs (equivalently, an upper bound on the number of parents of any factor). Although DBNs are quite useful, they fail to succinctly represent certain dependencies. For example, in the taxi domain, the value of the passenger variable after a drop off command is unchanged unless the taxi contains the passenger and is in the destination. The DBN representation for this dependency would indicate that the passenger variable depends on all three state variables (position, passenger, destination). On the other hand, a simple decision tree with two internal nodes can test whether the passenger is in the taxi and whether the taxi is in the destination. The decisiontree representation is an order of magnitude smaller than the DBN representation that uses tabular CPTs, in this case. Here, we allow the nodes of the decision trees to be simple decision rules of the form x(i) = z, where i {1, . . . , n} is a factor and z {0, 1} is a literal. Our algorithm has the ability to learn in factored-state MDPs whose transition functions are specied by decision trees. It needs only be given a bound on the depth of the trees.
the third condition above is called the learning complexity of the algorithm and represents the number of acknowledged mistakes (predictions of for a given input xt ) made by the algorithm. As stated, the Structure-Learning problem is solvable only by exhaustive input enumeration (wait for each of the 2n input bit vectors to be seen a sufcient number of times). However, if additional assumptions on the probability distributions Pt are made, then faster learning through generalization is possible. As one example of a problem that allows generalization, suppose that the output probability Pt depends only on the i th bit of the input xt , where the identity of the specially designated bit i is not provided to the algorithm. We call this scenario the Basic StructureLearning Problem. Later, we will discuss how our solution to this problem can be generalized to handle more than a single designated bit and to deal with the various modeling assumptions discussed in the previous section.
Basic Structure-Learning Algorithm
Our solution to the Basic Structure-Learning Problem is specically designed to be the simplest algorithm that easily generalizes to more realistic models. Intuitively, given a new input x, we must predict P (x), which depends only on x(i ), the i th component of x. If the algorithm knew which bit, i , mattered, it could easily estimate P (x) from a few sample input/output pairs whose inputs agree with x on bit i . Since i is initially unknown, our algorithm keeps empirical counts on the number of times an observed output bit of 1 occurs given an input whose ith and jth components are xed at some setting. It keeps these statistics for all pairs of bit positions (i, j) and valid settings to these two bit positions. This method is based on the observation that the correct bit position i satises P (|x(i ) = z1 , x(j) = z2 ) = P (|x(i ) = z1 , x(j ) = z2 ) for all other bit positions j and j and bits z1 , z2 , z2 . For a given input x, the algorithm searches for a bit position that approximately satises this relationship. If one is found, a valid prediction is computed based on past observations, and, if not, the algorithm predicts . Formally, our algorithm works as follows. It requires the following parameters as input: Experience threshold m Z+ . Precision parameter 1 R+ . The parameters essentially quantify the algorithms required level of accuracy. The algorithm maintains the following local variables: Position-pair counts. For every pair of distinct bit positions (i, j) {1, . . . , n}2 with i = j and every pair of bits z = (z1 , z2 ) {0, 1}2 , the quantity C(i, j, z) is the minimum of m and the number of timesteps t the algorithm has experienced an input-output pair (xt , yt ) with xt (i) = z1 and xt (j) = z2 . Next-bit counts. For every pair of distinct bit positions (i, j) {1, . . . , n}2 and every pair of bits z = (z1 , z2 ) {0, 1}2 , the quantity C(1|i, j, z) is the number of timesteps t during which the algorithm has experienced
Structure-Learning Problem
As we will show, the structure-learning problem for MDPs boils down to the following simple on-line learning-theory problem: Denition 3 (Structure-Learning Problem) On every step t = 1, 2, . . . an input vector xt {0, 1}n and output bit yt {0, 1} is provided. The input xt may be chosen in any way that depends only on the previous inputs and outputs (x1 , y1 ), . . . , (xt , yt ). The output yt is chosen with probability P (xt ), where P (xt ) depends only on the input xt . After observing xt and before observing yt , the learning al gorithm must make a prediction Pt (xt ) [0, 1] {} of P (xt ). Furthermore, it should be able to provide a predic tion Pt (x) for any input vector x {0, 1}n . We require that Pt (x) is a very accurate prediction of the probability, P (x), that y = 1 given input x. If the algorithm cannot make an accurate prediction, it must choose . Denition 4 We dene an admissible algorithm for the Structure-Learning Problem to be one that takes two inputs 0 1 and 0 < 1 and, with probability at least 1 , satises the following conditions: 1. Whenever the algorithm predicts Pt (x) [0, we 1], have that |Pt (x) P (x)| . 2. If Pt (x) = then Pt (x) = for all t > t. 3. The number of timesteps t for which Pt (xt ) = is bounded by some function (, ), polynomial in and . The rst condition above requires the algorithm to predict accurately or refrain from predicting (by choosing ). The second condition states that once a valid (= ) prediction is made for input x, then valid predictions must be provided for x in the future. This condition can easily be met by simply remembering all valid predictions. The function (, ) in
an input-output pair (xt , yt ) with xt (i) = z1 , xt (j) = z2 , yt = 1, and Ct (i, j, z) < m. During timestep t, the algorithm is provided an input xt and must make a prediction Pt (xt ). It must also be able t (x) for any bit vector x. For each to produce a prediction P queried bit vector x, our algorithm searches for a bit position i such that the following conditions hold: For all bit positions j = i, the algorithm has experienced at least m samples that agree with x in the ith and jth component. Formally, C(i, j, (x(i), x(j))) = m for all j = i. For all bit positions j = i, the number of times the agent has observed an output of 1 after experiencing an input that matches x in the ith and jth component (considering only the rst m such samples) lie within an m1 ball. Formally, |C(1|i, j, (x(i), x(j))) C(1|i, j , (x(i), x(j )))| m1 , for all j, j . If such a bit position i is found, then the algorithm uses C(1|i, j, (x(i), x(j))) C(1|i, j, (x(i), x(j))) = Pt (x) = C(i, j, (x(i), x(j)) m as its prediction (for any j = i chosen arbitrarily). Otherwise, the algorithm makes the null prediction . Theorem 1 The inputs m ln(n/) and 1 can be 2 chosen so that the Basic Structure-Learning Algorithm described in this section is an admissible learning algorithm 2 with (, ) n ln(n/) . 2 Proof sketch: The proof has four steps. First, we consider a xed setting (z1 , z2 ) {0, 1}2 to a pair of bit positions, (i ,j) that includes the correct factor. Using Hoeffdings bound, we show that if m independent samples of the next bit y are obtained from inputs x matching this setting (x(i ) = z1 , x(j) = z2 ), then it is very unlikely for the algorithm to learn an incorrect prediction of y given this xed setting (formally |P (x) C(1|i , j, (z1 , z2 ))/m| 1 for all x such that x(i ) = z1 ). Second, we observe that even though each input can be chosen in an adversarial manner (dependent only on the past inputs and outputs), the output is always chosen from a xed distribution (dependent on only the input). Thus, the probability of an incorrect prediction in the adversarial setting is no greater than when m independent samples are available. Third, we use a union bound over the 4n different pairs of factors and binary settings to show that all predictions involving a correct factor are accurate, with high probability. The rest of the argument proceeds as follows. If a prediction is made for input x that is not , then it is equal to C(1|i, j , (x(i), x(j )))/m for some factor i according to the conditions above. Suppose that i is not the correct factor i . The second condition (right above the theorem) implies that |C(1|i, j , (x(i), x(j )))/mC(1|i, i , (x(i), x(i )))/m| 1 . We have shown that with high probability, |P (x) C(1|i, i , (x(i), x(i )))/m| 1 . Combining these two facts gives |C(1|i, j , (x(i), x(j )))/m P (x)| 21 .
Thus, we choose 1 = /2 to satisfy condition (1) of Denition 4. Finally, a simple counting argument and an application of the pigeonhole principle yield the bound on (, ). 2
The General SLF-Rmax Algorithm
In this section, we describe our new structure-learning algorithm called Structure-Learn-Factored-Rmax or SLFRmax. First, we provide the intuition behind the algorithm, then we dene it formally. The algorithm is model based. During each timestep, it acts according to a near-optimal policy of its model, which it computes using any MDP planning algorithm. Since the true transition probabilities are unknown, the algorithm must learn them from past experience. Each transition component, Pi (|s, a), is estimated from experience by an instance of any admissible learning algorithm for the Structure-Learning problem described in the previous section. Thus, SLF-Rmax uses nA instantiations, Ai,a , of this algorithm, one for each factor i and action a. When viewed from Ai,a s perspective (as in the last section), each input vector x is a state and each output bit y is the ith bit position of the next state reached after taking action a from x. If any estimated transition component Pi (|s, a) is , meaning that the algorithm has no reasonable estimate of Pi (|s, a), then the value of taking action a from state s in SLF-Rmaxs model is the maximum possible (1/(1 )). Similar to the Rmax algorithm (Brafman & Tennenholtz 2002), this exploration bonus is an imaginary reward for experiencing a state-action pair whose nextstate distribution involves an inaccurately-modeled transition component. Formally, SLF-Rmax chooses an action from state s that maximizes its current action-value estimates Q(s, a), which are computed by solving the following set of equations: Q(s, a) Q(s, a) otherwise. In Equation 2, we have used T (s |s, a) = i Pi (s (i)|s, a). Pseudo-code for the SLF-Rmax algorithm is provided in Algorithm 1. 1/(1 ), if i, Pi (|s, a) = (2) (s |s, a) max Q(s , a ), T = R(s, a) + =
s a
Theoretical Analysis
We can prove that when given an admissible learning algorithm for the Structure-Learning problem, SLF-Rmax behaves near-optimally on all but a few timesteps, with high probability. The result is comparable with the standard polynomial-time guarantees of RL algorithms (Kearns & Singh 2002; Kakade 2003; Brafman & Tennenholtz 2002).
1 Theorem 2 Suppose that 0 < 1 and 0 < 1 are two real numbers and M = S, A, T, R, is any factoredstate MDP. Let n be the number of state factors. Suppose that StructLearn is an admissible learning algorithm for the Basic Structure-Learning Problem that is used by SLFRmax and has learning complexity (, ). Let At denote
Algorithm 1 SLF-Rmax Algorithm 0: Inputs: n, A, R, , , , admissible learning algorithm StructLearn 1: for all factors i {1, . . . , n} and actions a A do 2: Initialize a new instantiation of StructLearn, de noted Ai,a , with inputs (1 )2 /n, and nk , respectively (for and in Denition 4). 3: end for 4: for all (s, a) S A do 5: Q(s, a) 1/(1 ) // Action-value estimates 6: end for 7: for t = 1, 2, 3, do 8: Let s denote the state at time t. 9: Choose action a := argmaxa A Q(s, a ). 10: Let s be the next state after executing action a. 11: for all factors i {1, . . . , n} do 12: Present input-output pair (s, s (i)) to Ai,a . 13: end for 14: Update action-value estimates by solving Equation 2. 15: end for SLF-Rmaxs policy at time t and st denote the state at time A t. With probability at least 1 , VM t (st ) VM (st ) is true for all but nA (1 )2 timesteps t. O (1 )2 , n nA ln 1 1 ln (1 )
More generally, there are 2k n elements correspond to sets k of k input positions and binary strings (settings) over them. Using the extension of the Basic Structure-Learning algorithm in conjunction with SLF-Rmax, we have the following corollary to Theorem 2. It says that the number of times the algorithm fails to behave near-optimally is bounded by a polynomial in the representation size when k is a xed constant. This exponential dependence on k is unavoidable and appears in similar theoretical results for structure learning (Abbeel, Koller, & Ng 2006).
1 Corollary 1 Suppose that 0 < 1 and 0 < 1 are two real numbers and M = S, A, T, R, is any factoredstate MDP whose transition function is described by depthk decision trees (or DBNs with maximum degree k). Let n be the number of state factors. There exists an admissible learning algorithm StructLearn so that if SLF-Rmax is executed on M using StructLearn, then the following holds. Let At denote SLF-Rmaxs policy at time t and st denote the state at time t. With probability at least 1 , A VM t (st ) VM (st ) is true for all but
O timesteps t.
1 n3+2k Ak ln (nA/) ln 1 ln (1)
3 (1 )6
SLF-Rmax With Different Modeling Assumptions
The general SLF-Rmax algorithm requires, as input, an admissible algorithm for the Structure-Learning problem. For different structural assumptions, different admissible algorithms can be formulated. In a previous section, we provided an algorithm, the Basic Structure-Learning Algorithm, which is admissible under the structural assumption that the probability that the next output bit is 1 depends on a single input bit. This algorithm can be directly incorporated into the SLF-Rmax algorithm for use in factored-state domains whose transition functions are described by decision trees with depth 1. Similarly, it could be used in factored-state domains whose transition functions are described by DBNs with maximum degree 1 (one parent per next-state factor). In most realistic domains described by decision tress or DBNs, the maximum depth or maximum degree, respectively, will be larger than one but often much smaller than n (the number of factors). The Basic Structure-Learning algorithm can be extended to accommodate this situation. The extended version requires a bound, k, on either the maximum depth of the decision trees or the maximum degree of the DBNs. The main idea of the extension is to note that the true probability associated with the next bit is dependent on some unknown element (for instance, a decision-tree leaf or some setting to the parents in a DBN). The algorithm enumerates all possible elements and...