This preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: Introduction to Dynamic Programming Dynamic programming is a methodology that can be effectively applied to a subset of the
general class of mathematical programming problems. It is particularly useful for problems
where the variables are restricted to discrete values (though they need not necessarily be
integer), there can be stochastic aspects to the problem, and the functions need not be
convex. There are problem structure and computational limitations for the applicability of
dynamic programming. As with many methods, dynamic programming is an efﬁcient
solution methodology within its effective applicable range which in terms of the general
class of mathematical programming problems is relatively restiicted. The methodology is
surprisingly robust in the sense of the broad class of problems that can be formulated as dynamic programming problems and the apparent lack of standard structural limitations
such as continuity and convexity assumptions. Dynamic programming generally is used to decompose a multiple variable problem '
into a sequence of related problems each with fewer variables (usually a single variable) I
than the original problem This approach is particularly appropriate for multiple period
decision problems for which the methodology was originally developed. The development
of dynamic programming is generally attributed to Richard Bellman (1920—1984) who
published the ﬁrst paper on the method in 1952 [Bellman, 1952] concerning the solution of
(sequential stochastic decision problems, the ﬁrst book [Bellman 1957] on the topic general
methodology, and the ﬁrst computationally oriented text [Bellman and Dreyfus, 1962]. A
short history of the development of dynamic programming and the ﬁrst problem classes
_ studied is given in an article in memorial to Richard E. Bellman [Bellman and Lee, 1984]. In the chapters on dynamic programming, we present a variety of problems and
solution methods. Emphasis is on the computational aspects of formulating problems and obtaining their solutions. A widevariety of different problem classes will be covered
including serial, nonserial, stochastic, multiple constraints, sequencing and approximating 3
continuous variable problems. The use of dynamic programming as a theoretical analysis tool is not emphasized and can be found in various texts such as Dreyfus and Law [1977]. ' it We utilize two computer languages for this exposition, MGR/DP and MGR/ML.
The MOR/DP system was developed speciﬁcally for solving dynamic programming
recursion function formulations. In this language it is particularly easy to develop a
dynamic programming problem and solve it directly. The language has a great deal of
flexibility, but it is restricted to single state and decision variable formulations. For problems which deal with sets, such as the job sequencing and traveling salesperson
problems,the MGR/ML language is used. This more general purpose language has both set handling capabilities as well as recursive function calls which facilitate development of
certain algorithms. As an introduction to the techniques involved in dynamic programming, we
consider a production planning problem. This example will highlight the essential features
of dynamic programming, particularly the recursive formulation and resulting backward
solution technique. This problem falls into the class of network structures which have a
very descriptive graphic representation that'effectively illustrates the solution method. Example Problem A company has known weekly orders for varying numbers of a given product over the next
5 weeks. These requirements are 50, 30, 60, 45, and 25. To make a production run for
this product, it costs $25 to setup the machine and each unit produced costs an additional
$100. The company has production capacity and storage space for as many units of the
product as needed; however, it costs $1 per unit per week to store units needed in future
time periods. The production capacity is such that all five future period demands could be
produced in a single run._ Of course, the company would incur a considerable holding cost.
Thus, the problem is to determine in which periods to make production runs and how many
items to produce on each run. This analysis is to be based on a minimum cost basis. Note that the unit cost is the same regardless of the production periods and the total
demand is 210 units over this period. Therefore, the total cost to produce these units is 210
times $100. This cost could be omitted from the analysis and only the controllable costs
associated with machine setups and item holding analyzed. We will include the unit cost in our analysis for completeness andit should help illustrate how to directly analyze situations
where this cost varies with the time period. The production decision in period 1 is to decide how many period demands (1 to 5)
are to be produced It is obvious that the demands should be produced in their order of
use; to do otherwise would merely increase the holding costs over that associated with
producing the needed units during the latest possible production run before they are
required. The next production decision occurs when more units need be produced to meet
the demand requirements. Based on the associated setup and holding costs, the optimal
decision could vary from making a production’run each time period to making only one
production run in the ﬁrst period. We graphically represent this production planning problem solution via a network
representation of the problem in Figure 1, where the circles or nodes represent a decision
period and the arcs represent the decisions taken. The nodes depict both the decision and
the demand points. Thus, selecting a given are from a node corresponds to a speciﬁc
decision to produce a given number of period demands at this node; the arc ends at the next ,
period in which a production run must be made based on the decision taken at the previous
node. Note that a sixth node is included in the network representation of the problem.
This node is needed to complete the production of all the demands during the planning
period and represents where the planning horizon ends for this problem. Figure 1: Decision diagram of the production planning problem as a
network. To solve the problem, we need some notation. Let d, be the known demand in
period i, and let x, be the decision taken at node or period 1', with the x, value from the set
{d,, di+ dm, , d; + :15“ + + dn]. Let f, be the optimal (minimal) cost for node 1'
(period 1) to the end of the problem (node 6). We are actually interested in ﬁnding f1, but
as we will see, this requires that we obtain all ffs sequentially (backwards) from f5 to f1 (we deﬁne f6 = 0). We proceed to deﬁne these functions in the order that they occur in
Figure 1, starting with f1; f1 = min{25+100d1+f2, 25+100(d1+d2)+1d2+ f3. 25+100(d1+d2+d3)+1(d2+2d3)+ f4,
25+100(d1+d2+d3+d4)+1(d2+2d3+3d4)+f5_
25+ 100(d1+d2+d3+d4+d5)+1 (d2+2d3+3d4+4d5) }. Rewriting this equation in a more readable form, we have the full set: f1 = min{clg+f2, C13+f3, €14+f4a 615+va €15+f6 },
f2 = miﬂiczﬁfs. €24+f4: €25+f5a C28+f5 ],
f3 '' miﬂlca4+fm 035+f5a €35+f6 },
f4 = min{c45+f5, C46+f6 }, f5 = minl055+f6 }, where cil represents the cost in periodj of producing period demands from i to j—l (that is,
xi = d; + + all4), and we deﬁne f6 = 0.‘ A more concise representation is f, = $31k” +fj}. To actually evaluate f1 we must already know f2, , f6. These functions (including f1)
are deﬁned generally as fig=min{c,j +fj}, fori=1,,5.
J):
Note that to solve for f1, we must know f2, ,f6, and to solve for f2, we must know
 f3, ,f5, and so forth. Finally, to solve for f5, we need only know f6, which we
deﬁned as 0. Thus, we can solve for f5, then from this we can solve for f4, etc., until we
have worked our way back to solving for f1. This solution procedure is very common to
dynamic programming and is called backward recursion. Generally, we solve dynamic programming problems by starting at the end and working backward to the start of the problem This structure occurs because of the forward recursive deﬁnition of the
optimization ftmctions. Now to solve the production planning problem, we must compute Ctj and the
evaluate f5, , f1, in that order. For this particular problem, the cij's are: (:12 = 25 + 100(50) = 5025 c3 = 25 + 100(50+30)+1(30) = 8055 c14 = 25 + 100(50+30+60)+1(30)+2(60) = 14175 815 = 25 + 100(50+30+60+45)+1(30)+2(60)+3(45) = 18810 c16 = 25 + 100(50+30+60+45+25)+1(30)+2(60)+3(45)+4(25) = 21410
c23 = 25 + 100(30) = 3025 C24 = 25 + 100(30+60)+1g60) = 9085
cg = 25 + 100(30+60+45)+1(60)+2(45) = 13675 _
(:25 = 25 + 100(30+60+45+25)+1(60)+2(45)+3(25) = 16250. c3,4 = 25 + 100050) = 5025 C35 = 25 + 100(60+45)+1(45) = 10570 5,6 = 25 + 100(50+45+25)+1(45)+2(25) = 13120
c45 = 25 + 100(45) = 4525 cm5 = 25 + 100(45+25)+1(25) = 7050 as, = 25 + 100(25) = 2525 Then, 1% = 0 f5 = ' 1:37:3533223050 70 0 11‘an“ .7 $016M» }1§osg f3 = m {6025 + 7050, 10570 + €525, 1312040: 13075 0
or? re 0 ‘ 5 0,, s“ 5
f2 = min {302591i3075, 9085 a 39%01‘213675 + 2525, “62 = 16100 A 212%
f1 = min {soiiiglg 8055 + 13075, 14175 + 7050,
£
18810 + 2525, 21410} = 21125 an
The optimal solution cost is $21,125, and the decision taken at period 1 is to produce only :5 for that period. Notice, at this point, we do not know the values of other of the optimal decisions. It turns out, for this problem data, that the holding cost relative to the setup cost
is too expensive to inventory any items. .44» 2 ~=2 ?.—> 355%? How do we obtain the other optimal decisions (x2, , x5)? We trace the optimal
solution by taking the path of the optimal decision at node 1, x1 = 50, which puts us at
node 2. The optimal decision at node 2, x2 2 30, which puts us at node 3, etc. The optimal policy then is to produce only the amount needed in each time period (50, 30, 61),
45, 25). This model is generally called the Wagner—Whitin model even though their solution
procedure was based on a forwardrecursion dynamic programming method. The standard solution approach now is backwardrecursion since it is generally easier to present and
, understand. HomeWork: Resolve this problem with the setup cost of {Syd/rather than $25. '(Answer,
(30, 0, 60, 70, 0) for a total cost of $21,220.) References Bellman, Richard E. 1952. "On the Theory of Dynamic Programming." Prgg. Ngl,
‘ M, Vol. 38, 716. Bellman, Richde 1957. Dynamic Programming. Princeton University Press,
Princeton, N. I. Bellman, R. E. and S. E. Dreyfus. 1962. Applied Dynamic Programming.
Princeton Univ. Press, Princeton, N. J. Bellman, R. E. and ES. Lee. 1984. "History and Development of Dynamic Programming." iEEE gggntlgl Sysgems Magazine, Vol. 4, No. 4, Nov. 1984, 24
28. Dreyfus, S. E. and A. M. Law. 1977. The Art and Theory of Dynamic
Programming. Academic Press, New York. Wagner,H. M. andT. M. Whitin. 1958. "Dynamic Version of the Economic Lot Size
Model", Managemengsgen 11,9; V01. 5, 89 96. ...
View
Full Document
 Fall '10
 Curry

Click to edit the document details