Random numeber generation and Monte Carlo...

  • No School
  • AA 1
  • jramirezca
  • 399

This preview shows page 1 out of 399 pages.

You've reached the end of your free preview.

Want to read all 399 pages?

Unformatted text preview: Statistics and Computing Series Editors: J. Chambers w. Eddy W.W l e S. Sheather L.Tiemey Springer Y~rk Berlin Heidelberg Hang Kong London Milan Perfs Tokyo Statistics and Computing Dalgomd: lnboductoryStatistics with R. Gentle.. Elemak ofComputational Stptistics. OenfIe: Numerical Linear Algebra for Applications m Statistics. Oentle Random N& omaation andMonte &lo Mahods, 2nd Editim. Hcr*dwMWwlach: XploRe: An Intnactive Statistical Computing Bnvirommt RiOUFPN)Iron:me Basics of S and S-Pws,3rd Edition Lmge: NNllmrkal Analysis for Statisticians. b&r:Local Regrnsion and Lilihcd bRurmcrldh/Fibgemld.Numrical Baycsisn Mcmads Applied to Signal Roassing. Pluvrallw: VARIOWIN: Softwan for Spatial Data Analysis in 2D. PinheirOlBau1: Mixed-Effcds Models in S and S - h u s venabk.dRiy,l~: Modem ~ppliedStatisticswith S,4th ~ t l o n . venabler/Riprey: s ProgmEhg. . WWnmn: me Ibeof Graphics. James E. Gentle Random Number Generation and Monte Carlo Methods Second Edition With 54 Illustrati~ns Springer James H. Gentle School of Computational Sciences George Mason University Fairfax. VA 22030-4444 USA j gen [email protected] Series Editors: J. Chambers Bell Labs, Lucent Techonologies 600 Mountain Avenue Murray Hill. NJ 07974 USA W. Eddy Department of Statistics Carnegie Mellon University Pittsburgh, PA USA S. Sheather Australian Graduate School of Management University of New South Wales Sydney, NSW 2052 Australia L. Tiemey Sclool of Statistics and Actuarial Science Universily of Iowa lowa City. IA 52242-1414 USA W, Hardle Institut fiir Slatistik und Okonnmetrie Humboldt-University Spandaucr .Str. I D-10178 Berlin Germany Library of Congress Cataloging-in-Puhlication Data Gentle, James E.. 1943-Random number generation and Monte Carlo methods / James H. Gentle. p. cm. — (Statistics and computing) Includes bibliographical references and index ISBN 0-3S7-OOI78-6 (alk, paper) 1. Monte (,Carlo method. 2. Random number generators. I. Title. [I. Series. (QA2298 ,(G46 2003 519 .2'82—dc21 2003042437 ISBN 0-387-0017-6 e-ISBN 0-387-21610 Printed on acid-free paper CO 2003,3l')'1998Springer Science Business Media, Inc. All rights reserved. This work may not be translated or copied in whole or in pan without the written permission of the publisher (Springer Science Business Media, Inc., 233 Spring Strcoi, New York, NY 10013, USA), except for brief excerpts in connection wish reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in (his publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject In proprietary rights. Printed in the United States of America. 9 S 7 6 5 4 3 2 springeronline.cnm (HID Corrected second printing, 2005. SPIN 11016038 To Maria This page intentionally left blank Preface The role of Monte Carlo methods and simulation in all of the sciences has increased in importance during the past several years. This edition incorporates discussion of many advances in the field of random number generation and Monte Carlo methods since the appearance of the first edition of this book in 1998. These methods play a central role in the rapidly developing subdisciplines of the computational physical sciences, the computational life sciences, and the other computational sciences. The growing power of computers and the evolving simulation methodology have led to the recognition of computation as a third approach for advancing the natural sciences, together with theory and traditional experimentation. At the kernel of Monte Carlo simulation is random number generation. Generation of random numbers is also at the heart of many standard statistical methods. The random sampling required in most analyses is usually done by the computer. The computations required in Bayesian analysis have become viable because of Monte Carlo methods. This has led to much wider applications of Bayesian statistics, which, in turn, has led to development of new Monte Carlo methods and to refinement of existing procedures for random number generation. Various methods for generation of random numbers have been used. Sometimes, processes that are considered random are used, but for Monte Carlo methods, which depend on millions of random numbers, a physical process as a source of random numbers is generally cumbersome. Instead of “random” numbers, most applications use “pseudorandom” numbers, which are deterministic but “look like” they were generated randomly. Chapter 1 discusses methods for generation of sequences of pseudorandom numbers that simulate a uniform distribution over the unit interval (0, 1). These are the basic sequences from which are derived pseudorandom numbers from other distributions, pseudorandom samples, and pseudostochastic processes. In Chapter 1, as elsewhere in this book, the emphasis is on methods that work. Development of these methods often requires close attention to details. For example, whereas many texts on random number generation use the fact that the uniform distribution over (0, 1) is the same as the uniform distribution over (0, 1] or [0, 1], I emphasize the fact that we are simulating this disvii viii PREFACE tribution with a discrete set of “computer numbers”. In this case whether 0 and/or 1 is included does make a difference. A uniform random number generator should not yield a 0 or 1. Many authors ignore this fact. I learned it over twenty years ago, shortly after beginning to design industrial-strength software. The Monte Carlo methods raise questions about the quality of the pseudorandom numbers that simulate physical processes and about the ability of those numbers to cover the range of a random variable adequately. In Chapter 2, I address some issues of the quality of pseudorandom generators. Chapter 3 describes some of the basic issues in quasirandom sequences. These sequences are designed to be very regular in covering the support of the random process simulated. Chapter 4 discusses general methods for transforming a uniform random deviate or a sequence of uniform random deviates into a deviate from a different distribution. Chapter 5 describes methods for some common specific distributions. The intent is not to provide a compendium in the manner of Devroye (1986a) but, for many standard distributions, to give at least a simple method or two, which may be the best method, but, if the better methods are quite complicated, to give references to those methods. Chapter 6 continues the developments of Chapters 4 and 5 to apply them to generation of samples and nonindependent sequences. Chapter 7 considers some applications of random numbers. Some of these applications are to solve deterministic problems. This type of method is called Monte Carlo. Chapter 8 provides information on computer software for generation of random variates. The discussion concentrates on the S-Plus, R, and IMSL software systems. Monte Carlo methods are widely used in the research literature to evaluate properties of statistical methods. Chapter 9 addresses some of the considerations that apply to this kind of study. I emphasize that a Monte Carlo study uses an experiment, and the principles of scientific experimentation should be observed. The literature on random number generation and Monte Carlo methods is vast and ever-growing. There is a rather extensive list of references beginning on page 336; however, I do not attempt to provide a comprehensive bibliography or to distinguish the highly-varying quality of the literature. The main prerequisite for this text is some background in what is generally called “mathematical statistics”. In the discussions and exercises involving multivariate distributions, some knowledge of matrices is assumed. Some scientific computer literacy is also necessary. I do not use any particular software system in the book, but I do assume the ability to program in either Fortran or C and the availability of either S-Plus, R, Matlab, or Maple. For some exercises, the required software can be obtained from either statlib or netlib (see the bibliography). The book is intended to be both a reference and a textbook. It can be PREFACE ix used as the primary text or a supplementary text for a variety of courses at the graduate or advanced undergraduate level. A course in Monte Carlo methods could proceed quickly through Chapter 1, skip Chapter 2, cover Chapters 3 through 6 rather carefully, and then, in Chapter 7, depending on the backgrounds of the students, discuss Monte Carlo applications in specific fields of interest. Alternatively, a course in Monte Carlo methods could begin with discussions of software to generate random numbers, as in Chapter 8, and then go on to cover Chapters 7 and 9. Although the material in Chapters 1 through 6 provides the background for understanding the methods, in this case the details of the algorithms are not covered, and the material in the first six chapters would only be used for reference as necessary. General courses in statistical computing or computational statistics could use the book as a supplemental text, emphasizing either the algorithms or the Monte Carlo applications as appropriate. The sections that address computer implementations, such as Section 1.2, can generally be skipped without affecting the students’ preparation for later sections. (In any event, when computer implementations are discussed, note should be taken of my warnings about use of software for random number generation that has not been developed by software development professionals.) In most classes that I teach in computational statistics, I give Exercise 9.3 in Chapter 9 (page 311) as a term project. It is to replicate and extend a Monte Carlo study reported in some recent journal article. In working on this exercise, the students learn the sad facts that many authors are irresponsible and many articles have been published without adequate review. Acknowledgments I thank John Kimmel of Springer for his encouragement and advice on this book and other books on which he has worked with me. I thank Bruce McCullough for comments that corrected some errors and improved clarity in a number of spots. I thank the anonymous reviewers of this edition for their comments and suggestions. I also thank the many readers of the first edition who informed me of errors and who otherwise provided comments or suggestions for improving the exposition. I thank my wife Mar´ıa, to whom this book is dedicated, for everything. I did all of the typing, programming, etc., myself, so all mistakes are mine. I would appreciate receiving suggestions for improvement and notice of errors. Notes on this book, including errata, are available at Fairfax County, Virginia James E. Gentle April 10, 2003 This page intentionally left blank Contents Preface vii 1 Simulating Random Numbers from a Uniform Distribution 1.1 Uniform Integers and an Approximate Uniform Density . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Simple Linear Congruential Generators . . . . . . . . . . . . . . 1.2.1 Structure in the Generated Numbers . . . . . . . . . . . 1.2.2 Tests of Simple Linear Congruential Generators . . . . . 1.2.3 Shuffling the Output Stream . . . . . . . . . . . . . . . 1.2.4 Generation of Substreams in Simple Linear Congruential Generators . . . . . . . . . . . . . . . . . . 1.3 Computer Implementation of Simple Linear Congruential Generators . . . . . . . . . . . . . . . . . . . . . . 1.3.1 Ensuring Exact Computations . . . . . . . . . . . . . . 1.3.2 Restriction that the Output Be in the Open Interval (0, 1) . . . . . . . . . . . . . . . . . . . . 1.3.3 Efficiency Considerations . . . . . . . . . . . . . . . . . 1.3.4 Vector Processors . . . . . . . . . . . . . . . . . . . . . . 1.4 Other Linear Congruential Generators . . . . . . . . . . . . . . 1.4.1 Multiple Recursive Generators . . . . . . . . . . . . . . 1.4.2 Matrix Congruential Generators . . . . . . . . . . . . . 1.4.3 Add-with-Carry, Subtract-with-Borrow, and Multiply-with-Carry Generators . . . . . . . . . . . . . 1.5 Nonlinear Congruential Generators . . . . . . . . . . . . . . . . 1.5.1 Inversive Congruential Generators . . . . . . . . . . . . 1.5.2 Other Nonlinear Congruential Generators . . . . . . . . 1.6 Feedback Shift Register Generators . . . . . . . . . . . . . . . . 1.6.1 Generalized Feedback Shift Registers and Variations . . 1.6.2 Skipping Ahead in GFSR Generators . . . . . . . . . . . 1.7 Other Sources of Uniform Random Numbers . . . . . . . . . . 1.7.1 Generators Based on Cellular Automata . . . . . . . . . 1.7.2 Generators Based on Chaotic Systems . . . . . . . . . . 1.7.3 Other Recursive Generators . . . . . . . . . . . . . . . . xi 1 . . . . . 5 11 14 20 21 . 23 . 27 . 28 . . . . . . 29 30 30 31 32 34 . . . . . . . . . . . 35 36 36 37 38 40 43 43 44 45 45 xii CONTENTS 1.7.4 Tables of Random Numbers . . . . . . . . . . . . . . . . 1.8 Combining Generators . . . . . . . . . . . . . . . . . . . . . . . 1.9 Properties of Combined Generators . . . . . . . . . . . . . . . . 1.10 Independent Streams and Parallel Random Number Generation 1.10.1 Skipping Ahead with Combination Generators . . . . . 1.10.2 Different Generators for Different Streams . . . . . . . . 1.10.3 Quality of Parallel Random Number Streams . . . . . . 1.11 Portability of Random Number Generators . . . . . . . . . . . 1.12 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 46 48 51 52 52 53 54 55 56 2 Quality of Random Number Generators 2.1 Properties of Random Numbers . . . . . . . . . . . . 2.2 Measures of Lack of Fit . . . . . . . . . . . . . . . . 2.2.1 Measures Based on the Lattice Structure . . 2.2.2 Differences in Frequencies and Probabilities . 2.2.3 Independence . . . . . . . . . . . . . . . . . . 2.3 Empirical Assessments . . . . . . . . . . . . . . . . . 2.3.1 Statistical Goodness-of-Fit Tests . . . . . . . 2.3.2 Comparisons of Simulated Results with Statistical Models in Physics . . . . . . . . . 2.3.3 Anecdotal Evidence . . . . . . . . . . . . . . 2.3.4 Tests of Random Number Generators Used in 2.4 Programming Issues . . . . . . . . . . . . . . . . . . 2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 62 64 64 67 70 71 71 . . . . . . . . . . Parallel . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 86 87 87 87 88 3 Quasirandom Numbers 3.1 Low Discrepancy . . . . 3.2 Types of Sequences . . . 3.2.1 Halton Sequences 3.2.2 Sobol’ Sequences 3.2.3 Comparisons . . 3.2.4 Variations . . . . 3.2.5 Computations . . 3.3 Further Comments . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 93 94 94 96 97 97 98 98 100 4 Transformations of Uniform Deviates: General Methods 4.1 Inverse CDF Method . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Decompositions of Distributions . . . . . . . . . . . . . . . . . . 4.3 Transformations that Use More than One Uniform Deviate . . 4.4 Multivariate Uniform Distributions with Nonuniform Marginals 4.5 Acceptance/Rejection Methods . . . . . . . . . . . . . . . . . . 4.6 Mixtures and Acceptance Methods . . . . . . . . . . . . . . . . . . . . . . 101 102 109 111 112 113 125 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CONTENTS 4.7 Ratio-of-Uniforms Method . . . . . . . . . . . . . . . . . . 4.8 Alias Method . . . . . . . . . . . . . . . . . . . . . . . . . 4.9 Use of the Characteristic Function . . . . . . . . . . . . . 4.10 Use of Stationary Distributions of Markov Chains . . . . . 4.11 Use of Conditional Distributions . . . . . . . . . . . . . . 4.12 Weighted Resampling . . . . . . . . . . . . . . . . . . . . 4.13 Methods for Distributions with Certain Special Properties 4.14 General Methods for Multivariate Distributions . . . . . . 4.15 Generating Samples from a Given Distribution . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Simulating Random Numbers from Specific Distributions 5.1 Modifications of Standard Distributions . . . . . . . . . . . . . 5.2 Some Specific Univariate Distributions . . . . . . . . . . . . . . 5.2.1 Normal Distribution . . . . . . . . . . . . . . . . . . . . 5.2.2 Exponential, Double Exponential, and Exponential Power Distributions . . . . . . . . . . . . . . . . . . . . 5.2.3 Gamma Distribution . . . . . . . . . . . . . . . . . . . . 5.2.4 Beta Distribution . . . . . . . . . . . . . . . . . . . . . . 5.2.5 Chi-Squared, Student’s t, and F Distributions . . . . . . 5.2.6 Weibull Distribution . . . . . . . . . . . . . . . . . . . . 5.2.7 Binomial Distribution . . . . . . . . . . . . . . . . . . . 5.2.8 Poisson Distribution . . . . . . . . . . . . . . . . . . . . 5.2.9 Negative Binomial and Geometric Distributions . . . . . 5.2.10 Hypergeometric Distribution . . . . . . . . . . . . . . . 5.2.11 Logarithmic Distribution . . . . . . . . . . . . . . . . . 5.2.12 Other Specific Univariate Distributions . . . . . . . . . 5.2.13 General Families of Univariate Distributions . . . . . . . 5.3 Some Specific Multivariate Distributions . . . . . . . . . . . . . 5.3.1 Multivariate Normal Distribution . . . . . . . . . . . . . 5.3.2 Multinomial Distribution . . . . . . . . . . . . . . . . . 5.3.3 Correlation Matrices and Variance-Covariance Matrices 5.3.4 Points on a Sphere . . . . . . . . . . . . . . . . . . . . . 5.3.5 Two-Way Tables . . . . . . . . . . . . . . . . . . . . . . 5.3.6 Other Specific Multivariate Distributions . . . . . . . . 5.3.7 Families of Multivariate Distributions . . . . . . . . . . 5.4 Data-Based Random Number Generation . . . . . . . . . . . . 5.5 Geometric Objects . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 133 136 137 149 149 150 155 159 159 165 . 167 . 170 . 171 . . . . . . . . . . . . . . . . . . . . . . . 176 178 183 184 186 187 188 188 189 190 191 193 197 197 198 198 201 202 203 208 210 212 213 6 Generation of Random Samples, Permutations, and Stochastic Processes 217 6.1 Random Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 6.2 Permutations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 6.3 Limitations of Random Number Generators . . . . . . . . . . . . 220 xiv CONTENTS 6.4 Generation of Nonindependent Samples . 6.4.1 Order Statistics . . . . . . . . . . . 6.4.2 Censored Data . . . . . . . . . . . 6.5 Generation of Nonindependent Sequences 6.5.1 Markov Process . . . . . . . . . . . 6.5.2 Nonhomogeneous Poisson Process 6.5.3 Other Time Series Models . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 221 223 224 224 225 226 227 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 230 233 235 236 239 240 241 245 248 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 249 250 2...
View Full Document

  • Fall '19
  • Randomness, Monte Carlo method, Monte Carlo methods

  • Left Quote Icon

    Student Picture

  • Left Quote Icon

    Student Picture

  • Left Quote Icon

    Student Picture