Fluent_Python.pdf - Fluent Python CLEAR CONCISE AND EFFECTIVE PROGRAMMING Luciano Ramalho Fluent Python Pythons simplicity lets you become productive

Fluent_Python.pdf - Fluent Python CLEAR CONCISE AND...

This preview shows page 1 out of 792 pages.

You've reached the end of your free preview.

Want to read all 792 pages?

Unformatted text preview: Fluent Python CLEAR, CONCISE, AND EFFECTIVE PROGRAMMING Luciano Ramalho Fluent Python Python’s simplicity lets you become productive quickly, but this often means you aren’t using everything it has to offer. With this hands-on guide, you’ll learn how to write effective, idiomatic Python code by leveraging its best—and possibly most neglected—features. Author Luciano Ramalho takes you through Python’s core language features and libraries, and shows you how to make your code shorter, faster, and more readable at the same time. Many experienced programmers try to bend Python to fit patterns they learned from other languages, and never discover Python features outside of their experience. With this book, those Python programmers will thoroughly learn how to become proficient in Python 3. proud to have been “ Iaam tech reviewer for this excellent book—not only will it help many intermediate Python programmers on their road towards mastery, but it has taught me quite a few things, too! ” This book covers: ■■ The Python data model: understand how special methods are the key to the consistent behavior of objects ■■ Data structures: take full advantage of built-in types, and understand the text versus bytes duality in the Unicode age ■■ Functions as objects: view Python functions as first-class objects, and understand how this affects popular design patterns ■■ Object-oriented idioms: build classes by learning about references, mutability, interfaces, operator overloading, and multiple inheritance ■■ Control flow: leverage context managers, generators, coroutines, and concurrency with the concurrent.futures and asyncio packages ■■ —Alex Martelli Python Software Foundation Fellow is a “treasure Fluent Python trove full of useful programming tricks for intermediate to advanced Python coders who want to push the boundaries of their knowledge. ” —Daniel and Audrey Roy Greenfeld authors of Two Scoops of Django Metaprogramming: understand how properties, attribute descriptors, class decorators, and metaclasses work Luciano Ramalho, a Python programmer since 1998, is a Python Software Foundation fellow, co-owner of Python.pro.br—a training company in Brazil— and cofounder of Garoa Hacker Clube, Brazil’s first hackerspace. He has led software development teams and taught Python courses in Brazilian media, banking, and government sectors. PROGR AMMING/PY THON US $49.99 CAN $57.99 ISBN: 978-1-491-9-46008 Twitter: @oreillymedia facebook.com/oreilly Fluent Python Luciano Ramalho Beijing Boston Farnham Sebastopol Tokyo Fluent Python By Luciano Ramalho Copyright © 2015 Luciano Gama de Sousa Ramalho. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles ( ). For more information, contact our corporate/ institutional sales department: 800-998-9938 or [email protected] Editors: Meghan Blanchette and Rachel Roumeliotis Production Editor: Melanie Yarbrough August, 2015: Copyeditor: Kim Cofer Proofreader: Jasmine Kwityn Indexer: Judy McConville Interior Designer: David Futato Cover Designer: Ellie Volckhausen Illustrator: Rebecca Demarest First Edition Revision History for the First Edition 2015-07-24: 2015-08-21: 2016-03-18: First release Second release Third release See for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Fluent Python, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-491-94600-8 [LSI] Para Marta, com todo o meu amor. Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Part I. Prologue 1. The Python Data Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 A Pythonic Card Deck How Special Methods Are Used Emulating Numeric Types String Representation Arithmetic Operators Boolean Value of a Custom Type Overview of Special Methods Why len Is Not a Method Chapter Summary Further Reading Part II. 4 8 9 11 12 12 13 14 15 15 Data Structures 2. An Array of Sequences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Overview of Built-In Sequences List Comprehensions and Generator Expressions List Comprehensions and Readability Listcomps Versus map and filter Cartesian Products Generator Expressions Tuples Are Not Just Immutable Lists 22 23 23 25 26 27 28 v Tuples as Records Tuple Unpacking Nested Tuple Unpacking Named Tuples Tuples as Immutable Lists Slicing Why Slices and Range Exclude the Last Item Slice Objects Multidimensional Slicing and Ellipsis Assigning to Slices Using + and * with Sequences Building Lists of Lists Augmented Assignment with Sequences A += Assignment Puzzler list.sort and the sorted Built-In Function Managing Ordered Sequences with bisect Searching with bisect Inserting with bisect.insort When a List Is Not the Answer Arrays Memory Views NumPy and SciPy Deques and Other Queues Chapter Summary Further Reading 29 30 32 33 34 35 36 36 37 38 39 39 41 42 45 47 47 50 51 51 54 55 58 61 62 3. Dictionaries and Sets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Generic Mapping Types dict Comprehensions Overview of Common Mapping Methods Handling Missing Keys with setdefault Mappings with Flexible Key Lookup defaultdict: Another Take on Missing Keys The __missing__ Method Variations of dict Subclassing UserDict Immutable Mappings Set Theory set Literals Set Comprehensions Set Operations dict and set Under the Hood vi | Table of Contents 68 70 71 72 74 75 76 79 80 82 83 85 86 87 90 A Performance Experiment Hash Tables in Dictionaries Practical Consequences of How dict Works How Sets Work—Practical Consequences Chapter Summary Further Reading 90 92 95 98 98 99 4. Text versus Bytes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 Character Issues Byte Essentials Structs and Memory Views Basic Encoders/Decoders Understanding Encode/Decode Problems Coping with UnicodeEncodeError Coping with UnicodeDecodeError SyntaxError When Loading Modules with Unexpected Encoding How to Discover the Encoding of a Byte Sequence BOM: A Useful Gremlin Handling Text Files Encoding Defaults: A Madhouse Normalizing Unicode for Saner Comparisons Case Folding Utility Functions for Normalized Text Matching Extreme “Normalization”: Taking Out Diacritics Sorting Unicode Text Sorting with the Unicode Collation Algorithm The Unicode Database Dual-Mode str and bytes APIs str Versus bytes in Regular Expressions str Versus bytes on os Functions Chapter Summary Further Reading 102 103 106 107 109 109 110 112 113 114 115 118 121 124 125 126 129 131 132 134 134 135 138 139 Part III. Functions as Objects 5. First-Class Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 Treating a Function Like an Object Higher-Order Functions Modern Replacements for map, filter, and reduce Anonymous Functions The Seven Flavors of Callable Objects 146 147 148 150 151 Table of Contents | vii User-Defined Callable Types Function Introspection From Positional to Keyword-Only Parameters Retrieving Information About Parameters Function Annotations Packages for Functional Programming The operator Module Freezing Arguments with functools.partial Chapter Summary Further Reading 152 153 155 156 161 163 163 167 169 169 6. Design Patterns with First-Class Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 Case Study: Refactoring Strategy Classic Strategy Function-Oriented Strategy Choosing the Best Strategy: Simple Approach Finding Strategies in a Module Command Chapter Summary Further Reading 174 174 178 181 182 184 185 186 7. Function Decorators and Closures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 Decorators 101 When Python Executes Decorators Decorator-Enhanced Strategy Pattern Variable Scope Rules Closures The nonlocal Declaration Implementing a Simple Decorator How It Works Decorators in the Standard Library Memoization with functools.lru_cache Generic Functions with Single Dispatch Stacked Decorators Parameterized Decorators A Parameterized Registration Decorator The Parameterized Clock Decorator Chapter Summary Further Reading viii | Table of Contents 190 191 193 195 198 201 203 204 206 206 208 212 212 213 215 218 218 Part IV. Object-Oriented Idioms 8. Object References, Mutability, and Recycling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 Variables Are Not Boxes Identity, Equality, and Aliases Choosing Between == and is The Relative Immutability of Tuples Copies Are Shallow by Default Deep and Shallow Copies of Arbitrary Objects Function Parameters as References Mutable Types as Parameter Defaults: Bad Idea Defensive Programming with Mutable Parameters del and Garbage Collection Weak References The WeakValueDictionary Skit Limitations of Weak References Tricks Python Plays with Immutables Chapter Summary Further Reading 226 227 229 230 231 234 235 236 239 241 242 244 246 247 249 250 9. A Pythonic Object. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 Object Representations Vector Class Redux An Alternative Constructor classmethod Versus staticmethod Formatted Displays A Hashable Vector2d Private and “Protected” Attributes in Python Saving Space with the __slots__ Class Attribute The Problems with __slots__ Overriding Class Attributes Chapter Summary Further Reading 256 256 260 260 262 266 272 274 276 277 279 281 10. Sequence Hacking, Hashing, and Slicing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 Vector: A User-Defined Sequence Type Vector Take #1: Vector2d Compatible Protocols and Duck Typing Vector Take #2: A Sliceable Sequence How Slicing Works A Slice-Aware __getitem__ Vector Take #3: Dynamic Attribute Access 286 286 289 290 291 293 295 Table of Contents | ix Vector Take #4: Hashing and a Faster == Vector Take #5: Formatting Chapter Summary Further Reading 299 305 312 313 11. Interfaces: From Protocols to ABCs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319 Interfaces and Protocols in Python Culture Python Digs Sequences Monkey-Patching to Implement a Protocol at Runtime Alex Martelli’s Waterfowl Subclassing an ABC ABCs in the Standard Library ABCs in collections.abc The Numbers Tower of ABCs Defining and Using an ABC ABC Syntax Details Subclassing the Tombola ABC A Virtual Subclass of Tombola How the Tombola Subclasses Were Tested Usage of register in Practice Geese Can Behave as Ducks Chapter Summary Further Reading 320 322 324 326 331 333 333 335 336 340 341 344 347 350 351 352 355 12. Inheritance: For Good or For Worse. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361 Subclassing Built-In Types Is Tricky Multiple Inheritance and Method Resolution Order Multiple Inheritance in the Real World Coping with Multiple Inheritance 1. Distinguish Interface Inheritance from Implementation Inheritance 2. Make Interfaces Explicit with ABCs 3. Use Mixins for Code Reuse 4. Make Mixins Explicit by Naming 5. An ABC May Also Be a Mixin; The Reverse Is Not True 6. Don’t Subclass from More Than One Concrete Class 7. Provide Aggregate Classes to Users 8. “Favor Object Composition Over Class Inheritance.” Tkinter: The Good, the Bad, and the Ugly A Modern Example: Mixins in Django Generic Views Chapter Summary Further Reading x | Table of Contents 362 365 371 373 374 374 374 374 374 375 375 376 376 377 381 382 13. Operator Overloading: Doing It Right. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385 Operator Overloading 101 Unary Operators Overloading + for Vector Addition Overloading * for Scalar Multiplication Rich Comparison Operators Augmented Assignment Operators Chapter Summary Further Reading Part V. 386 386 389 394 399 403 408 409 Control Flow 14. Iterables, Iterators, and Generators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415 Sentence Take #1: A Sequence of Words Why Sequences Are Iterable: The iter Function Iterables Versus Iterators Sentence Take #2: A Classic Iterator Making Sentence an Iterator: Bad Idea Sentence Take #3: A Generator Function How a Generator Function Works Sentence Take #4: A Lazy Implementation Sentence Take #5: A Generator Expression Generator Expressions: When to Use Them Another Example: Arithmetic Progression Generator Arithmetic Progression with itertools Generator Functions in the Standard Library New Syntax in Python 3.3: yield from Iterable Reducing Functions A Closer Look at the iter Function Case Study: Generators in a Database Conversion Utility Generators as Coroutines Chapter Summary Further Reading 416 418 420 423 425 426 427 431 432 434 435 438 439 449 450 452 453 455 455 456 15. Context Managers and else Blocks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463 Do This, Then That: else Blocks Beyond if Context Managers and with Blocks The contextlib Utilities Using @contextmanager Chapter Summary Further Reading 464 466 470 471 476 476 Table of Contents | xi 16. Coroutines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479 How Coroutines Evolved from Generators Basic Behavior of a Generator Used as a Coroutine Example: Coroutine to Compute a Running Average Decorators for Coroutine Priming Coroutine Termination and Exception Handling Returning a Value from a Coroutine Using yield from The Meaning of yield from Use Case: Coroutines for Discrete Event Simulation About Discrete Event Simulations The Taxi Fleet Simulation Chapter Summary Further Reading 480 481 484 486 488 491 494 500 506 507 508 516 518 17. Concurrency with Futures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523 Example: Web Downloads in Three Styles A Sequential Download Script Downloading with concurrent.futures Where Are the Futures? Blocking I/O and the GIL Launching Processes with concurrent.futures Experimenting with Executor.map Downloads with Progress Display and Error Handling Error Handling in the flags2 Examples Using futures.as_completed Threading and Multiprocessing Alternatives Chapter Summary Further Reading 523 525 528 529 533 534 536 539 544 547 550 550 551 18. Concurrency with asyncio. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557 Thread Versus Coroutine: A Comparison asyncio.Future: Nonblocking by Design Yielding from Futures, Tasks, and Coroutines Downloading with asyncio and aiohttp Running Circles Around Blocking Calls Enhancing the asyncio downloader Script Using asyncio.as_completed Using an Executor to Avoid Blocking the Event Loop From Callbacks to Futures and Coroutines Doing Multiple Requests for Each Download Writing asyncio Servers xii | Table of Contents 559 566 567 568 573 575 576 582 583 586 589 An asyncio TCP Server An aiohttp Web Server Smarter Clients for Better Concurrency Chapter Summary Further Reading Part VI. 589 594 598 599 601 Metaprogramming 19. Dynamic Attributes and Properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 607 Data Wrangling with Dynamic Attributes Exploring JSON-Like Data with Dynamic Attributes The Invalid Attribute Name Problem Flexible Object Creation with __new__ Restructuring the OSCON Feed with shelve Linked Record Retrieval with Properties Using a Property for Attribute Validation LineItem Take #1: Class for an Item in an Order LineItem Take #2: A Validating Property A Proper Look at Properties Properties Override Instance Attributes Property Documentation Coding a Property Factory Handling Attribute Deletion Essential Attributes and Functions for Attribute Handling Special Attributes that Affect Attribute Handling Built-In Functions for Attribute Handling Special Methods for Attribute Handling Chapter Summary Further Reading 608 610 614 615 617 620 627 627 628 630 631 634 635 638 639 640 640 641 642 643 20. Attribute Descriptors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 649 Descriptor Example: Attribute Validation LineItem Take #3: A Simple Descriptor LineItem Take #4: Automatic Storage Attribute Names LineItem Take #5: A New Descriptor Type Overriding Versus Nonoverriding Descriptors Overriding Descriptor Overriding Descriptor Without __get__ Nonoverriding Descriptor Overwriting a Descriptor in the Class Methods Are Descriptors 650 650 655 661 665 667 668 669 670 671 Table of Contents | xiii Descriptor Usage Tips Descriptor docstring and Overriding Deletion Chapter Summary Further Reading 673 675 676 677 21. Class Metaprogramming. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 681 A Class Factory A Class Decorator for Customizing Descriptors What Happens When: Import Time Versus Runtime The Evaluation Time Exercises Metaclasses 101 The Metaclass Evaluation Time Exercise A Metaclass for Customizing Descriptors The Metaclass __prepare__ Special Method Classes as Objects Chapter Summary Further Reading 682 685 688 689 693 695 699 701 703 704 706 Afterword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 709 A. Support Scripts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 713 Python Jargon. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 741 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 751 xiv | Table of Contents Preface Here’s the plan: when someone uses a feature you don’t understand, simply shoot them. This is easier than learning something new, and before too long the only living coders will be writing in an easily understood, tiny subset of Python 0.9.6 <wink>.1 —Tim Peters, Legendary core developer and author of The Zen of Python “Python is an easy to learn, powerful programming language.” Those are the first words of the official Python Tutorial. That is true, but there is a catch: because the language is easy to learn and put to use, many practicing Python programmers lever‐ age only a fraction of its powerful features. An experienced programmer may start writing useful Python code in a matter of hours. As the first productive hours become weeks and months, a lot of developers go on writing Python code with a very strong accent carried from languages learned before. Even if Python is your first language, often in academia and in introductory books it is presented ...
View Full Document

  • Fall '09
  • AnneRogers

  • Left Quote Icon

    Student Picture

  • Left Quote Icon

    Student Picture

  • Left Quote Icon

    Student Picture