2.5 - Introducing with and defaultdict

00:00 In the previous lesson, we covered the early history of Python leading up to the 2.4 release. This lesson covers Python 2.5, the with keyword through context managers and defaultdict from collections.

00:15 Python 2.5 saw the introduction of the functools module. This is kind of an interesting point in the language. Python started out as a traditional procedural scripting language, then it added some object-oriented concepts several times over.

00:28 And here it added tools for functional programming. Functional programming uses a methodology where you compose functions together to perform actions. It’s more declarative than procedural programming.

00:39 This isn’t really Python’s strong suit, and in fact, there are languages designed around this paradigm. But functional programming is good with certain kinds of algorithms, and so Python mixing it in gives you a little more choice.

00:52 The any() and all() functions fit this concept well. The first returning True if any item in an iterable is True and the second returning True when all items in an iterable are True.

01:04 The min() and max() functions existed already, but 2.5 added the key argument, allowing you to specify how to determine if something is min or `max.

01:13 This is particularly useful if you’re trying to get the min or max of a series of objects. You can use the key to access an attribute within those objects.

01:23 2.5 also introduced context managers, which you use through the new with keyword. It also added the defaultdict class, which is an extension to dict that automatically sets a key to a default value.

01:36 Alright, enough of this def by PowerPoint approach. Let’s head to the REPL and dig into two of these features. First, context managers with the with keyword, and then the handy little tool defaultdict.

01:50 To better understand with, I’ll first explain what it looks like without. See what I did there. Say you want to read a file,

02:00 you call open() and get back a file handle. Then maybe use readlines() to get at its content.

02:07 My file had the numbers one, two, and three on separate lines and readlines() returns those as a list of strings corresponding to each line in the file.

02:18 The .closed attribute tells you whether a file handle is open or closed. since I haven’t closed it yet, it’s returning False. Closing files is important.

02:27 File handles take up resources. Not only do they cost memory, but most operating systems restrict the number of handles that you can have open at one time.

02:36 For our simple example here, it doesn’t matter, but lots of interfaces on your computer use this same file handle mechanism. If your program deals with sockets and lots of things are connected to your code, each connection gets its own file handle.

02:49 If you don’t clean up after yourself, your program might eventually crash. And that cleanup is as simple as calling the .close() method.

02:58 Great. So now it’s closed.

03:02 Every open() should be paired with a close(). It’s easy to create a bug by forgetting to close the file or by closing it in one place and thinking it’s still open in another. Context managers help with this.

03:14 Let’s use the with keyword in the bottom window to do the same thing. The with keyword can be used with the same open() function as above.

03:27 Seeing this is a block of code. You get at the file handle using the as keyword along with with. In this case, the open() function is returning what’s called a context manager.

03:37 The context in this case being the file’s open state inside of the with block, the file is open. Once the block is done, the file automatically gets closed, meaning you don’t forget to close it.

03:49 I won’t cover it here, but all this magic happens with dunder methods. So like most things in Python, you can write your own classes that take advantage of this feature.

04:00 Like with above, I read the lines from the file

04:05 and here I’ll report the state of the file

04:08 and then I run the block. Same output as before,

04:14 but this time the file got closed automatically. I’m a big fan of context managers and anytime you’re dealing with state-based resources like files, database connections, sockets and more, the with keyword is the better approach.

04:28 So have you experienced something similar in another language like JavaScript where you’ve left something open accidentally? Yeah, so this was a common bug in C.

04:38 The destructor concept in C++ kind of deals with this as well. So in addition to having a constructor in C++, you also have a destructor, which is a method that can be called once the compiler decides it’s out of scope.

04:53 And so I remember coming across constructor-based ideas that were essentially this, and this is similar to like Python uses, as I mentioned, it uses dunder methods, it’s .__enter__() and .__exit__(). Yeah.

05:05 And so there were smart file libraries out there that were using objects in C++ because of the same thing. So as soon as the object went out of scope, the destructor got called and the destructor would close the file handle for you so that you don’t muck it up.

05:18 So this idea’s been around in a few different languages. I don’t know that I’ve seen the with context before. Okay. But Python does steal from a whole bunch of places.

05:28 So if somebody told me it was in Scala or Ada, I wouldn’t be terribly surprised. And have you ever used the .__enter__() and .__exit__() special dunder methods to, you know, create something unique in your own context manager?

05:40 Yeah, I’ve built a couple of these. It almost always comes down to that sort of resource management thing, right? Right. So you want to have something on and then you want it to be off once it’s out of scope.

05:50 When that pattern is showing up in your code, then you know, this is a good one. One of my favorite uses of this actually is inside of unit tests. Yeah. Sometimes you need to check a failed case.

06:03 So you want something to raise an exception. Well, in order to test that, the exception is raised, that’s going to raise an exception and you could put a try block around it and then put some code in that checks that the exception got raised.

06:16 But in fact, there is a with method for this, which is with assertRaises. So you can essentially say the code underneath this block is supposed to raise an exception, and if it doesn’t, it raises an exception.

06:27 So it’s kind of the counter to help you test error-handling code. And it’s a really, really useful one of these. It actually saves you a whole bunch of extra lines of code.

06:37 Yeah, I feel like we’re going to see more and more of this under the hood sort of special method stuff that is architecture that you don’t necessarily need to, you know, create and edit yourself.

06:48 But as you become deeper and deeper inside the language and need these sort of hooks, if you will, they’re available. Which is kind of nice to see that being built out.

06:57 Yeah, it’s, I think it’s one of the powers of Python. So many of the things that the language itself uses are mechanisms that are available to you. Right. So in other languages, you know, they’d add a keyword and then you’d be screwed because you can’t add a keyword.

07:12 Yeah. Whereas you know, Python’s default answer to everything is a dunder method. And it’s a little weird if you’ve never seen it before, but once you get used to it, I didn’t like the name of it originally.

07:20 The yeah. That I would always see of magic methods. Yes. Yeah. But once you’ve seen it and you’ve gotten used to the idea, it’s all of a sudden it’s like, oh hey, there’s a place where using a less than comparison in my objects makes sense.

07:33 I should be able to implement it, right? Yeah, yeah, totally. Cool. Okay, let’s try the defaultdict.

07:41 So defaultdict is a special kind of dictionary. In order to explain it, it’s a little easier if I show you the case without it, say you’ve got a dict that’s going to act as a thesaurus.

07:54 Each entry in the dict maps to a list, in this case, the synonyms of “hello,”

08:02 which means if I want to have something empty inside of it, it has to be an empty list. That way I can use the append on any key that’s in there.

08:17 If you don’t start with that empty list each time you go to use the dict, you’d have to check whether or not the key existed and create that empty list before calling append.

08:27 That ends up meaning an if-else block each time you added an entry. And of course this means you have to remember to create that empty list.

08:34 Or do you? defaultdict is in the collections module and helps you take care of this kind of thing.

08:46 For the most part, it’s like a regular dict, except it allows the creation of a default entry.

08:55 Anytime you do something with a key, the class checks if that key exists, and if it doesn’t, it creates the default. In the example here, I’m setting the default value to a list.

09:04 Just like with synonyms above, it’s important to note that you pass in the list class, not an empty list. You don’t want an instance; you want defaultdict to create the instance for you.

09:16 But now I can act as if that empty list was there. Even if I haven’t created an entry.

09:26 Because of the default, I can append to something that isn’t there. Because by referencing it, the defaultdict creates the list, allowing that append. Examining the class is a little more verbose than a regular dict.

09:39 You can see what it defaults to and what is currently inside of it. One thing to keep in mind is that any reference causes the default to happen. So by examining a not yet existent key,

09:54 you actually create the entry as a side effect. This is mostly a good thing as it’s what causes the class to work. It can mean that if you’re using this to deal with sparse data structures, you have to be a little careful that you’re not accidentally creating empty entries when you don’t want one.

10:11 Like a lot of things in Python, this really only saves a line or two of code and that I guess isn’t a really big deal, but it accumulates over time. Every little line or two is one less line you’ve gotta maintain and think about.

10:22 Have you used these? I have. I find it useful in lots of ways. Just the idea that, like you said, you don’t have to create a bunch of if-then-else statements sort of stuff to kind of, you know, confirm that this is properly ready to go here.

10:39 I find your person saying “hello” in this third case. A little rude though, saying “later, dude”,

10:46 my hello was a goodbye. Yes. Yeah, exactly. Well, those are antonyms, that’s why. Oh, that’s right. Okay. Sorry. That’s quite alright. There you go. I’m trying to remember.

10:56 I think like 98% of the time when I do this, it’s a list. Like it’s almost always this pattern. I’m trying to remember if I’ve ever used it for anything else.

11:07 To learn more about functional programming, read this tutorial, or for context managers, this course and tutorial will help. Earlier you saw the guitar to denote Mr. Bailey’s courses.

11:18 The eight ball is for mine. I’ll let you guess whether I play billiards or just like hanging out with mystical plastic spheres. Outcome is unknown. Ask again later. Ask again later.

11:28 That’s right. The next lesson covers a monumental time in Python history. The 2 to 3 transition.