3.1 → 3.3 - Exploring yield from & unittest.mock

00:00 The previous lesson covered the 2 to 3 transition. This lesson talks about Python 3.1 through 3.3, the yield from statement, and unittest.mock module.

00:13 So moving forward into Python 3.1, which is 2009 into 2012, the OrderedDict was added. Up to this point, Python dictionaries had always been an unordered collection.

00:25 So from the collections module you could import the Order edDict and keep things ordered for your dictionary. And along with that, the Counter was added.

00:35 And by adding a __main__.py file inside your module, you could run the module just by using its name. In Python 3.2, we’ve discussed this before back when we were discussing Python 2, argparse was added and the idea again to replace optparse and then going forward, extensions now had a stable application binary interface or ABI.

01:00 So all extensions from 3.2 be compatible moving forward.

01:06 And to continue learning about this more, we have Python’s collections, a buffet of specialized data types as a tutorial. So if you want to learn more about the collections modules and the things that were added, and if you’d like to learn a little more about how you would include the __main__.py file in order to be able to run a module directly, this code conversation that was with Geir Arne and Ian Currie was a really great resource for you.

01:32 You also learned a lot about constructing your own modules.

01:37 Next came Python 3.3. Notice its lifetime here was from 2012 to 2017. That’s the first release with a five-year window that has become the norm. Now the release dates aren’t quite consistent yet.

01:49 That once a year thing didn’t start until 3.8. Feature-wise, Python 3.3 was the first to include venv for dealing with virtual environments.

01:58 The solution of virtual environments predates this through the third-party virtualenv library. I remember Mr. Bailey once asking me what I used and I still use virtualenv, and that’s because I was using it before there were other tools and I’m an old man stuck in my ways.

02:13 There you go. Oddly, Python 3.3 also added the u prefix for strings. This was purely for backward compatibility. If your script needed to run in both Python 2 and 3, you no longer needed to do fancy things to deal with those Unicode strings.

02:27 You could just use the u prefix again. You might recall that Python 2.2 added generators and with them the corresponding yield keyword.

02:35 Yielding from something that yields was a bit problematic though. So yield from got added to deal with exactly this challenge. There are lots of things in this release that I use all the time.

02:46 Another feature that I like was the introduction of the mock module. It allows you to fake certain behaviors out when running tests. Rather than explain more, how about some examples?

02:56 Let’s look at yield from and mock in the REPL. To better understand yield from, let’s go through this step by step, starting with a list, then turning it into a generator.

03:08 I’m declaring a function

03:13 and this function returns a list by calculating a comprehension containing the double of each of the items in the function’s argument.

03:21 Now if I run it,

03:25 no surprises here. Back when I introduced Python 3.0, I mentioned that a bunch of the functions that returned lists got changed to return generators instead.

03:34 This makes them use less memory. And if you don’t end up consuming the whole thing, it can be faster as well. To see how this works, I’ll write a similar function that uses the generator syntax instead.

03:47 This time I’m tripling, looping through the iterable that’s passed in, and for each item I’m using the yield keyword. When your interpreter hits this statement, it returns the value in place.

04:02 So if you loop over the return result from this function, each value gets yielded into the iterator. That means you don’t need the whole result in memory, and if things get cut short, you don’t waste any time creating the other values.

04:15 Let me run this. If you’re not used to this, it looks a little weird. Since generators are only meant to be interacted with, you get back a generator object rather than the values inside of it.

04:28 That’s kind of the whole point. To actually run it, you either have to use it in a loop or you can pass it to the list() constructor. The list() constructor takes an iterable, which in this case iterates over all of our generated values and puts them in the list.

04:44 Okay, so that’s generators and yield. Remember those functions that used to return lists and now return generators, like range?

04:51 What if you want to use one of them with yield?

04:56 So a function that counts down from a value,

05:02 and I’m yielding the range function. I’m going to run it, it makes me kind of hopeful.

05:12 And then when you convert it to a list that hope evaporates. This is the worst kind of problem. There’s no exception, just the wrong thing got put in the list.

05:20 So instead of getting the values, you get a reference to range’s own generator. Not really helpful. I’m sure you’re two steps ahead of me. This problem is why yield from exists.

05:31 Let me redo countdown.

05:37 Instead of yielding to range, I yield from instead.

05:43 And this time it worked. This example’s a bit of a simplification as you likely won’t need to yield from range, but if you are writing more complex generators where one generator calls another, yield from prevents you from having to do funky things to get them to expand out. Before explaining mock, let me show you the problem that it solves.

06:03 Say you’ve got a function that returns True if today is Friday the 13th. Import the datetime module, which is how I’ll figure out what today is. Define function,

06:17 get the current day. For clarity’s sake, I’m going to print it out as well.

06:30 And here’s my comparison. weekday() zero is Monday, so I check if weekday() is four, meaning Friday, and if the day is the 13th, if it is, this will return True.

06:41 Let me try it out and nope, not Friday the 13th. Now how can you test this function? Well, you could wait for a Friday the 13th to make sure it works or you could use mock.

06:53 What mock does is it allows you to fake out what the today() function returns.

07:00 There are several different tools for this kind of faking out stuff in the mock module, in unittest. For this case, I’m going to use patch.

07:07 patch can be used as a decorator or as a context manager, and it changes the behavior of the code that it wraps.

07:22 Here, I’m using it as a context manager. There’s a lot here, so let’s do it a piece at a time. The first argument to patch is what is being replaced.

07:31 Note the __main__. This is different than the __main__ that Mr. Bailey was talking about earlier, which is a file. This is the __main__, which is the module name.

07:39 Because I’m in the REPL, that’s actually what the default module is. If it were in a script here instead of __main__, you would use the name of the script.

07:49 The second argument to patch indicates what module is being patched. This isn’t always necessary, but datetime isn’t pure Python.

07:57 It’s C code underneath. So without this part, it’s not gonna work properly. And since I’m using patch as a context manager, what gets put in my as clause here is a mocked version of the object.

08:09 I’m still not quite there yet. Just having mock isn’t good enough.

08:21 To test our code, I need to make the mock do something. You can arbitrarily add attributes to a mocked object. So here I’ve added both a date object and then a today object.

08:32 So my mock_datetime is a mock of the module. It has a pretend fake date, which is inside the module, and then a pretend function today() inside of that. This allows me to supplant the real today() function, and then the return_value attribute in a mock tells the object to return this value when the fake function gets called. With today mocked out, any call to it inside of the context block will return Friday, February 13th, 2026.

09:03 Inside my test, I’m gonna call the actual function and that’s it. There you go. Our function works. This is a bit messy and it can be a little fiddly to get the thing patched correctly, but it’s really, really powerful.

09:18 When testing, does your code connect over the network to a system you can’t control? Mock it out. Is your code reliant on anything with a date or time? Mock it out. Are you trying to run a smoke test on a wrapper library that calls something that takes a really long time?

09:32 Mock it out for performance reasons, then test the underlying code elsewhere. There’s lots of good uses here. And one of the important things you’re doing with that is including it in that context manager.

09:43 That’s right, because this is unittest, 99 times out of a hundred, you’re going to be doing this inside of some test code. You usually aren’t going to use it in normal code because you’re essentially mucking with the behavior of things, right, which you don’t typically want to do.

09:58 I find this is one of those things I have to look up every single time. Like I, you know, quite honestly, it took me about 10 minutes to get the right form of this to work with datetime.

10:07 So there’s a little bit of Googling involved, but because you’re not actually mocking the module itself, you’re mocking where it loads the module. That’s one of those things that sometimes takes a little bit of trial and error till you get it right.

10:20 Yeah, it is a little fiddly, but it’s, it’s really, really powerful.

10:25 I’ve only scratched the surface of mock. There’s a Real Python course and tutorial if you want to learn more. Same goes with generators. This course and tutorial cover a lot, but it actually doesn’t touch yield from, but if you haven’t done generators before, this might be the place to start.

10:40 And this is a course on a lot more than just generators, but it does have a section on that yield from piece. So if you want to learn more about iterators and iterables, there’s great stuff in here.

10:51 Or if you’re just interested in the yield from piece, go look for that subsection in the course.

10:58 In the next lesson, you’ll learn about enums and how to use pathlib.