Async Python Tutorial: Foundations for those with no prior async experience

this tutorial on asynchronous programming in Python assumes no prior knowledge nor any experience with asynchronous programming in other languages as a result it'll contain content that you won't find in a single tutorial elsewhere and with that brief introduction over less chatter and more Python a generator is a function that produces a sequence of results instead of a single value they contain one or more yield statements they can be awkward to learn at first because their behavior differs from normal functions for one when you call a generator function the generator object is created but that object remains an imposed state it doesn't just start running automatically we can see this with our first example which takes an input integer while this remains greater than zero we yield it and then subtract one from it the way if normally using a function like this is in a for loop here we've done that and passed in 10 and see our results ten-to-one of course as with any function we can assign this countdown 10 to X whereas we would normally expect a result here we can see that we have the generator object itself in a suspended state calling next on it will execute every line up to and including our first yield but nothing further when we run next again execution resumes first with our line n - equals 1 and then as n remains greater than naught we yield for a second time when there's nothing more to yield we get a stop iteration exception of course if you don't believe me you can put unique print statements in between each line here to see what's executed and when a useful exercise that I recommend first let's take ourselves from generators to co-routines pep 342 introduced yield as an expression which meant that you could now use yield on the right side of an assignment if you knew nothing of co-routines I bet you could take one look at finder our next example function and tell me that this went into an infinite loop had some input next and whatever the user passed in as an argument string if that string was in the input text then the input text would be printed to the terminal but the confusion arises if your only experience to date has been of the yield keyword in generator expressions and generator functions if you use yield more generally then you create a KO routine which do more than just generate values but can also consume values sent to it when you call a KO routine nothing happens we've called finder with the string Python as our argument assign this to F and when we look at F we see our generator object just as before we know that we can send values in so let's send in some texts including the word Python as we've done here the way this is supposed to work is that the string that we've sent in becomes a sign to input text and then seeing as X is python and python is in this new string that we've passed in that string will be printed to terminal but of course we just saw right now in our previous example that when you call a generator function it starts off in a suspended State no lines of it have been executed as yet it's only when you call next that execution begins the line containing yield is executed and then it pauses once more and indeed the exception that we get is that we can't send a non non value to adjust started generator this first step is known as priming the KO routine and it can be done either by sending none or by calling next and this only needs to be done once at the very beginning now any texts that we send in is assigned to input text and if input text has X in it in this case B string Python then that input text is printed to terminal exactly the behavior that we see and I'll just demonstrate that sending in nun Prime's our co-routine just in the same way that next does the kuru team closed method shuts it down as it could otherwise run indefinitely and indeed this is what the interpreter calls during garbage collection you can catch this though with normal exception handling of the generator exit exception and this is if you need to implement specific cleanup code furthermore you can also raise exceptions inside a KO routine with the throw method and these originate at the yield expression to avoid getting mixed up now that you're clear in your mind on the topics covered so far it's best not to mix the two concepts of generators and co-routines it's all too easy to get distracted because methods meant for ko routines are sometimes described for generators this is for the most part erroneous despite the similarities generators and ko routines are basically two distinct concepts generators produce data for iteration co-routines tend to consume values their consumers of data there is use of having yield to produce a value in a KO routine but it's not tied to iteration if you've come across yield from before then you might have seen it explained away as merely being shorthand for yielding values from an iterator including other generators but it's real you slice in allowing delegation to a sub generator or in other words any next or send can be passed on to nested generators acting as a tunnel that passes data back and forth and so this is at the heart of how Co routines operate from Python 3 5 onwards the preferred way of declaring co-routines is with the async/await syntax that we'll now go over these are known as native co-routines to distinguish them from generator based Co routines that were used before this native syntax was introduced and you still might see these around from time to time they are decorated with the async IOSCO routine decorator to turn our attention to native clear routine syntax for a moment let's quickly define a regular function as we know and love it Griet takes an input string and returns it concatenated with the string hello to define a native co-routine we use async def note that it's completely legal just to put a past statement here but to start off we'll return that same string as above both function objects look similar the difference occurs when we call them call our first function output as expected call our Co routine while the outputs different calling the Co routine has left us with a co routine object we've assigned this object to G and we send none into it what happens seems quite strange we get a stop iteration exception and the value attribute of the exception has the return value that we defined earlier her routines are there to be driven by something else as we'll do here with our run function earlier we saw that when we included yields in a regular function we had to prime it by sending none or by calling next on it with native co-routines we can't call next as it's not an iterator so we'd get a type error the way to get the data we want is to send in none the run function sends in none catches the stop iteration exception and returns the value attribute and in this way it drives our greets Co routine a sink functions can call other async functions but if you're going to call another async function you have to use the await keyword before it before we get on to event loops and how you'd use everything we've talked about together in complete code let's run a few examples our async main functions runs the async greet function and prints the result to screen where can you use await let's define an async function that computes numbers in the Fibonacci sequence it's not a particularly efficient way of computing Fibonacci numbers and I ask that you overlook this you'll see that await is particularly robust in where it can be used and so it's easier to look at where it can't be used before that though you see that it's all too easy to forget to put the awaits keyword when calling another async function here we've forgotten to put it before fib 15 in our main kuru teen this is the error message that you'll get fortunately it's descriptive enough not to make debugging harder than it needs to be getting back to where awake can be used see in cell 93 that we've even used await as a dictionary key a wait can't be used outside an async function in cell 95 we have await outside a function and it has produced our result this works now in the repple it wouldn't work if you placed it in a module outside an async function and you ran it we can't await an object that isn't away table you there's no restriction on defining async functions as instance class or static methods you now knowing everything that we know let's run some examples with event loops compare synchronous code to asynchronous code and talk about async IO you here time sleep one represents work that would take place on a real-life server we'll take it to represent waiting for network based input-output as opposed to intensive CPU work this program counts the three three times synchronously taking a look at the output we can see that each cycle of count running to completion before the next cycle begins you the asynchronous equivalent on the other hand looks something like this this took three seconds to run as opposed to nine seconds the first count cycle was started and as soon as it hit the awaits sleep one Python was free to do other work for instance starting the second and subsequently the third count cycles this is why we have all the ones than all the tubes then all the threes in the output programming concurrently can be a very valuable tool multiprocessing has the operating system do all of the multitasking work and in Python it's the only option for multi-core concurrency that is having your program executed on multiple cores of a CPU if you use threads then the operating system is still doing all of the multitasking work and in cpython the global interpreter lock prevents multi-core concurrency in asynchronous programming there is no operating system intervention there's one process there's one thread so what's going on well tasks can release the CPU when there are waiting periods so that other tasks can use it the classic example is of exhibition chess Holger is playing 24 opponents she typically makes her moves in five seconds and the opponents make their moves in 55 seconds these chess games average 30 move pairs each game runs for half an hour so playing these 24 opponents sequentially would take 12 hours that is 24 times half an hour this represents synchronous programming in asynchronous programming Holger will make her move on the first game and while she's waiting for the first opponent to decide what they want to do for their turn in those 55 seconds she moves on to the second opponent and makes her move and five seconds later moves on to the next opponent by the time she comes round to the first opponent again the opponent has made their decision moved the chess piece and polgár is able to take her turn so a single move on 24 games will take her 24 times five seconds that's two minutes so in this case those 24 games are completed in one hour 2 minutes times 30 so we've gone from 12 hours - one hour by taking an asynchronous approach as opposed to a synchronous approach asynchronous frameworks need a scheduler usually called an event loop this event loop keeps track of all the running tasks and when a function suspended it returns control to the event loop which then will find another function to start or resume and this is called cooperative multitasking async IO provides a framework an asynchronous framework that's centered on this event loop and it efficiently handles input/output events an application interacts with the event loop explicitly it registers code to be run and then it lets the event loop the scheduler make the necessary calls into application code when the resources are available so if a network server open sockets and then registers them to be told when input events occur on them the event loop will alert the server code when there's a new incoming connection or when there's data to be read if there's no more data to be read from a socket than the server then yields control back to the event loop the mechanism from yielding control back to the event loop depends on co-routines co-routines are a language construct designed for concurrent operation the cur routine can pause execution using the awake keyword with another ko routine and while it's paused the co-routine state is maintained allowing it to resume where it left off one co-routine can start another and then wait for the results and this makes it easier to decompose a task into reusable parts this example asks two phases that must be executed in order but that can run concurrently with other operations the awake keyword is used instead of adding the new co-routines to the loop because control flow is already inside of a KO routine being managed by the loop so it isn't necessary to tell the loop to manage the new co-routines for now it's sufficient to know that async functions are awaited but there's more detail behind that but isn't particularly complicated and has explained well in the documentation I hope you've enjoyed this primer and introduction to asynchronous programming in Python this is part one of a two-part tutorial is absolutely necessary to put the foundations in as always I would be grateful to hear your feedback your comments and your requests for future tutorials take care

Channel: Live Python

Views: 52,916

Rating: 4.4393444 out of 5

Keywords: async python, async python 3, asynchronous python tutorial, async python beginners, async python tutorial, python async, python asyncio, asyncio python, async await python, python async await, python tutorial, python 3 tutorial, python 3, python

Id: 6kNzG0T44SI

Length: 16min 59sec (1019 seconds)

Published: Mon Sep 23 2019

Reddit Comments

[u/PDiracDelta]

audio is scrambled; around 6:40 the video falls silent (happens multiple times), please check and re-upload

EDIT: OK maybe this is how you intended to make the video. It's a bit confusing, the text on the screen (around 6:40) is going very fast while there is no explanation. Also, while you talk very slowly at the beginning, sometimes you go way too fast (for a noobie like me) in later parts.

Looks like interesting content and I think if you polish the timing of the audio a bit better it will be a great video!

[u/KawaiiScythe]

Thank you! This will help me a lot!

[u/Niohzxs]

Amazing simple and clear

[u/iwpanda]

Thank you

[u/feclar]

Subscribed

I really like how this dude lays down logic

[u/ravenchilde]

This is timely bro, I had left myself a note last week to figure out new python 3 features and Async

[u/Mateo2]

Is the voice computer generated?

[u/ReDucTor]

Starting with the lower level building blocks of yield/yield from might end up creating confusion and also confuse the use-cases for them.

I'd probably introduce them in terms of solving issues blocking functions and how they introduce the ability to suspend and resume a function and how there is a scheduler (event loop) which will resume them