python: what is weakref? (intermediate - advanced) anthony explains #366

hello and welcome to another video in this one we're going to be talking about weak references in python and what they're used for as well as a few of the fun little things in the weak graph module uh but anyway let's jump into it okay so the main well the way i think about the weak reference module is it's a way to sort of bypass reference counting in python um let me just give you a short little demo of reference counting so you can kind of see what i'm talking about we're going to make a class and we're going to give it a dell method this is the method that gets called while the object is garbage collected normally you don't implement these yourselves because you know the garbage collection can happen at any sort of time and it's not really predictable it's not really a destructor as you would think about it in other languages but you know it kind of works like that but let's do i'll just say bye and do self and if we make one of these objects and actually i'm going to show you a little quirk of the interpreter i've actually done another video on this already so i will link that in the description if we construct this object you would think that there would be nothing referencing this after this statement and it would you know garbage collected immediately but we see that that doesn't happen we print out the rubber here and this object still lives and the reason it still lives is in in c python at least or at least in the interactive interpreter the last expression is stored in a special underscore variable and so you can see this object lives on now if i were to make some other expressions such as one you'll see that the reference count fell to zero and it caught garbage collected and to demonstrate this a little bit further if we assign this to a value and we use sys.getrefcount which is a way to ask about the ref count of an object you'll see it actually returns two which is interesting here one of the references is this assignment here so this is one reference to this object so it has a refcon too but actually when we pass it in as a parameter into this function the parameters of a function have a hard reference to the value as well so it increments to 2. now of course if we assign this to something else so if we do d equals c so another reference here you'll see that we now have a reference count of three and that's stable if we do this a bunch of times if we unassign d here we will decrease the reference count back to two and of course if we unassign c uh we've you know sent the sent the reference count to zero and that will cause the object to get garbage collected and of course this is all c python specific the garbage collector works differently in pi pi and other python implementations as well okay so now that we have kind of a base idea of what reference counting is a weak reference allows you to create an assignment to an object but not increment the reference code so we go back and make our same example here we make a c we implement import the wecraft module and we make another assignment but in this case we use weakref.ref of c and if we do sys.getrefcount it will still be 2. even though we sort of have a c object here you can see that it is a weak graph that points to this particular c object and the way the weak graph object works by default is if you do if you call it you will either get the object back or you will get back none if the object has been collected so for instance if we do that same c equals none here actually we can even assign it to some other value just to differentiate between getting none back and getting that value back if we assign c here uh which should have gotten collected why did it not get collected this is still going to give us our object oh because um because we have an expression right when i called this expression here it stored this in underscore which is super confusing let me let me redo this demo where i don't actually uh store it in the underscore accidentally so we'll do c equal c we'll do d equals we craft dot ref of c and we will print the call of d this way we don't store it in that underscore variable and now if we assign c to five this will cause c to get garbage collected and if we try and call d again we'll get back none uh and so this this call method is how you explicitly access that underlying object and this can be useful in kind of two main cases that i've seen one is to create caches because caches are often global variables and naively if you just stick things into them and uh you know they get garbage collected later and they're no longer referenced in the program you naturally have a memory leak in this global cache but if you use a weak ref uh they don't keep that it doesn't keep that object resonant because it doesn't have a strong reference the other case that i've seen is to improve the performance of garbage collection in the case of circular references let me just demo that really quickly now in python and or at least see python other implementations probably also have this as well uh the garbage collector is smart enough to garbage collect circular references uh but it won't do it immediately it has to do kind of a deeper garbage collection so let me just show you an example of that if we make a c object and we assign c dot c equal to itself so we have a cycle here so c.c.c.c.c is you know still referencing this object here you know do an expression to get rid of the underscore variable uh and if we assign c to none you'll see that it's not garbage collected immediately and we actually have to trigger a deeper garbage collection which normally happens as part of program flow but you can trigger it directly by doing gc.collect and you'll see here that gc.collect was able to find that reference cycle and clean it up deleting this object even though nothing actually referenced this object anymore except for itself and so that's kind of a circular reference now you can uh you know avoid the circular reference by using a weak graph and we're actually going to use a different weak graph object a different weak graph um method or function this time we're going to use weakref.proxy which means that you no longer need to do this call to access the underlying weak graph you will either get you know an attribute error if it's collected or you will get um the it'll proxy to the attributes of the actual object so if we do c dot c again uh let's just do c dot x equals one just so that we have something to show that it works and we are going to do c dot c equals weak ref dot proxy of c and so we can still do this same silly infinite chaining because it's a circular object but now if we were to clear this reference here by setting c to none it should get immediately collected yeah you'll see we got immediately collected and that's because this reference here this attribute reference was not a strong reference so it didn't keep the object alive and the garbage collector was able to clean this object up much much easier now i'm going to show you uh you know that that's sort of the circular reference part of this let me also show you the caching part of this and for that oh sorry we're partially through the we crept talks for that we're going to look at either we key dictionary or weak value dictionary i've actually used i think it was weak key dictionary yeah i actually used wiki dictionary which is funny to say at yelp for a pre-request cache because the cache is a ephemeral object but you tend to you tend to generate derivative data from the from the request all the time and so if you make a weak key dictionary that is keyed on the request and it has you know some expensive computed value as the value um you know as the request cycle ends it will get purged out of this dictionary automatically so for that if we do some dictionary equals weak ref dot week key uh let's do weak value dictionary just because this c is not hashable uh weak value actually is it hashable it might be default hashball yes okay it is default hash full so we can make a weak key dictionary um you can see it got immediately collected cool uh so dictionary equals wecraft.weakkeydictionary and let's say we assign one of these and maybe we put it into our dictionary as some value further down in the code we accessed this and we're able to you know see see that there is a object in there but if this got collected let's see actually we do dick.items and you'll see everything that's that's live in this dictionary if we were to assign c to none causing it to get garbage collected i didn't get garbage collected where's the reference uh uh oh because this is this is our expression right okay i gotta remember the the magical underscore very i'm glad i pointed that out at the beginning because it would have been way more confusing okay let's do this demo again [Laughter] okay make a dictionary uh diction oh whoops dictionary sorry i did ctrl z there uh and we assign c to two right we need to make a c again we assign it to two and if we print list dict.items instead of just having the interpreter show us that you'll see that we have this and now if we assign c to none it will get garbage collected and this dictionary automatically cleans at those objects inside of it so you can kind of depend on you have a dependable dictionary cache uh but it doesn't keep the objects alive anyway that's that's kind of the basics of weak graph it's a way to keep a reference that is a non-strong reference which doesn't keep the object alive hopefully this was useful if there are additional things you would like me to explain leave a comment below or reach out to me on the various platforms but thank you all for watching and i will see you in the next one you

Channel: anthonywritescode

Views: 791

Keywords:

Id: GGKerIMqHCk

Length: 10min 39sec (639 seconds)

Published: Fri Dec 03 2021