Going Async with Python!

[Music] [Music] [Music] [Music] [Music] hello uh I I don't know if you enjoy the music as much as I I but the thing that you can't see while that video is playing is that I'm just kind of sitting here moving around to the beat it's uh yeah it's a good good way to to start the stream uh welcome to the stream uh especially if you haven't been here before um I always encourage people to just kind of let us know in the chat kind of uh who you are where you're joining from it's uh it's kind of otherwise I'm just kind of speaking to a camera uh and it's really nice to get an idea of of who's joining um and kind of how far away you're joining from um although some sometimes we do we do get people from Edinburgh which is where I'm streaming from right now um it is a miserable wet gray day outside um so I have nothing better to do than to sit in doors and um and fumble around with ating code um and Mong B so hopefully that's something that you'll enjoy that's that's what we're going to be doing over the next hour um and we'll be like covering some tips and um well we'll be trying to get asyn code to work that that works when it's blocking and um talking a little bit about motor and maybe talking about some some sort of modeling techniques with uh Mong B we'll we'll kind of follow the thread where where it leads um I see aned is with us um this week hey anid um nice to see you again anid is one of our regulars and uh is often one of the people that helps me out when I'm struggling to get some cod to work so it's always reassuring to see see your avatar in the chat um thanks for joining again um right so I guess I have uh kind of talked enough um let me get this uh video screen out of the way um and then I'll give a quick intro to anybody who's new uh as to what I'm actually trying to achieve oh that's not big enough there we go so um I'll cover this in a moment uh let's just start with the GitHub page so what I'm doing is I'm building a python library for interacting with mongodb um there are a bunch of Li that already do this from the kind of lower level libraries that are maintained by us at mongodb and we've got two drivers we've got P which is blocking uh works well with um some of the more traditional Frameworks maybe like d Jango and flask um anything that isn't using ASN Kyo basically and then we've got motor uh which is U functionally equivalent to Pongo but it's designed for async Frameworks like tornado async Kyo um mainly these days it's async in Kyo um then on top of that there are these libraries called odms uh stands for object document uh mappa and that's those are Loosely they're mapping libraries they're kind of equivalent to ORS if you've worked with an OM in um with relational databases this is kind of the equivalent for a document database uh but they they tend to work they they all have different feature sets um some of the more interesting ones uh work with pantic so sort of built on top of pedantics um schema uh design and um validation um capabilities and also kind of serialization capabilities that you get with things like Fast API uh but they all kind of work in a similar way so you define um objects that map directly to the documents that you're storing in your database it's not so much of a mapping it's very sort of thin mapping layer and I wanted something that um that would just give me a a richer ability to represent documents for in the correct way in mongodb itself and then map those into objects that are nicer to deal with in Python so you've got your sort of business representation in your application um but then you've got your kind of physical representation on disk that's optimized for queries um that kind of thing so and these this this generally very good tip for when you're building applications with mongodb if you want them to be uh fast and cheap and scalable to run um you need to think about how you model that data it's not just a matter of taking Json and like dumping it into your database in most cases um we have uh somebody uh here who has sent me two uh hand emojis um hello I'm afraid I can't read that character set is it Korean um so I can't pronounce your name but it's h it's lovely to have you here thank you for joining um and I've been introduced to a new Emoji so it's nice to nice to see those too uh right so yes this is my project called dockbridge which is an odm with a more emphasis on the mapping so um to give you a rough idea there's there's a Manifesto at the start of the read me um the idea is that you might store your data um like this this is actually a straight mapping uh of some beats on data that stored in the database um I can define an empty document subass to represent that users profile in my application um and it will dynamically map any of those fields to the object so it has a field uh in the document called user ID and so my object now has an attribute called user ID so far so good um then it has the ability to actually specify Fields so I can specify my user id field and I can say well it's stored as a string but actually it should be a number and this will then pass it through the int function which returns a numeric representation and this then just standardized is the result that comes back when you look up that value this was super fun to write it's um it's using descriptors behind the scenes it's kind of similar to The Way D Jango does its um modeling um and uh yeah that was part of the the reason for doing this series of streams uh we've had a few episodes so far um they're all available actually the most of them are linked at the bottom of this read me that I'm I'm uh skimming through right now um I hope I have a link to this in the description if you there should be a link in the description to this GitHub repo if not uh let me know in the chat and um I will paste up a banner on the screen or something um moving further away from just simple type conversion um well we can specify names so we got this full through field that allows you to it first attempts to find a full name field and then it attempts to find a name field and it returns whichever field value it finds first when you look up name um then there's the subset pattern so this is really what I had in mind when I was designing this Library so there's a pattern called subset um if you haven't if you're not aware of the mongodb design patents blog post you should look that up I don't think that's in the description for this video that's my bad I'm sorry about that but if you Google mongodb design patterns or click on the link in this read me um it will it will take you it it's a a list um of patterns that we've identified um that solve various problems like index is getting too big efficient um queries uh documents getting too big that kind of thing uh subset is um is a pattern where um one of the common questions we get asked is if uh I have a document that contains an array that's going to be continuously added to let's say a a list of followers for a user um what happens if it gets too big to fit in a single doc and this is really the answer to that question so there's two things going on here firstly your there's a limit of 16 megabytes on a mongod DP document that you never want to exceed um and secondly uh you really shouldn't have an array with more than we say 200 items in it because um that just makes it slower for mongodb to kind of Traverse that document and pull out data in the document so 200 is a rough uh rough ballpark for a sort of limit of what we should keep in an array um So to avoid both of these problems you can store um extra items in another document you can store them in another document in the same collection but usually you'd store it in a separate collection um so the sub the subset pattern specifically says well you just store the first 200 followers in your main user profile and usually you'll never have to scroll past that right if you go to your followers page and a social network you usually don't go past the first say 200 um so that's actually much more efficient you're going to be send sending less data to your client you're storing less data in the document um and you're storing this colder data like somewhere else so it has kind of less of an impact on the performance of your database all good um the problem comes when you actually want to query that data so it's that if you do want to start you you if you if you request that user profile and then start looping through the followers it's fine for the first few and then you've got to re in your code you have to recognize when somebody's reached the end of that and then you have to do a query to look up the extra items uh and then you have to kind of dynamically append them to your document um in or or start appending them to the the page in your web application whatever it may be um I wanted to hide that away that was really the the first sort of thing I wanted to hide away in my application um sorry anaid I know you've seen this a few times uh but I did I feel like it's kind of important to explain the background to this Library so here this this represents my solution to this problem in my followers field um followers attribute in Python uh I have this defined the sequence field and I've said it contains followers and that just means when you looping through will return you instances of a follower document subass um but if you get to the end when they're looping through I want you to silently do another request to get the rest of the followers and then stream those in so I've got the superet collection that says that my other followers are stored in this collection called followers um and then I have um uh I have the actual query to get each of those followers it's a little bit more complicated than it needed to be because I've bucketed the followers so I won't explain the query but I will say it's just wrapped up in this Lambda so it has to this superet query takes a callable and the callable um returns it takes the object so that would be this current user profile instance uh and then it returns a query to look up the followers for that um that profile um the reason for that is that the profile contains the user ID that then allows you to filter down um to the correct documents in that followers uh collection so the way this is used and this is really the point right that looks like a lot of code for for setting up um an object um but the way it's used is these three lines here so you you can look up a profile with some Bon data and then you can start looping over um the followers and it will just Sprint out all of the followers one by one and we're taking advantage of a few things the fact that curses are lazy um so those are those it's just running the query and then you then streaming through the cursor to add in those extra followers so if you never get to the end of the followers it's that you haven't loaded all the data from the database it's just whatever batches of data have been sent uh as I say if you want to catch up on where how this Library started there's some videos here you can you can use to catch up uh I think most of the episodes are in there not quite all at the moment but certainly lots of lots of stuff to view um this was built as a blocking Library using Pongo but I want to use fast API um there's there's a new thing in API there we go uh there's a new thing on the fast API website let me make this big so that you can actually see um we have mongodb logo on a fast API website because we're now sponsoring fast API um this is the start of a program to kind of support some of the um open source initiatives that we think kind of align um with what mongodb is doing so um we helping out the fast API project so I want my library to work with fast API but fast API is asynchronous um so I'm taking my blocking library and making it work in an asynchronous way um and anybody who's been kind of watching The Stream over the last couple of weeks um we had an interview with my colleague annaya last week that was absolutely excellent interview uh talking about this cool product she's been playing with called neur um but uh the week before we're working working on making this Library asynchronous um and this is where we are so I have tests usually got pretty good test coverage for this Library it's been quite nice sort of writing my tests and then um making the failing tests pass uh it's especially useful with asynchronous code I think because there's a certain amount of setup um to to sort of getting the code to run rather than just writing a few lines of script uh and the library itself isn't very big at the moment it's 230 lines of code and one another one of the things I I really wanted to demonstrate here is although this is an experimental library and it's not ready for production it doesn't take a lot of code sometimes if you wanted to build your own odm with your own kind of design in mind like it's not actually a huge amount of work and it's maybe worth it for the you know to to sort of Hide Away some of the complexity that you've got behind the scenes rather than repeatedly kind of like writing logic like looking up extra data um because mongodb can be a little bit application heavy um and so it's kind of much like it's it's important to design your data correctly uh for efficient lookup um it's kind of important to design your application so that you're not writing the same sort of uh lines of code uh to account for those patterns again and again and again um so uh this is this is my code uh it does not run at the moment so let me just uh bring up a terminal uh and it says that I've got a virtual environment activated let's just check very nice it worked um I'm on a new computer this week so um I have uh gone through like setting up this project and just making sure that things run but uh it's always possible that we may uh get some teething problems um so let's let's hope not um so we've got failing tests and this is because I haven't completed the transition to async um uh C- workking um specifically updates I think but oh actually no this is sequence field and sequence field suet so I have two test files and I was only running um test acing Kyo um last week uh oh my head's in the way um let me bring in a panel on the left hand side so that you can actually see the things that I'm running on the line apologies for that uh so yeah tast test ASN Kyo works fine but the other file test do bridge doesn't so we got these two fa failures and basically that just means I need to Port over my test from the Asing Kyo uh file into my main um uh folder um so uh I'm just checking the chat now and I can see uh Monty has said cheers um you're welcome Monty uh not entirely certain what you're saying cheers for if you want to let us know in the chat that would be cool um but I'm I'm glad you're happy that's that's awesome um and it turns out that I was right about I can at least recognize the Korean character set um so that's uh that's something maybe the next thing is to actually learn Korean um and yeah that would be awesome uh so here I've got my tests that are failing let's go and have a look at them so we've got test sequence field and test sequence field superet so you pretty much get an idea about the thing that's not working which is the last thing I just showed you so we got test sequence field um async generator object is not an iterator and yeah this is yeah so this means um I am loading profile um getting the first item out of followers and checking that it's a follower and it isn't um because you can't call next on an asynchronous uh iterator um so and that's what's now being returned because this is now an asynchronous um there you go it's an async generator uh so I know I don't actually is there an a next function I'm not quite sure I feel like there's some stuff that just hasn't been ported over to aceing Kyo um so Monty is just rolling in glad to have you here Monty thanks for joining us um I hope you enjoy the stream um so yeah let's let me have a look asynchronous I'm gonna I'm I'm gonna Google um I got told off by loose one of my colleagues last week for constantly going the wrong way when I was um going between screens but I got one got it right once uh this this week so that's good uh ASN iio this is what professional developers do all the time ASN iio uh next generator so using next on an async generator I like this [Music] um have a look at the answer it's got a tick we like a tick and aex where did aex come from oh okay and they're just Global I was expecting them to be in um Asing Kyo so right let's let's I hope we've all learned something new uh so um and I think I'm going to have to await that I suspect uh oh didn't like that oh wait only allowed yep uh and now I'm going to have to make my test asynchronous which was something that we were doing that I was doing last week this was really one of the reasons for keeping the tests um in a different file um and then if I check what I've got here I've got um this tag to just say that it's an async um test I have to import p because it's got uh yeah I might as well just import py test um and then run a test again ha uh there we go so that was nice I've solved the one uh test already let's let's hope the other one is just as easy um this may be a short uh There is almost certainly a shortcut for doing that in uh Visual Studio code but I haven't got it memorized um so uh you you just get to watch me type the whole function name in um so this one was a little bit more complicated because this is now doing the join so it's looking up the extra items and so I've checked the first one uh did I actually I don't think I did I just checked the yeah I checked the last embed did follower which was the 20th uh and then I um and then I pulled got one more out and uh that um and then check those oh maybe I got two more out you get the idea I'm getting one embedded one and one joined one just to make sure that they work um and I know the data this is all done on a real database I'll I'll give you a quick skim of the fixtures in a minute once I got this working oh and this is also async and there we go now there's one thing I've I've I fixed those aex but this I slice is not going to work with an asynchronous uh generator so I going to need to I don't think there's an equivalent in core python uh um oh yeah so um there's a couple of interesting comments here so um tan has pointed out that I should mention beanie uh and motor um in the Stream um and uh yeah absolutely should so we have uh beanie uh beanie is an awesome odm uh it integrates with pantic uh it's there's there's a synchronous version of it if you want if you do want to um use Pongo in instead of motor um uh that's called bunnet bunnet is a type of Scottish cap uh it was named by somebody who lives in Edinburgh and has two thumbs uh so yes uh yeah Roman the author of Beanie has been doing great work on that Library so if you do want a production quality odm definitely definitely check that out um and uh our friends with the uh Pink hand emojis has uh said that next Returns the next item in an iterator and that's absolutely true but these are no longer iterated these are asynchronous generators um so essentially I need I needed an asynchronous version of next and that's what aex is so that's why I've been changing the names of these um uh functions over so hopefully that kind of makes that a little bit clearer um if I do go too fast or I do something without explaining it properly um or I just explain something badly or you have any questions like do let me know in the chat um I do uh love to uh help people in on that mongod DB Journey that's why I'm here really um so okay uh follow a boundary I need something equivalent to I slice I'm going to do what I just did um and Google it because you know I believe the new version of this is uh ah and uh my friend is called H Han so uh thank you for joining us Han it's it's great to great to have you here um so uh where are we so async generator so this is um islice async generator there we go async version of ice so somebody's implemented a version here I mean this is one of the nice things about iter tools is that uh it tends to show the implementations of the utility functions that it provides a sort of part for the explanation of how they work um and so porting them is not too hard but I don't really want um I don't really want to paste in my own implementation though I am only using in one place and it is only in a test so let's do that for the moment if anybody knows of anything in the core library that can do this for me like do uh do drop it in the chat uh oh I'm trying to do stuff in the browser now um does anybody here use Jupiter notebook like I find when I'm because Jupiter notebooks um work in the browser I find I'm constantly trying to use my keyboard combination to switch between an ID and the browser when I'm switching whereas I really need to switch browser tabs and it's just just infuriating um one day I will stream a jupyter notebook and you will see how badly I uh I do that I am going to put this here um this is the thing oh and there's I need an enumerate oh this is this is going further than I really wanted to go uh yeah so this is just giving me yeah I quite like this it's it's you know it's nicely composed I will say it's um so we've got an async generator here called enumerate that takes an asynchronous um uh iterator and uh as it goes through it it yields um the item from the iterable and the number that goes with it so sorry the index of that item so it's basically a simple equivalent to an iterable or the um the sorry the enumerate function which is blocking and then this is an asynchronous version of iiz which uses that um oh and I don't have CIS so I'm GNA have to import sis oh yeah I so it's possible I may go and clean this up um in between live streams but I think at the moment like just to keep things moving um we'll paste this code in here and hope for the best um so we've now got the AI slice um [Music] and anid is just um sympathizing with my uh jupyter notebook problem and uh yeah missing all the ID short shortcuts I know what you mean um hello I I used to be a very heavyweight um Vim user I used to know a lot of shortcuts in Vim uh am I in the right function I can't oh there it is AI slice and just check that the parameters are the same because either simplified yeah this looks good okay um failed async generator is is not isable yes I've I've missed one of the weights I think so yeah I used to use themm a lot and then I retrained myself to type on a dvorac keyboard layout uh and so yeah if you've ever seen it before all the vowels are under your uh on the home row under your left hand um it's it's and I'm a lot faster at typing these days but all of the standard Vim shortcuts are designed for quiry keyboards and so I've I never managed to retrain to dvorac mapping for vim and so these days I just use uh gooy text editors um like a normal person so that's that's good um do uh so where um where am I getting this bug asyn generator is not say it's this I SCE here oh I'm wondering what that is now uh so we got field profile Su test field superet oh this is weird I'm wondering whether I just didn't save it now because that's nope H well this is exciting um so we've got last in bed uh so last in bed username is N brown so this this is not giving me the next item we're so close aren't we like you can just feel it um so what we got here we got ASN generator asent object so why isn't an asend object so what's actually what's happening inside here uh let's have a look at AI slice and see what it's actually doing now oh uh I said anid tends to help me out when I'm coding you're you're a very active reader it's I think a lot of people just sort of let the code wash over them but um you're very good at seeing this kind of thing uh let's have a look sorry test do bridge uh outline oh that's not working huh okay uh test field superet and yeah we've got await I've been just kind of nesting these AWA of um inside each other and then obviously that that uh started to hurt so async generator can't be used in an await expression I'm guessing that that doesn't get awaited because we're not actually looping through [Laughter] it uh T let's have a look at the code um so we did get a follower object where's my error async for item in uh related is this a different test um so uh Han has asked uh do we need a wait and the answer is is um yes when we're trying to move through the anything that's lazy so uh AI slice doesn't actually need to do anything but then we need to start looping through the result so normally we just call next right but in this case we're calling a next and that's driving this awaitable um and so we need to await it it see if you see the signature here um I was about to point at my screen and you weren't going to be able to see that so that's that was is silly um but I can't move my cursor because then I lose the popup but yeah you can see this returns an awaitable so yes I need to await it to get the actual follower um let's have a look uh at where this error is actually occur occurring so yeah so this is this is actually a problem with my code now which is quite exciting because I thought this code worked um so doing async for item in related and then related doesn't see to be an asynchronous iterator and I know why this is I know why this is this is because it's actually because of the test um the this um fixture that I'm providing to the function so again apologies to people who have seen this before but I have some fixtures um that I can use with py test uh I'm quite proud of them even though it's like five lines of code um so here we have um mongodb the mongodb fixture that I'm providing into that function and this the job of this is just to initialize a client um at the session level so at the start of the tests and um and then shut it down at the end ofal of the tests uh that's why it's a scope session fixture um and all it does is get this environment variable um creates a client um and it checks that the client is actually okay uh by sending a ping um because this is completely lazy doesn't actually contact Mong B at all you need to send a command or start iterating through some results to get to actually get it to contact the cluster and then it Returns the client which means that in tests that have this uh mongodb fixture that is going to be a client um from Pongo and that's the problem right because this is now asynchronous uh which means it should be motor um so let's just have a look at what I was doing in the test asyn Kaio tests um so this is taking a motor fixture now where is the mot I think the motor fixture may also be in comest there we go so this just does the same thing but for motor um so I'm going to change this test there we go and then I'm going to need here uh and yeah there we go so I'm expecting yeah so some of this stuff there's still some asynchronous porting that I'm going to have to do um as you can see some of these tests are working when they shouldn't because they're still working with with mongodb so we've now got this weird hybrid library that is partly asynchronous and partly synchronous so I yeah need need to tidy some of that up that was what I was expecting to do first today actually if I'm being honest I thought these failing tests were going to show me um that uh I had some more stuff to implement and in fact I didn't really um yeah I didn't really this is bugs in my tests which is is still good right I'm there still moving forwards uh so now we've got get attribute and this is interesting so this it's looking up something H this is really interesting so I'm looking up user ID which is not asynchronous it's just it's just a value providing the Bon and now it's trying to wrap that so for some reason like this is quite interesting I feel like so what line is that it's 56 let's go and have a look uh so this document you know this is funny like every so often I think I've got document sorted and every so often I come back and it's like oh still still got a bug um so the problem here is it's saying uh the future object is not subscriptable which implies that it thinks that underscore Doc is a future object and underscore doc should not be a future object that should just be um it's it's provided in the Constructor so but the problem is there is a lot of magic going on in looking up objects uh looking up attributes on this object so you can see here that I've actually overridden uh doc so if you look upore do uh it actually it looks it up on using kind of standard lookup Behavior so this is object get attributes that it uses um I feel like it should be using get at instead of get attribute that's interesting but we'll ignore that for the moment that may come back to bite us uh so why does it think that this is a future um I'm going to go back to my test and I'm just going to check the code that's being executed here because this isn't making a lot of sense to me at the moment um so I am checking did just check that this is the same test yes um and then oh this is not the right line why am I there um here we go so this is checking profile is user ID and this is now motor get database get collection ah this is why because I'm doing find one and not awaiting the result there we go so now I I didn't think I was actually using motor at this point um because in a lot of these tests I'm just providing raw Bon data to initialize the profile but in this one I'm actually looking something up so I need to it's IO and generally anything that's doing any IO with ASN IO is UNA awaitable um it's one of the quite nice things is that there's that that um there's a really seminal blog post I mentioned it before uh called what color is my function um talking about how async is kind of viral you know you you think you're just going to swap something somewhere like in a small part of your codebase over to async and you find it just like Cascades up through all the interfaces but one of the in interesting um look at that way I'm getting Applause in the chat got a great and a yay from Han and anid has given me Applause thank you very much by the way let me just talking about a blog post while I while I've got my test to go green so uh yeah that's I should should take the win when I get it um but yeah one of the nice things about the fact await Cascades through your interfaces is that it actually tells you something about the code everything that's awaitable is doing some IO of some kind that's the whole point right you yield uh your functions that that do IO need to yield for you to get efficient use of of your sort of your single thread that's running running your Asing Kyo uh loop um so yeah I once I once I heard it put that way yeah and anid says Turtles all the way down reference to one of my favorite authors um and uh thank you Alo um uh yeah once I heard it put like that like actually this async thing it's not a code smell it's actually it's like extra information um about your functions I I became a lot more relaxed about the fact that async kind of spreads through your code so um there we go uh right so I mentioned before um we have some tests that are now using wodb some that using motor so I'm going to do the simplest thing possible to identify those tests which is I'm just going to get rid of my mongodb fixture um if you like that uh motor fixture it's it's you know reasonably boring um then you may like my rollback session uh fixture which I don't think I've tested yet um so this um we'll need to take uh motor um and then I think start session yeah it's not been properly hasn't had the type hints added to it yet um so um I'm going to put an await there oh and this is async and I'm going to suggest that that's awaitable and that's away to ball and then we'll hope for the best um I somebody has said I look like Eric tenh hard and I don't know who that is um I will take it as a compliment um and yes very good point anid thank you I should probably uh get everything into I should probably commit everything before I go through this stage let's do that I mean my tests are going to fail now um but I'm this is where I am so uh I I use lazy git for this usually went horribly wrong when I tried to demonstrate this uh the other week but um I do these are operations that I do all the time um oh what was that unable to Auto detect email address okay uh this like really that is not the first time that I've had to commit um you know what I'm just gonna I'm GNA YOLO it we'll we'll just go because I I can't be bothered to look up the command for setting my email address in git uh this is very silly ah ten hog is a football soccer manager you know I knew I'd heard the name somewhere but I don't know what he looks like oh well I suppose if he looks a bit like me then uh then there's that bold man with a beard is uh generally the summary uh right let's run the test again so that we can see oh errors oh that's a lot of Errors uh we got six errors and four passes so let's uh go YOLO I'm gonna leave that up for a bit um it's always good to have a YOLO on the screen um right this is good probably is it good um let's go back to the tests you know this is what I like about test coverage right I'm I I I know that I know that I haven't finished updating my library um partly because now that I've I'm updating my tests like I know my tests need to be updated and then I need to update my code it's kind of a Cascade to just make sure that I've done everything I need to um so let's have a look at test mongodb client can I oh it's a link is it no it's not oh vs code you have failed me um test. bridge and uh test mongod DB client this is a nice easy one um motor uh motor this is the simplest way to check if you have um a uh a valid connection to your database uh otherwise it can come and bite you later when you're actually trying to run a query um but if you have a connection and you want to check it it's valid Um send a ping to the server and then it Returns the document it returns will have a field called okay which should hopefully uh be a float 1.0 I don't know why it's a float I don't know whose idea that was um and I probably shouldn't check that it's I usually check that it's greater than 0.5 because that just seems like a nice whoa it's zero if it's not okay is what I'm trying to say and I think I need brackets around this no oh because it's not asynchronous I think I need to tag that but let's just check I think there's an automatic way to get these to be tagged uh is let's try and get us to fail oops oh you know what I'm getting a warning here which is basically saying that the test is not is not being run because it's just una awaitable right um so um yeah this is a failing test now and I do need that tag on it there is a way to configure pest a Kyo so that I don't need to mark my asynchronous tests it will just do it automatically but for whatever reason when I was setting this up I did not use that um there we go we got uh one failed five errors that's good it's supposed to fail um oh we getting that future object is not uh subscriptable um oh of course I put my bracket in the wrong place um yes and that's telling me so now I should get it'll be 1.0 if everything's okay okay and everything is now okay so my test pass that that was that was more effort than it should have been for one liner code but hey it's working woohoo um okay now this is going to take motor and this is going to be testing the roll back session as well which I've just ported over to motor um so we'll see if that works shall we uh a wait motor um why does oh because this is acing and I think I need that y and we've got motor so this all runs within a transaction that then gets rolled back um so this is really a test um you notice I'm not using my library at all I'm using insert one and find one so because these operate within a transaction um I should be able to get this document back and then after the transaction's over the document should disappear there we go async generator object has no attribute in transaction um so it's not liking my roll back session that I'm returning I think I think that's what's happening uh this may actually be a problem because I haven't used uh transactions within um uh with motor before so I may have have to just check the documentation of that I mean that's a good thing I guess that I I need to I'm now using transactions with motor I um uh one thing to note is if we get asked this a lot about MB like does it have transactions or we get a lot of people who don't think it has transactions and it has for oh six years seven um and the answer is complicated right because transactions have a uh have side effects like on the scalability of your application like transactions fundamentally don't scale uh into in a sort of clustered server environment um and so you should use them judiciously like you don't need transactions quite so much in mongodb as you do in relational databases because in relational databases if you're updating a complex object you're going to be making updates in lots of tables and so fundamentally that sort of has to be done in transaction whereas with mongodb assuming you're updating data um that's all kind of stored together um those updates to a single document are are done automically anyway if you're sending a bunch of set commands to set different fields in a document that's all done um in uh yeah essentially transactionally but it's really cheap operation because the data is all stored locally so it's one of the benefits you get of using mongodb in high right um database environments so anid has said that roll back session is non async and it's using Mong goody B I thought I'd just fixed that didn't I fix that uh I was talking while I was doing it so I apologize if that kind of got missed but yeah this is this is what I've got at the moment um aned uh obviously didn't commit and push um so I think I've just got I've probably got too many weights here maybe um let's if it's possible yeah I I the fixture is not acing no the fixture is oh well that fixture is not async uh oh you're right thank you very much so there we go instead of a p test fixture it's now a pyn py test asyn Kyo fixture oh okay um Okay so transaction context can't be used uh can't be used in a weight expression so this is the second thing that I was saying was you can't use it uh you don't await some of these um so I'm going to guess start transaction uh doesn't uh isn't una awaitable what is that is motor client why doesn't it like that but motor client is defined motor client is defined here oh no it's there damn I was trying to keep my uh async dependencies kind of very tight um but I don't know why I did that I knew I was going to have to Port the whole Library over to as and Kyo at some point um whoa and yeah fixture Mong B not found okay that's good um so these are the errors that passed it passed woohoo and yeah awesome appreciate appreciate your help as usual aned it's uh yeah always very reassuring to find you on the stream so let's go back to the test and take the next one test sequence field I really want to click on those and it doesn't work so uh let me just this is going to have to be ported to async as well so I might as well just take that line uh I nice we've done this one oh wait test sequence field I thought I'd done test sequence field oh so yeah this is async and it's getting oh got the wrong oh interesting I wonder uh I wonder if it's even using that I don't know that it's using that um fixture oh behind the scenes it is cu it's passing it to uh the Constructor sample profile no it's not interesting okay uh let's see what happens seven passed three failed now we're getting to the updates so update and save um this is the code that I haven't ported yet so let's uh let's Port these tests and then go and have a look at what's required in the main code oh eight minutes left uh let's see how much update filled um we've got quite a few people on the stream this week so I I really appreciate you hanging around while I've been kind of like um yeah essentially doing sort of low levish housekeeping stuff but I hope some of the problems uh that I've been coming across coming up against have been interesting um it's uh you know it's it's non-trivial to Port blocking Cod to acing Kyo and um I'm kind of hoping by this by the next stream so I'm not streaming for the next two weeks because I'm on holiday next week um uh but I'm hoping when I come back we'll be able to start building a relatively complex fast API application on top of this library and we'll be able to implement some more patterns um I'm going to implement this Twitter clone and so there's a bunch of interesting scaling problems with something like a you know a Twitter type application um and so we'll cover some of those patterns and um hopefully extend this library in some other directions um sort of while also building a nice nice practical application to sort of see how you how you should architect a mongodb application um sort of aiming at production so uh test update field is oh yes here we go um so an is actually just saying um that there was an advanced uh data modeling webinar uh where the speaker mentioned that longer array should be split into multiple documents so yeah it's it's advice we give out quite a lot um I think I may have actually been doing moderation on that webinar so I was I think it was given by uh my colleague Michael Lynn and so I was behind the scenes just sort of um taking the questions and uh organizing them so so Michael could uh Michael could answer the the questions um yeah and 100 to 200 uh um with yeah with the main one having 100 to 200 I I um those are the numbers I kind of said we we recommend a maximum of 200 items um in an array uh oh okay so see there's a little bit of a misunderstanding here I so I'm I'm I don't quite have time to bring up the atlas interface to show you the documents but actually that isn't what I've modeled so what I've modeled in dock bridge for the tests um I actually have 20 in the main document and then the rest it has 20 in the next document and 20 in the document after that and 20 in the document after that each one of those documents is in a bucket um and it means that the it keeps the index sizes small so when you've got small items of data that will generally be retrieved together like you can just store them in in these sort of 200 item arrays um and then you can just get the whole document in one go makes updating a little bit trickier because youve got to find the right document to update to add an extra item in for example um but yeah it means that you don't get index explosions um so you instead of getting one index entry for each follower you you've got one index entry for each bucket of followers um so yeah they're being split is that bucketing is the reason that the the test query is a little bit more complex um than it could be so the query is these three stages yeah that's although of course you already noticed that but yeah there split into buckets of 20 items each um okay glad glad I could clear that up actually that was obviously something that I hadn't explained properly um so we got test update field we've got roll back session which is especially important here because this is an update um and getting my motor and I have to oh I don't have to so th this is one of the things that I was saying before a weight is uh it exposes when you're doing IO so you'll notice I'm I'm not actually going to implement this now but I do want to sort of just do a quick skip through the code um now that this is becoming a bit more apparent so motor client is not awaitable so this is not contacting mongodb right because it's not doing any IO because it's not awaitable um sending this command is awaitable so that's actually sending a command to the database so something is actually going to happen um the I noticed client do admin which actually looks up a database is not async that's that's not una awaitable um so I don't have to await twice here um so that so that means when you look up a database it's also not going it's not validating that database exists until you actually try to do something with it um so yeah so the same thing here get database I do try to do this explicitly by the way usually um so I don't need to put in a weight here for motor because it's not doing anything get collection also so doesn't contact the database find one does so I will actually need to wait here um find doesn't it's only when you start looping through the cursor that comes back uh that you need to do in a wait so um yeah it's uh it's just interesting that you this stuff this stuff is hidden with Pongo you don't know when it's actually going to contact the database but with um with motor it it's it's really visible when you're actually talking to the database which I quite like um and karic has asked a question is mongodb or nosql the same thing um it's a complicated answer to that question so when people talk about nosql they are usually talking about mongodb um it is the most popular non-relational database or general per I don't know it starts to get when you start to add extra criteria and it starts to get a little bit marketing um but uh nosql is really a term to describe um all of the non-relational databases so things that that that basically don't store stuff in tables um so I would use nosql to describe things like um reddis elastic mongodb so yeah it's just but it's it doesn't really mean anything um it's a uh you know it's it's too General a term it just it's it it was really a brand name for for companies that were thinking differently um back kind of 15 years ago when mongodb sort of first started out companies were experimenting with different ways to store their data and nosql so became the the term to sort of describe that excitement more than anything else we don't tend to use it very often um and yeah so in many ways mongodb is no SQL but also it kind of isn't uh and yes I trust you all implicitly uh oh we getting more we're more yeah yes yes I'm saying yes right okay uh we have run out of time I've got one minute left um thank you very much for joining me this week uh you know it's really nice to just sort of get through some of these uh tasks feel a little bit like housekeeping tasks and then I think they when I get back to the live stream uh in three weeks time uh I'm really excited about some of the stuff we're going to build and some of the stuff we're going to talk about in terms of how you build these applications because I think it's it's very easy to get started with mongodb but actually relatively complicated to get started kind of the right way so um yeah if you're interested in keeping up with our streams um like do uh sorry I'm just trying to get myself up on the screen and then actually be able to see myself on my main monitor um so uh yeah do um hit subscribe or uh keep an eye on our YouTube um live tab because we've got a bunch of these live streams every week and my colleagues are just fantastic explaining this stuff answering questions we do a bunch of different streams and different topics um and yeah I hope to see you all in a few weeks and uh yeah thanks thanks very much for for being here I'll I'll see you all soon

Channel: MongoDB

Views: 642

Keywords: LS, Data Architecture, MongoDB, Document model, Python, FastAPI, Data Modeling, Research, Developer, Intermediate, DA

Id: 0_mMQrP9RCM

Length: 60min 15sec (3615 seconds)

Published: Thu Apr 04 2024