How to Use Poetry in Python to avoid Dependency Hell

[Music] python is known for being a great programming language right great for beginners great for experts alike there's a package to to accomplish everything that you need so it can make you feel really powerful and useful but if you've used python for any amount of time you've likely ran into issues with dependencies with packages conflicting with virtual environments and requirements.txt and just trying to get the whole thing to work and there's some weird see lib that this thing depends on and you can't get it working and you just want to pull your hair out I don't know maybe that's just me but why is that right why is dependencies such a challenge in Python and like there are compiled languages where this seems to be less of a problem right you don't see these complaints with Java which is you know compiled but jitted you don't see these complaints with with roster go so is it something to do with the the dynamic nature of python being interpreted but there's Dynamic interpreted languages that have less problems right uh node has packages uh Ruby has packages and there seems to be less issues so what's going on here and how do we address it so today I want to show an answer to this to this problem of dependencies which is poetry the python tool which I think is awesome there's a lot of ways to solve this problem in Python but I feel like they're all to solve part of the problem or need to be combined and poetry tries to to solve the whole thing I'm going to show you how to use it it's pretty pretty quick to be honest and some of the other benefits to using poetry and then I'll talk about why python has this issue and how poetry can help how in the extreme case containers can help isolation and uh just how and why I think we need this tool poetry python so what is a virtual environment and why do you need it that's the big question right it's not strictly a python thing like you can have this problem in Java uh or or many languages like in Java you can have a diamond dependency conflict right so say i' I've one program and brings in two dependencies and both these dependencies depend on some third party Library XML paring library right but they use different versions of it and those different versions say have some sort of incompatibility so I want to use these two things but they need to pull in these two things and it's not possible to have these both called their own version of that library right so it becomes a diamond so this happens in Java it can happen in a lot of places right in Python it's worse in Python it's worse and and the reason why is in Java that has to be in the same program right but in Python by default packages dependencies are Global right so it doesn't just have to be one program it could be two completely separate programs using two versions of a package that's not going to work right the easiest way for me to show you that is is like this let's look at my Cod here I am in my little sample directory where I'm going to show you some things if I run pip list then it lists all the python uh packages that I have installed right and I've had this computer for like a week and there's quite a list of them and they're all one specific version and if I have to install something that needs a different version that's going to be a little bit of a problem there could be incompatibilities so virtual envir M are a way around that solution there are different tools you can use to create and manage virtual environments in Python one of the first ones was virtual M it's it's been around for a while um but it is largely superseded by what came next which is VM VM comes built in with python 3.3 and higher and it allows you to easily create virtual environments it doesn't offer dependency um management so with VM you can start a virtual environment and you're inside of it and then inside there you use pip to install the packages that you need and then inside that environment you won't have conflicts with that Global list that I showed you earlier hopefully that's not confusing it's going to make more sense as we go trust me pi M uh specifically focused on managing the versions of python right so what if my Mac comes with python 2 but I need Python 3 for 311 for this project and 38 for this project how do I keep track all those python interpreter versions Pym helps with that and then came pipm pipm is a little bit hard to say but it's a combination of pip and the virtual environment and Pym so that you can have virtual environments that point to specific python version and you know have certain packages in them and then things keep going from there right all of these Solutions only help with python packages um which is great but what if you have some sort of C library that you depend depend on what if you're a data science person you're doing some sort of Cuda stuff on the GPU crazy C++ stuff it's not going to work to just use uh Pi M and so that's where cond came from it's a package manager that is used a lot in data science to my understanding I've never actually used it but the tool I want to talk about today is poetry poetry is a all-in-one tool for virtual environments for dependency management for even packaging and distribution so packaging up my Cod and putting it up on piie where people could use it it simplifies a lot of of tasks it it provides a nice outof thebox solution if you're if you're getting started from scratch Green Field development I think it's totally the way to go so yeah let me walk you through that okay first thing to do is you need to install poetry so for that I just use pip install poetry you got to spell it right like that you can see it has a lot of dependencies but there you go it is installed once it's installed I'm going to focus on just like a green field project so we're starting from scratch and you just use new so I'm going to do poetry new weather uh update app actually the app that I'm building doesn't actually matter because we're going to be focused mainly on the packaging but I'm building a library which you can use to get updates on the weather and when I run that right you can see over here uh ignore some my other blah projects but it created this new project weather update in it it created two packages one for tests one for weather update it created this blank and it Pi in each of them and it created this project. toml which we'll go into actually let's go into it now so it it has all these various sections that you can add things to and you can see all the Poetry ones are under tool. poetry it sets up a name a version it's gotten my author name from somewhere created a read me it's told poetry that for this project we need to use Python 3.11 and it has some other build system stuff which I don't totally understand but once we have this set up we have this nice package structure right it's easy to add dependencies to it so instead of using like a requirements.txt process and like piping that into pip we can just do this we're going to add or requests oh interesting where are we okay we're going to have to change into our folder and then we're going to add requests and we can see here that it installed requests and then there was some dependencies of requests and then if we look here right we can see it's picked a specific version of requests right so this is how I add a dependency to my to my project if I'm using poetry once I have all my dependencies in there the next thing I need to do oops is install them I go poetry install then it says installing dependencies from lock file and then it creates my lock file which we can see here right so now it's locked down the specific versions that it's using of these based on a Shaw there we go there's our requests along with all the dependencies that it depends upon all right so that's our lock file you want to make sure that when you commit this code you're going to put this uh lock file also in Source control so that whoever's using this is going to use the same version so once we add all our dependencies like that you know we would add our code probably have a main.py who knows what else a bunch of tests and add all the dependencies using the command line and then to run things we would set up a virtual environment right so in poetry we can do poetry M and there's a bunch of commands we can run there including uh info so when we wrote poetry info it tells us where our poetry environment is set up you can see it's created uh this virtual environment in this cache folder you can change this some people like it to be inside of their project in like a VM folder I'm going to leave it here and then here's our python executable and once we have that set up we you know we can go inside of it by typing poetry shell and now remember I was saying how these virtual environments let us get uh a local package context right so we don't have all those global packages we just have our local list so that if I'm in here and I run pip list all I see is the packages that have been added by poetry that's it right and same similarly right if my main version of python and Mac OS is is python 2 but here I've specified 31 then you can see it's also got that hooked up for me too right so it's providing this nice set of isolation yeah what else can we do so we may want to add tests right we already have our test folder so we would do it you know we add P test but poetry actually lets us group dependencies and you know we actually I think that's the syntax let's see what happens oh see there's a a deprecated I should say group Dev but Dev does work so now if we look at our Tomo file here uh let me see now we have Dev dependencies right so it's a separate group of dependencies just for Dev because to run this you don't actually need the python project just to build it I mean similarly uh I like to use black as a formatter we can install that and this is how we just add dependencies right we add them here we can remove them the same way so we can remove requests right and this lets us manage them uh so now requests is gone those blank lines got in there now requests is gone and this is the system right you add and remove dependencies using the Poetry command line and then your environment has those locked down and if you want to leave your virtu environment you can just exit yeah that's the basics that you need to know but poetry you know it tries to be an all-encompassing tool it has this project. toml and with this there's actually enough information that you can build packages and push them to pii so to do that you do like poetry build and that will build my project into a wheel and then I could publish that and yeah I mean I don't have the permission set up to actually push that but in theory I can push a package that way and that that's kind of one thing I like about poetry is trying to be this all-encompassing tool that covers a lot of you know just important project management things you do right it helped me set up this structure so that these packages would would work nice when I import you know when I use this package I can import like weather update dot whatever and uh it helps me package them up and it has the virtual environment and it lets me set up python it's kind of all inclusive so I like that one of the downsides of a environment like this though is they do take a space right so we can manage the virtual environments so I can do poetry and flist here's the list of all the environments I have set up right and then I can remove this if I'm moving on to another project dis space is a little bit scarce um I could do this and remove I have to copy this and now my virtual environment is gone you can see that by running list again right and then um what happens if I go into a shell then has to recreate it right but it recreates it pretty fast for this particular example and now I'm back inside which I can see with my pip list and now nothing is really set up I think I have to do o Tre stall there we go and now I've recreated my environment again and I've got everything I need in there so that's poetry that's the very basics of it it's enough to get you started I think if you're starting on a new project is a great way to go there's there's reasons why I'm a big fan of poetry right it forces you to be explicit about managing dependencies I'm not just going to import something and then you know pip install it and then forget to add it to the requirements.txt and then later when I try to set things up on another computer I'm like what's going on it says I don't have this package it forces you to set up a virtual environment ahead of time so you don't eventually run into this conflicting packages problem I would say if you're using it try to remember to always activate your environments try to remember to commit your lock file your project. tal file absolutely do that and then when you're running tests not just when you're running the code but when you're running tests make sure you're inside the package and in CI right when you run things in CI you want to also be inside this virtual environment and that way you can be sure that everything's running the same everywhere you might be wondering as I mentioned at the beginning why do we even need virtual environments for python you know is it because Python's interpreted if you've worked on other programming languages like I don't know rust go C Java um You probably haven't quite had the the same problems and that those programming languages especially rust and go are are ahead of time compiled right and this does make this problem less likely to occur if my Russ program requires a bunch of different stuff that it Imports when it compiles it it's going to statically link those dependencies same and go right besides a couple exceptions like Lipsy when I have a go program and I bring in dependencies going to statically link those into a fat binary as they call it like an executable with all the dependencies in it when it does that it can run into problems of not finding the right package but that only has happen once right at compile time whereas with python because it's interpreted it needs to find the packages at runtime so that is a contributing factor to to why python needs virtual environments but there are other interpreted run times that don't have this issue right in no j is possible to set up local packages they don't have to be Global right in Ruby it's possible to have gems local gems to your project the reason python hits this problem is because python by default does Global Packaging right and the virtual environment is a way to work around that now this isn't Python's fault right a lot of the the solutions that I just discussed you know there things that have been discovered Python's been around since 1991 no JS go rust none of those existed at the time and and python helped make developer experience of software development a lot better in a lot of ways including having ppie but some of these other things like great tooling for dealing with a package manager and having packages local to your specific solution and they came about mainly afterwards and so that's why I kind of like poetry because I think it's an it's an attempt to take this more Modern Way of keeping packages local and having more tooling that helps you set up your project that helps you build it that helps you package it more aggressive tooling that's that's more helpful poetry is just bringing that idea to python right and you may say oh okay there's a whole bunch of ways to do virtual environments in Python we're just adding one more we're further splitting things up and I mean I think that is a risk with a new tool is it's just a one more way to do something but but my hope is the the way that poetry works and helps you get started with your projects helps you manage packages helps you build I think that if we can all come together on it that it can bring python into a into a better place yeah so that's that's poetry please use it and it will help you solve a lot of your problems the problem it won't help you solve is if you have dependencies that are are C based or dependencies that are written in another programming language if you need to call say a different interpreter that that's not python then it's not going to help you manage that or if you need to bring in a dependency that is not python for instance I have a project that uses image magic and it needs a specific version of it image magic you know is a complicated piece of software as I understand it and I have another project that needs a different version of image magic and like that problem is just not possible to be solved with poetry once you get to system level dependencies or other languages dependencies it's time to look at containers as a way to package things up so I would suggest using poetry to wrap up your python dependencies but I would wrap the whole thing once you have multiple languages or dependencies that are outside of python specifically in a container build system the the one I'd recommend is Earthly I work for Earthly Earthly lets you build your software inside of a container and inside that container you know container is a lot like a virtual environment but it's at the whole level of a Linux system so the file system is is isolated and you can that way have specific versions of anything you desire that runs on Linux for that particular project so that allows you to make sure that you are reproducible once you leave the bounds of working strictly in Python if you're just in python and a lot of times you just are just use poetry it's a great tool both poetry and Earthly are open source and uh yeah little tangent like yeah containers like Docker containers are very similar to Virtual environments so yeah they let you abstract the whole Linux file system but to do that they need kernel support from the Linux kernel and it gets a little bit more complicated so it's great to have tools that can make it a little bit easier that's what I do when I need to have two different python programs that use two different versions of M and magic or they need to pull in some dependency that you know is outside of the realm of interpreted code yeah so this poetry video was inspired by somebody asking in another python video to handle poetry and that's how I actually got introduced to poetry and I'm a big fan so please leave in the comments what other types of videos you would like to see and I'll keep making them and and check out Earthly right Earthly is a pretty cool Tool uh yeah thank you for watching

Channel: Earthly

Views: 11,753

Keywords: python tutorial, virtual environment python, virtual environments python vscode, virtual environments python, python virtual environments, python virtual environment, python virtual environment explained, virtual environment, python venv, virtual environments, python virtualenv, python virtual environment mac, python poetry, poetry python, poetry python 3.11, poetry python package, poetry python mac, python packages, poetry python tutorial

Id: V5rKVrVhEh8

Length: 20min 21sec (1221 seconds)

Published: Tue Oct 17 2023