Don't Use Pip For Big Projects - Use These Instead

have you ever been in a situation where you're in a high stakes production environment and everything is riding on your code working seamlessly you've been slaving away on this project for weeks and you're ready to deploy but then you run into a nightmare scenario your dependencies are incompatible and your code won't run that's exactly what happened to me in my last job where we relied solely on pip for python package management the situation was a nightmare and it took me hours to get everything working correctly again that's why I decided to dive deep into the world of python environmentals and let me tell you it's a mess that's a story for another time though and in this video I'm going to share with you some common alternatives to pit specifically pip and poetry and conduct you'll learn which tool is best suited for your needs and how to avoid the headaches that come with dependency help so buckle up and join me on this informative and practical journey to mastering python environments before we actually get into the meat of this video I want to give a shout out to another tool that might make your life easier if you're insistent on using pip or if you can't switch tools for other reasons pip tools is a python package that comes with a really handy command pip compile this will read your requirements from a packaging file like Pi project.tamil or setup.config and compile a requirements.txt file from them with pinned versions that way you only have to update your requirements in one place When developing packages which helps to eliminate the it works on my machine problem pipen is a python environment management tool that calls itself the python Dev workflow for humans it sits on top of pip high end and virtual M to help you manage your virtual environments across projects and keep track of your current dependencies in a file called pipfile one nice benefit over just using pip and virtual end or vent is that it also produces a pipfile.lock file which contains hashes of all the packages you're using which not only keeps your builds deterministic but is also a first line of defense against supply chain attacks on Pipi also if you have Pi end available it can automatically install the required python version for you when you clone a new project which is really nice to get started install the PIP end package I recommend using pipex to keep your CLI Tools in their own environments then in your project directory just run pipnv install along with the package you want to install in this case I'm going to install quiffen which is one of my own packages pip end will then create a virtual environment for your project if you don't have one already and then create both the PIP file and pitfile.lock you can remove packages with Pip and uninstall or update packages with pipenf update to run a command in the paper environment use pip and run for example to run my main.pi file here I can use pipenver on Python main.pi and it will run as if I was in the virtual environment if you want to run commands directly from the virtual environment you can use pip and shell to open a shell within the environment itself you also have Pip and block to install packages in the specified lock file and pip-end graph to display all your pendencies and their dependencies in a nice tree finally you can run pip and check to expose any security vulnerabilities in packages in your lock file or the current environment pip end is great and super easy to set up but it doesn't actually solve the problem of dependency management that I mentioned earlier the next couple of tools do though they solve it in slightly different ways before we move on though if you're finding this video helpful why not give it a thumbs up so as to appease the algorithm gods and best subscribe too while you're at it other tools we're discussing today poetry is the one I'm most familiar with by far poetry is both a dependency and environment manager and a build system I won't be going to the build system side of poetry too much in this video but let me know in the comments if you'd like to see a deep dive into python packaging tools like poetry hatch and Maturin getting started with poetry is easy but a little bit more effort than Pippen first install poetry with pipex then from outside your project directory run poetry new followed by your project name this will scaffold out a basic project structure with some key files particularly a pi project.tomel which is used for specifying dependencies and other project metadata if you have an existing project already that you'd like to add poetry to you can do that with poetry in it to add a dependency to your project run poetry ad like pip and if this will create a virtual environment for you install a package there and then create a lock file to record details about the package however the magic with poetry is when you install more packages let's say I want to be able to export my quiffen models to yaml if I run poetry ad play yaml you'll see that it spends some time resolving dependencies and finding a compatible version of Pi ml to add to the project this can really save your ass in a lot of situations as poetry is very thorough when checking compatibility between package versions one downside to this is that it can sometimes take a long time to add a new package to your project I've seen it take over three minutes on some larger projects however you only have to do this when adding dependencies to your project or refreshing your lock file and once the lock file is in place poetry will use that for installing packages which is great when you've just cloned someone else's poetry project like pip end poetry supports the Run update lock and shell commands poetry remove is used to remove dependencies and you can view your project dependencies with poetry show if you want to share your package poetry build will create a wheel and a tarball for you but dependencies that you don't want built into this wheel like linters or testing packages you can use Pro poetry installed bass G Dev package name to install dependencies into the dev Group which is ignored by builds finally let's discuss every data scientist's best friend and worst nightmare Condor conda is a little different to pipenf in poetry in that it doesn't use Pi Pi the python package index to retrieve its packages instead it uses external repositories like the Anaconda repository to install what are called recipes unlike pip and tools like pipen for poetry which just use pip under the hood Condor packages are not just limited to python libraries this is very useful for data scientists who often need fortrans c or C plus plus binaries to go alongside their python code and for anyone who's gone through the absolute agony of setting up machine learning libraries on their computer using pip you'll know what I mean conda is also an environment manager though I personally don't find it as friendly as poetry or pip-end and it's very difficult to automate with scripts as the Condor CLI is very particular about how it's set up in your terminal and condo is a little tricky to set up the recommended way of installing condo is by via Anaconda which is a whole data science platform or through Condor Forge a community-run condor repo I have Condor installed via mini condo which is a Bare Bones version of Anaconda that only contains Conde itself you could also install condo via pip but it might not be up to date and won't be installable on newer versions of python by default condo will create and activate a python environment called base to create a new condo environment run contact create name followed by the environment name any packages you want to install and optionally a python version condo will actually install python for you so you don't have to do that yourself so I'll be really wild here and create a python 3.7 environment I can then activate the environment using conda activate filming which puts me in a shell in that environment and then search for packages using condo search finally I can install a package with conda install conda doesn't actually create any project files for you like pipen for poetry so if you want to share your project you'll need to export your environment with conda and Export this will output some yaml to stand it out which you can then write to to a file which is typically called environment.yaml future users can then do condo create Dash F environment.yaml to create an environment from a file like poetry conda has a more advanced dependency resolver so we can sometimes take some time to install packages I personally don't really use Condor but having worked as an engineer on a data team I've seen the benefits that it provides to data scientists and data analysts first hand the python packaging and environment management ecosystem is a bit of a mess but hopefully this video will help you navigate the tools brawl and find the best tool for your project overall I'd recommend using poetry for most applications as it works well for small and large projects be it a package or a full-blown microservice if you're a data scientist unless you fancy installing a Fortran compiler on your machine I'd suggest sticking to conda until something better comes along if you want to get started with a poetry project watch this video on itemple a CLI tool I built for templating all sorts of python projects

Channel: Isaac Harris-Holt

Views: 23,054

Keywords: programming, python, technology, software engineer, pip, poetry, pipenv, conda, pip-tools

Id: jVcN49sHbBQ

Length: 7min 55sec (475 seconds)

Published: Sun May 07 2023