Matthew Croughan - Use flake.nix, not Dockerfile - MCH2022

okay he is ready well thank you and a big applause for matthew so who here has heard of docker wow that's amazing i thought i was gonna have to explain what docker is but you all know what it is right great so this talk is called use flake dot nix not docker file this is i don't know if i'm preaching to the choir maybe you know what the issues are with building software with docker maybe you don't but i'm going to try and explain why it's not really that reproducible and why you can use nix instead so why use dark files well i don't know but why use flake.nex because build via dockerfiles aren't reproducible but they are repeatable so that means that you know you can run the same instructions but it does not mean that you'll get the same result so what is nix nyx is a expression language it's a domain specific language which was invented by elkodulstra in his phd thesis in 2003. um it's lazily evaluated which means it's more performant for doing package management it's a purely functional programming language which means it's not procedural so the order in which you define things does not matter it's declarative rather than imperative so things are defined as a expression rather than as a sequence of events that you want to occur much like in a doctor file and yeah in that in that way everything is an expression and there are no side effects so here's a basic abstract example this is called an attribute set in this example x is equal to a list which has a string inside of it y y is equal to one and z is equal to the expression one plus one so let's evaluate what x would have been so x is a string uh y inside of a list and the same for y is just one and z is one plus one which is two so here's a basic concrete example of me compiling hello world from gnu source code so i'm going to cut the examples hello world.next this is the nyx expression which defines how to compile the software and then i'm going to nix build it i'm going to look at the result that it creates the result is a sim link to this object store called slash nic slash door where the immutable software packages live right we can see that it's a dynamically linked x86 executable with its interpreter patched to the correct interpreter and then i run it at the end so you see hello world is there so what is the expression at the top where it says let packages import next packages etc so that is a function call to standard m mk derivation the first argument is his name which is a string so i'm going to call this package hello world and the source code is going to come from uh good news ftp server and we're going to hash this and this is really important we're going to say what the sha-256 256 sum is ahead of time now this way if anyone else sees the expression in the future and they build it and that tarball has changed but we'll know about it because the shah 256 won't match so this data is really important for reproducibility and security so by comparison let's see what a dockerfire would look like so from ubuntu latest there's two problems here right ubuntu even if we could find the original docker files for ubuntu could we reproduce it or do they have many steps inside of it that aren't reproducible like apt install this apt install that because every time you do app install you've lost the reproducibility because it's going to give you a different result the next time it runs latest latest when when i run the docker build right so the the docker files are not intrinsically attached to their to their results so the result of a docker file is basically like a binary you give that binary to people and you may never see the dockerfile ever again right and that's not quite true for next expressions so apt-get update it's going to resolve a different package list the next time it runs so the next time we're on docker build hello might just not be there so we're gonna have to run it five times and then hopefully uh debbie and servers are uh are up again rather than down and yeah but that's not that's not the only problem every time we do apps get installed hello it's going to create a package database and this package database is going to have time stamps in it there's going to be tons of reproducibility issues just at every level in this dockerfile and then last offender is cmd hello this assumes that the hello package actually puts a binary called hello on the path so maybe hello is not the name maybe it's hello dash world so there's nothing but assumptions in this dockerfile and indeed if we use this method here where we save the results of building it into a tarball and then we look at the tarball the tarball's different now this isn't the full story this is a bit of a cheat right as a matter of fact if you extract this you'll get another tarball and inside of that is a layer and a file system hierarchy like you slash user slash bin etc so um we'd have to inspect that with difficult to understand the true reason that this is not reproducible but i mean this this is a nice example because it shows that you know even when they package the tarball for the oci image this tarball this image that we're going to load into docker even that step is not necessarily reproducible because tarballs pack and unpack serialize and deserialize in a different order every time operations are performed so how do we do this with a nick flake instead so there's a function someone made called i didn't i only wrote this presentation this morning so i missed that out but it's called um docker tools build layered image right so someone in nix packages which is a uh as a library of 80 000 software packages and mixed expressions and functions and things um so they made this this function called build layered image and this is essentially a reproducible bash script for producing container images right so my container image equals running that function with the first argument name my container image the tag latest contents is a list of things that i want to be in the container and config.cmd is the entry point that we saw in the dockerfile earlier which was hello so let's just have a look here so this hello here that's what we're going to map to in the next expression config.command equals hello so this is the first thing it's going to run when the container boots so this is in a data structure called a flake so a flake has inputs and outputs we say inputs.nextpackages.url equals this github repo and whenever we run a nyx command on this it's actually going to generate a flake.lock so i'd just like to show you that before i get too far in so flake.lock so we no longer have that problem of like latest being latest whenever on the command because it creates a lock file and tells you what it was when i run the command and this is immutable and stored in the git repo where i provide these expressions or tell you to run it from so yeah let's uh let's build it so we're going to build my container image we're going to look in the result and we're going to load that result into docker and then we're going to run it with docker and this has all been done like the container image has been built with necks reproducibly and then ran with docker so i'm just going to run that oh dear ah there we go i got got scared there for a moment okay so it created a result put it in the nick store where it lives uh reproducibly and it loaded it into docker and then it ran it so there you go much more reproducible than the docker file right so why is this the case so docker is repeatable but not reproducible defined instructions but it trusts the internet unconditionally and it doesn't perform any hashing or at least doesn't encourage people to do that doc files do not provide you with a tool set for performing reproducible builds and nyx is basically a c-plus plus binary written by alcodultra which implements the nyx expression language which gives you a tool set for implementing reproducible builds so what does next guarantee so it guarantees that we're going to get the same input from the internet every time and we're going to perform the build process inside of a sandbox and hopefully the result is deterministic uh it guarantees that but it does not guarantee that the build process in fact is deterministic an example of this would be the java compiler the java compiler will put timestamps into every jar file that creates every piece of java source code now of course we're not going to get rid of the system call that allows the process to gather the time at this point in time and but that was that was a decision because it built sorry it um it broke a lot of build processes so instead we have to do things like touch 1970 at every build step so that we get rid of the timestamp and the intermediate steps when we compile things um so yeah there's lots of reasons that a build may not be deterministic and nyx can't help with that what it can do is perform the same build with the same input every single time and hope that the output's deterministic which is a lot more than many other tools can do so do i have any extra time do i have yes yes we have enough time for questions oh yeah i didn't want to go with the question straight away i just wanted to show uh three really cool things so nick's shell is a way of trying things before you buy so i'm going to ssh into my server because the internet here might just drop out um so i'm going to next build oh sorry next shell next package is hashtag i don't know python3 and the reason i'm going to do that is because i have python 3 now but if i get out of the python 3 interpreter and try and run python 3 again after exiting out of this temporary shell i don't have python 3. i don't have python on any of my systems installed globally right it's just i install it when i need it and only when it's necessary now because it's an expression language if i go to the repl i can say something like let's load next packages and then python 3 has a member function inside of it with packages and i can say inside of p put all of the whatever it is 20 000 python libraries that exist in the world from pi pi and i'm going to give it a list of python libraries that i want for this particular instance of python p dot i don't know numpy it's going to evaluate that and create a derivation and then i'm going to build that derivation and it's going to do all of this reproducibly so if you you know evaluated this expression you get the same path you get the same output path on your machine and then i can ls in there and see that we've got bin and we've got python interpreter and if we run that we'll have a python three that can import numpy oh but if i exit this and exit again you know i don't have python anymore i've gone to a next wrapper and say i want python three whoops got a load of next packages i'm going to build python 3 note that i'm not defining a function i just want python 3. i don't want python 3 with numpy let's take a look in the output this python cannot import numpy because there is no such thing it's not global there is no global state there's no global thing that's being modified everything is copy every single time with something guided or without something added another thing that you can do is you can cross compile using nex so in next packages there is a package set called packages cross which is all of the 80 000 nix packages that you can get um whoops yeah so like i don't know python's there we've got um numpy and we've got um firefox and we've got you know a set of 80 000 packages which we can do what we want with so what do we want to cross compile all those 8000 packages well packages cross has in it if we tab complete all of the architectures we want so let's say risk 5 64 bit and then i don't know hello world now i think i've done this and this is another benefit of reproducible build as that things are cached i did this last month they put something in my slash nick slash door it's still there it's not going away i haven't garbage collected it yet and that's another concept that nick has his garbage collection the file system is memory is what they call it in the thesis so um i i did this last month so it's still there so that's why i completed but if i want to rebuild it just to show off it can do this and this is reproducibly cross compiling hello world for risk 5 64 bit and it's done and in slash bin we have a dynamically linked wrist 5 binary and the fun part is despite the fact that i'm actually running on an x86 kernel i can still i can still actually execute this because i have something called bin fmt registrations enabled on my system and the way that that's done is also very easy um thanks to nyx because nixos of course has many options i can just say okay i've been fmt registrations yeah i want to be able to run um windows binaries or arm binaries i can do that on my x86 system just by wanting a list of emulated systems in this option so that that should um convince you to use nexus mix os if nothing else does um yeah i think i just showed off those three points so any questions if i have time if not and that's my talk yeah if there are any questions the mics are still in the middle uh if you walk the mic otherwise the video doesn't have audio so we all can hear you but the people re-watching can't well in the docker file you have a little bit closer to the mic please yeah thank you docker files you have latest and that's great you don't have to upgrade it in half a year is there something like nyx maybe specify well latest up to this timestamp uh yeah so i mean docker could if it wanted to probably like implement a lock file um although i i haven't seen that so so the benefit of nyx here right is that you can't do such a thing as latest it doesn't exist and that's a good thing for reproducibility so we take a look at this docker file right it says latest now there's no there's no when you run the command it's going to go and fetch the latest and everyone that runs the docker bill command every single person on the planet will get a very different result right there is no guarantee so i have to target i have to say like 2004 right but that 2004 could still change technically if i wanted to be really reproducible and docker does do this you can specify a hash right and so so you know there is a way around that but it's the it's the least of docker's issues right so this is just one well the question would be is there a next upgrade yes indeed so let's say flake.lock flake.next here has um nexus 202205. this is a tag right so it changes so if i want to see what that is i go into the flake.lock and we see that indeed it's the e3583 revision of nix packages oops and if i go to next packages on github and i go to that a hush what i'm saying it's a lot of steps to keep your system up to date no it's not i'm just showing you the full path as to how it's reproducible so we go to that commit we'll see that that commit exists and this commit was made a few days ago you know four days ago so in four days you know a lot might have happened so we want to go on and we want to advance from this hash this revision here so all we have to do is next click update and it will go and find what the latest hash is looks like d1ca for ea right and so this is the step that you do to update the lock file um and once that's done right now it's extra it actually went and fetched the latest mixed packages and it's extracting the tarball on my laptop so it's a bit slow because my laptop disc is a bit slow but um but that lock file will be uh updated that was great yeah okay i think we need to be rounding up the questions do we have time for one more or should we now we've got one more speaker so if you the other questions i think they can approach you yeah afterwards because we really gotta move it along sorry yeah thank you so much question but in one sentence if you go to reapology.org you will find of all the distros on the planet all their versions and you'll find that nyx packages has the most recent of any distro beats the pants of anything top right next package is unstable it's the most fresh it's the largest size so it's got more packages than arch they're fresher than orange they're more up to date than arch they're more maintained than a lot of other distributions too thank you

Channel: Matthew Croughan

Views: 21,807

Keywords:

Id: 0uixRE8xlbY

Length: 17min 24sec (1044 seconds)

Published: Tue Jul 26 2022

Reddit Comments

[u/apfelkuchen06]

nice hat.

[u/ElCondorHerido]

I'm no expert at neither (nix nor docker), but I'm about to embark in a home-server project and was very confused about what to use. I already use nixos in my desktop and laptop (mainly to make them exact replicas), but had no idea if it was a good idea for a home-server project.

This will be of great help to make that decission.

[u/bew78]

That was intense! o/