Ansible Contributors Summit 2023.02 - Galactory

all right um yeah we'll get started so uh this is about a project I created called galactic um this is an ansible Galaxy server this is for storing your collections and all of that um uh who am I I'm Brian I go by Brian test on GitHub and IRC and most online things I'm in the ansible steering committee retainer of community Hesh evolved the docs GitHub actions which uh Felix and I will be talking about after lunch um and a bunch of other stuff so first I'll talk about artifactory and what what this is um artifactory is this product that basically has a bunch of different repository types to store all your stuff there's this concept of a local repositories which are your your things is remote repositories which sort of Shadow a remote and there's virtual repositories which are like a single endpoint that will combine both of your local and remote artifacts uh it stores many types um this and a whole bunch of other stuff and not Galaxy there was a request for it in 2016 before collections that request is closed there's a new request for 2019 it's gone nowhere so um so talking about Galaxy for moments Galaxy means roughly three things in the context of this the ansible Galaxy command the protocol of Galaxy which is the communication between the client and the server and Galaxy servers themselves so of course as we all know Galaxy the public one Galaxy interval.com um and then there's a possibility to have a private Galaxy servers as well so we want a private server probably for our own stuff our internal collections company stuff that kind of thing um and so there's galaxy Ng and this is I believe slated to replace the public Galaxy software um has lots of contributors people pay to actually maintain it um it is full featured it's got a whole UI and search and authentication and all of that stuff um and go after it has none of that uh so so why not why not Galaxy NG uh so there's pulp which is the actual software behind Galaxy NG is stored as artifacts with different types content plugins for extending the types and Galaxy NG is a plugin and it sounds a lot like artifactory but I already have more detectory um and that's really why I built this so to be clear I'm not saying don't use NG I'm actually saying you should use ngd if you can um but if you've already got an entire artifact storage system I could see why you might not want to run another one for one repository type um so that's what that's kind of what this project is um it's a very small thing it influence Galaxy V2 including publishing um and so the two main features are to store and retrieve Collections and then to also transparently proxy to an upstream Galaxy server uh this is meant to give something similar to artifactory as virtual repositories so you can kind of have one in Hawaii get your your internal stuff and the public stuff so the basics um about the factory it's a flask-based web application uh it's only for collections it's already role enthusiasts um and it isn't really simple it's there's no UI there's no way to manage collections in it itself there's no built-in Authentication or authorization uh but we can't pass through that to artifactory we'll talk about that in a bit there's no native TLS support right now um that will probably change but for now I'm basically figuring um if you're not running this locally you'll probably put it behind a reverse proxy or a balancer or something like that so uh direct production storage in artifactory this uses a a repository type called generic it's just stores files and metadata um and so for this we really just have to put the tarble in in artifactory at some properties that um electrically use to uh to serve API requests it also supports the publish command so we can do publishes with the Expo Galaxy collection publish and uh that will make it into Art Factory Authentication there's a bunch of options in your library for this so as I said all all authentication is handled by artifactory so if you want an anonymous Galaxy you can make your repository an artifactory allow Anonymous reads you can configure the lottery with with a API key that will allow to do rights that all I want to also do caches um one other neat thing is like ansible the ansible Galaxy command supports a token um but the antelope Galaxy command doesn't know anything about the token it really just puts it in the request so uh we can exploit this by putting an artifactory API key as your Galaxy token that will get simple on your request galactory can use that to authenticate to artifactory so you can have sort of like every user Authentication so the then there's the Upstream proxy feature this will merge remote and local API results and that way you get a single single point of view the API results are also cached in artifactory this does retries to to upstreams in case there's you know some kind of connection problem or something like that uh the first time a collection is retrieved from your from the Upstream Galaxy it gets put into artifactory so that it then basically becomes local and that means it will be served directly from there the next time you can also exclude namespaces from Upstream requests so if you're processing to a public Galaxy and you've got internal collections even let's say your company's namespace you can prevent those requests from ever hitting down the street um this is really helpful for avoiding uh Galaxy outages and and Fortune 9 throughout the others I know there's a lot of Galaxy outages and stuff I think that's going to be much improved soon um but the throttling errors are very real I had a I was doing some CI for an internal collection one collection um but the test Matrix probably you know I think it's been six ansible versions and two python versions and uh that alone just died I got throttled immediately from Galaxy um just trying to install the uh dependencies so um some ways to run it it's a it's a python package you can fit install you can run the module there's a container as well just to note that the container that is on the repository is using the internal Flex web server which you're not supposed to use in production um the flask uh you're really meant to use it with a custom you know a production wsgi server I don't have any examples of that but it's it's pretty easy to do um how to configure it there's a whole bunch of command line options you can give it every one of these options has an Associated environment variable so you can you can set things that way as well um you can also put these same options in configuration files in a couple of locations and some scenarios of how you might run it so running it on the ansible controller this is actually how I ran it for a long time this could be on your on your workstation this could be in CI pipelines which is what I used to hold up um if you're not publishing anything in your repository allows Anonymous access you don't even need any secrets and so you had no service to run and maintain with this kind of setup or you know you could choose to run it somewhere centrally like in your company you have to see the URL for everyone um you can use a setup where you have credentials in the service but those credentials will not allow a client to publish and so the client can still provide their own Authentication um and running at Central evenings you can kind of have like an ansible config file that just points to that you don't have to do like dash s on every command or set environment variables to tell intimate Galaxy you where to look um so I want to acknowledge um uh civil who wrote The Amanda um which was a a very simple kind of Galaxy server that just read from the files in our directory um that's kind of what first inspired me to write this and uh JC Tanner has a project called Galaxy mirror which is a proxy for going to Galaxy and then it will kind of cache all those on your local file system so that's another project that might be of interest Okay so I have a very very short demo that hopefully will work as David said yesterday I kind of like pre-baked this you know kind of uh because artifact already takes like three to five minutes to start even with nothing in it um and so I did already start that VM at that container so this is uh artifactory I have this repository here called ansible collections there's two collections in there now um that I put in here already uh I will start galactory which doesn't take any time at all um and we can see collector is running this is a there's a little health check endpoint in there in case you need that you can set custom text in there um so I'm going to take a look at this collections uh endpoint here in uh Galactic so this is actually returning only those two portions that um that we have in there and if we looked Instead at like we asked galactory for all the versions of Community General it will give us that too because it that is proxied to the stream Galaxy and because I hit this endpoint um I can come back and refresh the repository here and see that there's a cache this is really just for API responses so um and this is because uh the Galaxy protocol kind of has to make up several calls when you go to install a collection figure out the versions and all that and so this was again to kind of avoid outages but mostly throttling issues and the cash is very configurable so there's a bunch of options for that [Music] um right just so let's try to do some commands do that okay yeah so that's you know that's it that'll that'll install something um we can also do so let's let's do an upstream one right we'll do like Community General s not not very exciting but um going back to artifact area you can see it is now in here until the next time it is installed it would be pulled directly from here but you know from the command line that is really transparent uh well I also want to show the properties which are the metadata um election info is the one that Galactic actually uses and the other ones are kind of in here as a convenience that's to help you be able to manage your collections within artifactory do searches or whatever um one more I think I can do a publish um right so this is the one I just published and yeah uh so real quick I will show the uh I'll just show that the docker composed of what I uh launched this with uh artifactory one is whatever uh the galactory one just showing like there's an environment variable we can we can set some option with that the uh these are command line options custom health check text I use this end file to put the um the super secret API key that I'm going to show you all now but that's how I was able to publish to artifactory without passing energy so yeah that is that is it any questions um so the question was will there be any support for execution environments um to be publishing execution uh no this is I mean this is just for Galaxy protocol so just collections for now maybe it could be roles at some point too yeah it's mostly a silly question is there a role to deploy galactory [Music] um there there's a module that's called it's called pip and it's all built-in pit okay security do you think the source openly did once Galaxy NGS of each complete with Community coaching uh that's a great question so the the question is um will this still be needed when Galaxy NG is feature complete um yes I think so but there's no there's no technical deficiency in NG or even current Galaxy um this is needed for right it's really all kind of organizational right like I we have a whole team who manages a bunch of tools one of which is artifactory right all of our effects are there everything is there um they're not going to run another one just for ansible collections um my team does not want to run a whole thing just for answerable questions either this is basically a middle ground of like here's a very small thing and then we can put our you know what amounts to a couple megabytes or 100 megabytes of stuff in with all the other artifacts um because you know that has internal support backups page review rotations you know that kind of thing yeah that's pretty good yeah that's not really a question for you probably but um is it maybe possible to change Galaxy ND so I can do something else involved and use that one with artifact group that would give you the same UI but um so the question is could Galaxy and GED change to support something other than pulp um it's true I can't directly answer that but what I can say is that artifactory does not have like a good plug-in system for this so they're already not even extensible I looked at that to see if maybe that would be a viable option um so probably not I mean not for artifactory anyway okay anything else wrap up [Applause]

Channel: OpenEvents (Conference Livestreams and Recordings)

Views: 46

Keywords: conference, lecture

Id: KMRn2vqCSVE

Length: 20min 46sec (1246 seconds)

Published: Sun Feb 19 2023