How to use Traefik (Part 5) - How HTTP challenge works with a real web server

hi welcome back to the traffic video series and now I've lost count what number it is okay so this is HTTP challenge and I'm going to demonstrate using a real web server how it works I have a web server in your ws which has certain IP address and which is providing virtual hosting to multitool dot lab tour prefer calm and nginx dot W bit calm these two hosts and I'm going to use HTTP challenge so let's do that and by the way just so you know so the DNS for each of these two completely different domains points to this IP address here which is let's check dig in the next dot W bit comm 3.1 21 hello okay and let's try to put it here so this is the IP address of the webserver fine and and the next or W bit calm points to discipie address and multi-tool dot lab dot track marcom also points to the same IP address right here and I have my server here and I have a stack running but I'm going to shut that down for you Doka temple a stop and dr. Campos remove - yeah so I can show you from beginning what this example looks like and ideally this is actually example for in in this directory at docker compose our camera yes I'll just show you directly on the server here and this is the web server I'll clear the screen I'll do docker PS nothing is there I will add it to talk a compose file for you I have traffic reverse proxy running and I am mounting a special file this is very important very very very important this is a special file which traffic looks for not all let's encrypt line look for this file but traffic looks for this and when it starts for the first time it goes and checks gets the certificate and then it needs to put it somewhere now you can eliminate this line and it will still get and fetch the certificate and we'll put it in the container in this file but it will not be persistent the moment container dies or restarts Nora stop this file goes away because it was inside the container so we have to have it outside the container and that's the only way to mount it inside so when the process traffic process we'll go and fetch the certificate if we put it in the file inside which is mapped outside and it will be stored on the filesystem this has a role this has special role for staging let's say you go and fetch a staging certificate we'll get it put it here and you start the service and let's say your service is experiencing problems every time it it's restarting every minute or usually started every few minutes and trying to write it desperately trying to fix things and every time if this this line is missing every time it goes fetches the certificate stores it in the container and then you kill it and then you restart and the same process happens again until you figure it out so you see no problem here not much of a problem until you hit staging servers rate limit as young you had production certificate here and you still did not have this volume mount so you will your computer I mean your process traffic process the let's encrypt client will go talk to the production certificate server get the certificate get it in the and store it in the container which is not mounted mounted outside so it is in the container it is running or working and if for some reason if it dies within a minute or 2 minutes or 5 minutes or you kill it yourself and restart it then the traffic profits comes up tries to see do I have a certificate it sees no certificate I'm a clean state I'm in a clean state so it goes talks to the server gets another server certificate it stores it fine but this can happen only five times and then you are locked out for one hour so you have a problem but if you have this line here in the or if you have this file mounted somehow from outside in the container or the other way around either way you have some way of persisting it then traffic will start it will find this file then it will first see if this file contains a certificate which is valid which needs which needs annual or not and if it doesn't need anyone it considers it valid in terms of times time then it just skips the request and fetching part from the let's encrypt so it just brings it up so if you have this persistent storage and you said it is being stored in this file on this story then your services can restart 100 times 1,000 times it's ok because then every time they restart they will find the certificate and they will only need to contact the let's encrypt server when the certificate needs renewal or when you revoke want to revoke it so this is a very important line and that's why I spend a few minutes on this one so this is one new entry in this file we have a multi to limit which is mounting pragma network multi-tool and it has this front-end rule multi-tool torque load bail lab tort rockman torque on and we have another engine X which is for engine ester WBGO okay two different domains two different horses and we are going to use HTTP challenge for that so let's modify traffic to more file and we have the standard excuse me standard HTTP and HTTPS entry points here the same admin username password and in Acme we have so this time we have acne section which you have not seen before this is new all of this this was not in the previous examples because until previous example we use self-signed certificates not acne so you need to have an acne entry you need to have entry points set to HTTPS and I would strongly advise to use acne logging set to true so you know if your if your acne processes encountering any problem and then you use an email address and then you specify storage where the traffic process we'll find the file or will store the certificate in same thing and then which state which certificate server to use so in the beginning I would always use staging to make sure that it works okay because I don't want to be locked out and this is something which I will show you in a moment but you need to have Acme HTTP challenge enable and its entry point should be set to HTTP and the reason is let's let me show you the diagram when you start the server and your server is it's not HTTP configured then where are you listening definitely on just port 80 right so you can't you can't ask the let's encrypt certificate the set of your server that please you can fetch your file from HTTP colon slash whatever because the server would ask it asked the less inclined to place a file from at a special location and will try to fetch it over HTTP port 80 right and that is what was wrong in the internet should encrypt how it works page it says it listens on HTTPS which is wrong the server contacts the web server for the first time on HTTP because there is no HTTPS anyway so anyhow let's go back to terminal so you have HTTP challenge set and then you specify which host you want certificate for right so right now I can modify it and make it laugh track ma and this is a domain so I need to say multi-tool dot lab dog track my comm so HTTP challenge can only get the one certificate per host I mean per fqdn so this is how it will look like let's start and let me show you how it was so doc docker PS docker compose up minus D and we do log follow-up mode very good so this is a follow up mode on traffic let's encrypt traffic this container it is starting listening on 443 listening one 8080 just neon 80 docker provider Acme provider and it's testing certificate renew and it's not doing anything else it has stopped because the certificate was there already and that's not what I wanted to show you sorry control see what's happening is let me show you what's happening is it has found the certificate in a file because of my previous attempt and if I do a cat on Acme dot JSON it's very simple it's it's just a file which contains there are some some certificates and the most important thing to look for is whether it's valid this is valid in terms of time not weighted in terms of browser certificate validation and if it is getting the certificate from staging server or production server and then what domain or fqdn tis getting the certificate for so let's skip this all of this control l for screen clear docker compose stop and Locker compose remove - F and I'm going to remove this Ahmed or JSON file remove Ahmed or JSON list it doesn't exist anymore this is a production certificate I just kept it for no reason I can delete it of course till the time I'm not abusing let's encrypt I can request a new one so that's alright so let me delete that as well to remove any confusion list - elf I have dr. compose I have traffic tortimer I don't have Acme dot JSON I need to have it on docker systems I need to have it so at least exist make it exist so I would do touch act mint or JSON just - L it's there but its permissions are to lose they need to be like this so chain more zero six zero zero act mint or JSON list - L and I see my file zero bytes in size okay I can start my container now my stack come on up mine's D and I'm going to monitor the logs what's going on let's see what's going on very good something is happening I started this version several servers provider Acme provider and you see it's testing certificate renew and it finds that it is completely empty okay so what it does it is it goes and registers it and here's the Acme registration process in front of you it just is my account it says I need a certificate for this fqdn and then it Acme protocol uses HTTP 0-1 resolver and then it finds whatever the internals the files are not shown here that which file is placed where it just say a server validated our request and validation is successful and certificates are being validation successful then it requests certificates and then certificate responds with a certificate and then the service is restarted this is a restart message because it's start listening on the pours again now if I reach try to reach this file from outside I should get HTTP right but before that I'm going to show you what the file looks like normally you'll find the file about 1313 kilobytes in size if I do cat ahmed or json it's a it's a file here it has my account ID status is valid which is valid in terms of time I am repeating again and again that means it does not expired 90 days I have not passed yet it is getting staging certificate this is the private key this is the host name for which the certificate is issued this is a certificate for this host name and this is the actual certificate content and here is the private key for the certificate and that's it and then let's encrypt handles that internally it's the service itself which is you consuming the certificates or three starts itself let's try to browse this I'm going to type in multi-tool dot lab dot track marcom and it's not succeeded why well a multi-tool no mati tool multi-tool yes here it automatically detected our redirect and there is a certificate I want to see the certificate here it says invalid but that's okay what it should show here two most important things is that it should show that it is issued to this name and it is issued by fake let's encrypt intermediate x1 and the third thing which might be important is at the time when it was issued that was few minutes ago so that's the certificate we are talking about and if you want to be super sure then you just go on and check these fingerprints but anyhow all right so this certificate which we see that it it is coming from let's encrypt and it's a staging certificate which is good enough for me for now I don't want to tell you my quota for production cell production certificates so I will for now I will just continue with staging staging certificates if I proceed I see my network multi-tools default.html phase which just says wrap my network multi-tool this is the container ID and this is the IP address it got from the docker all right very good so this works remember I had another service running here in the same docker compose there's nginx running which is listening on this host I mean this fqdn and if I do docker PS I have I see that it's running so traffic is running brought my network orgy tool is running and the Nexus running also let's see what happens when I accessed that one so it goes on and it tries to get some certificate and notice what's the problem here this is a great example actually if you notice it is a common it's set to traffic default certificate here and it is issued by a traffic default cert why is that it is because we have enabled in our docker compose we have three services we have traffic we have multi tool we have nginx we have ask traffic to get certificate for multi tool and it got it and multi tool is working on with the certificate issued from let's encrypt but we are also reader we are also serving and the next or WP comm through the redirected HTTP method and when the traffic comes here it doesn't have HTTP certificate at least we did not request it from let's encrypt right so this needs to serve needs to be served but what will traffic do it doesn't have certificate for this fqdn from lesson cream so it creates its own its create it creates its own excuse me it creates its own default certificate and issues issues assigns it to this acute again and that is what you see it will still go on and will still let you browse or visit this website whatever it is but the certificate is incorrect how to fix that so it's very good that it showed up and we fix it we at least understood it so let's fix this one let's first of all bring down our stack very good our stack is down and is cleaned up I will clear the screen list - L I will remove Acme JSON file because I'm going to get a new one since its staging I can take I have some lenience leniency available VI is there anything in docker compose I need to do nothing we had traffic dr. will file I go down to the Acme section and here what I'll do is I'll simply disable this Acme tor domains part which is the requesting certificate for just one fqdn and I'm going to say on whose role is true what this will do is and my CA certificate is still staging what this will do is I don't need to specify any hosts here so imagine a situation where you are virtual hosting 20 different websites for 20 different friends or to have 20 different clients you can't possibly keep modifying traffic door to Mel file every time some hostess up brought up or something some host is removed from your configuration so what a solution we have here is we just enable this on module is equal to true just set it once and exit and that's it what almost rule equals true does is it checks the docker compose it checks the label this label on every container traffic top front end dot rule and host row and if it finds it it automatically uses this to contact let's encrypt and say please give me a certificate for this one this fqdn and it gives it to it after of course going through the HTTP challenge this is only for HDTV challenge and then it requests for the next we host or the next container it finds and it asks let's encrypt please can I have another certificate for this one and say sure go through this challenge this HTTP challenge and here you go so you don't need to actually maintain the list in traffic go to one let's see it in action so we have docker compose PS nothing running very good screen clear docker compose up - d and did I remove my file sorry let me just stop before I mess up I mean I can't mess up but I don't want to confuse your mind so just assume it's still stopped nothing happened and we are going to clean up properly by removing Ecuador JSON as well yeah so there's nothing here and I'm seeing this is an eyesore for me so let me fix that well that's completely unrelated okay that this was just an eyesore for me okay so what I've done is I've removed the Acme nor JSON file completely I'm going to touch back mentor JSON if I don't do that docker will create a directory for this one and then it will be messy situation touch Ahmed on JSON and change more zero six zero zero two acne dot JSON file and this one's ll I see it's a zero byte file now I can bring up my stack once it's up I checked docker PS I see three services I'm happy I do LS minus L we something here which is three kilobytes but I know that certificate will take time it will take some time one to two minutes so I'll just more monitor the logs here it started here this is the Acme provider which started it tested certificates in Newell it saw that the key is empty so it has to register so first of all it goes and register nginx or W be calm and then multi-tool dot left or drachma calm and then we go through the HTTP zero one challenges and then the server validated our requires validation success succeeded requesting certificates server vegetated our request server responded with a certificate server responded what with a certificate so we are getting server responded with a certificate for this one and for also for this one both containers both vhosts so to speak although they are not we host but I'm just using the term here there are two different containers but there are two different of QD ends and traffic was successful successful in getting certificates for both of them of them from let's encrypt but let's see what what it looks like right so if I do LS minus L now so by this time and when I was talking all of this has happened and the filed has grew up in size let's see what is what's there so I see whether certificate in terms of time I see a staging certificate I see not selling certificate I see that staging search server was accessed I see some private key I see a certificate section issued to multi-tool and then the actual certificate with its key and then I see another section and then a certificate for it and this is nginx or W be calm and a key for it and that's it so I have two certificates now from that sin trip and I should be able to visit my websites properly still getting browser invalid certifications browser certifications in validations sorry that's opened another incognito browser let's go to multi - let's not go too muddy to all this time press enter.we is going to multi tool w vidcom sorry lab dot drama calm I can do HTTP and see what happens it gets redirected automatically and all I'm just in now is certificate so I see a certificate was issued to this one from fake let's encrypt very good I can now proceed which is there and now I can check in the next not WP calm and I'm being redirected and I see a certificate error I checked the certificate and it's issued to an extra WP calm by the let's encrypt certification server very good so my HTTP challenge works I've shown you example in which I was able to obtain the certificate for once one fqdn or even two or even more if you have so now it's very flexible and automated in a way the last thing which you might want to see is how does a real certificate look like come on so I'll show you that and to avoid any possible problem with production server I'm going to just limit first of all I'm going to limit to one fqdn because I also have to show you the main DNS challenge right so I do docker compose stop docker compose remove - f and by the way when you are done testing your certification mechanisms using staging server you are supposed to remove the staging certificate from your persistent storage so I'll clear the screen this - L I'm supposed to remove Acme torjussen then I want to use but I want to change to our switch to the production let's encrypt certificate server and I'm supposed to recreate this one as empty file it's empty and now I'm going to modify traffic determine go to the Acme section and this time I'm going to remove this and going to request just one server certificate for multi tool dot lab dr. Otteman calm and I want to get it from production so I'm going to comment the staging server URL and that's how it will get the different production one okay I think that let's encrypt or all these clients should enable staging by default and production should be on choice the default configuration you get staging services commented out that means you get production certificate and if there's a problem then users get blocked out so I don't think this is a good idea it should be other way round anyhow let's come back to what I was trying to do I'm trying to disable staging so I get production certificate and I'm just going to get one certificate for multi-tool door left or truck Montcalm that's it we're going to save and exit and my stack is down already which I verify so let's start our compose up - D and let's do a list - el Ahmed or JSON is very small in size I'm going to do the logs check the logs what's going on so here's the Acme provider and then it checks certificated see it's empty it goes for registration registering my account and it's using production server production URL doing SCTP challenge server vegetated our request the vegetation successful requesting certificate server responded with a certificate or a control C and do a list - al this time the certificate looks bigger those at least the file looks bigger let's have a look at it Acme no JSON and here is the first thing this is a production URL doesn't contain the word staging that's how I recognize it and as the private key and then there is a certificate which was issued to this fqdn on this host and this is the actual certificate with its key that's it so it's running we just have to verify control-shift-n for incognito multi-tool dot multi-tool lab dot prod marcom like so HTTP I want to see redirection and action as well there you go it didn't give us any error it just simply got redirected and it's just got displayed and the certificate padlock is also properly locked and if you click on this one it says certificate is valid if you click on this you will see certificate is issued to this common name or this fqdn fully qualified domain name or hostname and it's issued by let's encrypt Authority x3 it doesn't contain the word fake in it so this is a valid certificate very good so that's it it works the HTTP challenge I've shown you that it works and it's there on a public public server I showed you in the diagram a scenario HTTP scenario 2 in which you you are behind a Rooter right here and I said you can just forward the pores to your server and they still use HTTP challenge and of course it makes sense but the problem one of the problems in such setups is that sometimes you don't have control over this router like I'm here in this office right now and I'm in a shared setup and the internet connection is actually controlled by another company here and I don't have access on it or access to it so I cannot set up I cannot set up this NAT port forwarding to my server and even if I had access to it maybe I already have another server running or someone else as another server running and they could they already have port MAP port forwarding enabled already so in cases where you don't have access or control over this route or if you can't modify the port forwarding for whatever reason then you can't use HTTP challenge then you have to use the NS challenge and that is the topic of our next video but for now recap we served as we said we served to container serving to different web sites from using two different fqdn or host names using HTTP challenge and getting certificates from Hudson grip both staging certificates and production certificate and it works and just I just remembered one more thing I should show you I told you that restarts are a problem if you are not saving this file right so before I switch to the other video or go to the next topic I want to show you something docker compose PS shows that my process my stack is running on my public web server and I have my file as well notice the timestamp 1711 it's a file running I will stop by stack and I will remove the running containers right and I just started again I checked the list of files my AK mentor JSON file is still there 17 11 I will just bring up my stack again and just check the logs what's going on here what happened I did something very quickly what I did was I did docker compose up and then I immediately did docker logs - follow up mode and I see docker provider running I see acne provider and it's the production server I mean production certificate server what is what is it doing at 17-17 its first checking the certificate for renewal it has found a certificate because I don't see an entry which says key is empty I found an empty key nothing of that sort there's no registration going on or nothing so I can press ctrl C here I checked the list of files and my certificate file is still time stamped with 1711 and if I check my web browser and do a refresh on this it still works it's a new contain that you notice here there was some change here but the certificate if I click this one the certificate is still the one which was issued at 5 11 17 11 so when you have persistent storage it doesn't matter how many times your service restarts it will still find the same certificate but I am not advocating that your soldier your applications should restart every minute or so they should not but it's just something that you should know all right so the HTTP challenge topic is covered it's finished let's go to the next topic which is DNS challenge and as you show you examples using that in the next video [Music]

Channel: Eficode Praqma

Views: 4,284

Rating: 4.8039217 out of 5

Keywords: trefik, proxy, tutorial

Id: OHDTfJzL6jg

Length: 43min 57sec (2637 seconds)

Published: Wed Jun 19 2019