Self-Hosting Your Homelab Services with SSL -- Let's Encrypt, MetalLB, Traefik, Rancher, Kubernetes

so you've been self-hosting a lot of services at home you've built up your home lab and now you have lots of applications running within your servers and the way you've accessed them is on your local network or through vpn and up until now you haven't been able to reach them outside of your own home network well that's about to change [Music] hey welcome back so i'm techno tim and today we're going to talk about exposing your services securely using a reverse proxy as a quick reminder i stream every tuesday thursday and saturday so if you want to continue the conversation about exposing some services there we can so let's talk about setting up a reverse proxy so what is a reverse proxy a reverse proxy in simple terms is just a server that sits in front of web servers that forwards clients traffic to the web servers they can be used in many different ways you can set it up with ssl and you have a lot of options out there and if you're running kubernetes at home it's been challenging to try to expose services through a reverse proxy and it's even more challenging routing all of that traffic to your micro services and then to top it all off you need ssl certificates this is where we can get a little help from open source so how does it all come together well i'm going to show you so today in this video we're going to set up metal lb traffic and let's a crib so that you can expose your kubernetes cluster securely now i've set up kubernetes using something called rancher now if you're not familiar with rancher rancher is an easy way to spin up and get kubernetes at home if you need help setting this up i've got a step-by-step tutorial that will walk you through setting up brancher kubernetes and docker all within a couple of minutes and once you got that set up you're ready to start this tutorial so without the way let's get started so first you want to make sure your rancher server is up and running it's a good idea to back up your rancher server before we do any of this this is going to be a pretty complex tutorial so be sure to follow it step by step and it's a good idea to back it up just in case something goes wrong the next thing you'll need to do is port forward http and https to your rancher server this is because a reverse proxy and our load balancer is going to handle this traffic next you'll want to create a couple of dns entries these will be dns entries into whatever dns system you're using you'll want to add one for traffic.example.com and one for the domain you plan on hosting this is just so we can reach it internally while we're testing next you'll want to reserve a block of ips this will make sense later but just be sure we have a block of ips that aren't getting handed out by your own dhcp server next we'll want to set up coop control now some people say cube control and some people say coupe cuddle but i usually say coop control not really sure how it's pronounced but anyway you can set up coug control by looking at the kubernetes documentation if you're on a linux link system this is as simple as running a curl command if you're on windows i highly recommend setting up wsl on your windows machine this will give you a linux like experience on your windows machine and you'll be able to copy and paste a lot of this documentation if you need help setting this up i've got a guide on setting up your windows machine for javascript development now i know we're not doing javascript development here but it walks you through setting up your machine with wsl and then you can ignore the javascript stuff at the end or you can do javascript once you have it installed you should be able to run this coupe control version dash edge client and see it pipe out your version if not you'll want to get that set up so recoup config allows us to connect securely to kubernetes to manage it now we'll do most administration in management within the ui but we'll need it for a small piece so to get your kubeconfig you'll go into your cluster click cluster and then we'll click coupe config file we need to copy this config to our local machine so let's copy that to clipboard then we'll paste that in a config file within cube under your profile and then to test to see if it's working you can run a command like coop control get name spaces this should return with some name spaces okay so now we're ready to install metal lb so why do we need metal lb that's a great question and i didn't know the answer to this question a couple of days ago so kubernetes clusters are meant to be connected to a load balancer so if you try going to your kubernetes cluster you may have noticed that you either get an error or that you get a self-signed certificate this is the kubernetes ingress controller responding to you so kubernetes installs are always looking for a load balancer and metal lb spins up a load balancer for you so that kubernetes cluster can communicate with the outside world so it's really easy to set up and here's how you do it so we'll run this first command to configure the middle lb namespace then we'll run this next command to deploy middle lb then we'll run this last command to create a metal lb secret then we should be able to run coop control get namespaces and we should see our metal lv name space there we can also verify that too within the rancher ui if we go to cluster our default cluster and go to namespaces we should see metal lv here next we'll want to create a config.yml so this is the config that's going to be applied to our middle lb here's an example one that i'm using that works really well so the interesting part is down here under addresses so remember how i said to set aside some ip addresses within your subnet well here's why so i've set aside a small block the way that i've been using metal lb so far is that i only need a couple but you'll want to change this according to your subnet this is the network that rancher lives on and then once you've saved that yaml file you want to apply it so we'll run kube control apply f config.yml so we've done all that and we don't see anything yet that's okay once we set up traffic you'll see how this works so the way that we're gonna set up traffic is going into our cluster and we're actually gonna go into the system namespace now typically you wouldn't install anything here but traffic needs to be installed here so let's click on system and once we get here we'll click on apps now there isn't anything here oh that's because we're going to deploy them so let's click launch so now you'll see a bunch of apps here so these are pre-configured apps that you can install a reminder don't install them in the system name space unless they tell you to but in here we can find traffic once we click on traffic let's fill out this form so the name can be traffic the namespace this is really important so let's choose use existing namespace then we'll want to choose the kube system namespace we'll keep the service type the load for balancer use default images fine we can keep debugging to false ssl we'll set this to true now we won't change http and https yet permanent redirects we'll keep defaults for now and let's encrypt let's turn on set this to true email let's enter our email keep on host to true logging to true we're going to change the challenge type we're going to change this to dns01 persistence is true and enable we're going to keep to true in enable dashboard we're going to set this to true and here we're going to set a name for our traffic dashboard now you can set the domain to anything you want but i set this as traffic.example.com basic auth this is where you would set basic off you'll need a htgenerator to do this and this is where you would enter it now before we hit launch we're going to change a couple of things so let's go up here to edit as yml and here we're going to add a couple of fields so under acme we're going to add a couple of properties we're going to add dns provider then the name of my dns provider as well as existing secret name now we'll set up the secret in a second but going back to the dns provider traffic supports lots of dns providers and we'll actually need to get an api key from our dns providers so that we can verify our domain i found that this is the simplest and most reliable way to get certificates you'll want to check the traffic documentation for all the supported dns providers but i use cloudflare and this is how you do it and while you're there you'll probably want to set up your dns record now i'm going to be standing up one on my website it's technotim.live and i have a dns record pointing back to my ip address and then i have a dns record internally pointing at my rancher cluster you want to be sure that this is set up before proceeding so after clicking launch traffic is here but it's not quite ready yet we need to set up one more thing so now we need to set our secrets for our cloudflare config you can do this by going to cluster and go to system make sure that you go to system because this is where traffic is running and then we'll go into resources and secrets and you'll want to create a new secret so i called mine cloudflare dns and for secret values it's a key value pair the first one is cloudflare api key and then your api key from cloudflare and the next one is cloudflare email and that's going to be the email address you use to register with cloudflare and if you're using a different dns provider you'll have different values here you can check traffic's home repo for the supported values and if you don't see it there you'll see it in the logs if you try to start traffic without these values and take note of the name of this secret because you'll use this in your traffic configuration mine's cloudflare dns so we need to set up a persistent volume claim for this this is so it can store the certificates for us so we do that by going back to our cluster and then click on storage and go to persistent volumes here you want to add volume and you'll want to name it so here i'm just going to name this custom and for a volume plugin let's just store it on this host so let's choose host path then you'll want to specify a path where we're going to store these certificates this is the path on the server and it can be anywhere as long as rancher can reach it under customize we're not going to change anything here and then we'll save and then this should create a persistent volume if we go back to cluster system and go to apps and we go into traffic we should see that traffic spinning up so a couple of things to call out here first under volumes you want to make sure that your claim is working this should say bound here if this doesn't say bound that means your claim's not set up properly so you want to get that working next you'll want to be sure that you see some endpoints here now if you don't see any endpoints here that means that metal lb isn't doing what it's supposed to be doing and you'll need to troubleshoot that but you'll notice that one of the eyepiece is one of the ips in the range that i set up for metal lb so this means it's working and traffic.example.com this is our traffic dashboard if you set up that dns record locally it should be working so if we open it up here's a traffic dashboard so this doesn't look too interesting now but it'll get there okay so if you see these endpoints here this would be a good time to switch your port forwarding so previously you had port forwarding going to your rancher server now you want to switch the port forwarding for http and https to the metal lb ip address which is right here so mine's 192 168 3.201 and on my firewall and my nat rule i forward http and https so you want to set that up so now we'll need to deploy our kubernetes workload for me this is going to be my simple website so here i'm going to go to cluster default and i'm going to click launch and then make sure i'm on workloads here we're going to click deploy here i'm going to name my workload so it's techno tim live then i'm going to set my docker image this is a custom one for me because it's a custom one i built and this is again part of the reason why i like using kubernetes at home this allows me to deploy workloads at home just like i do in the public cloud and if you'd like to see a tutorial on how to build custom docker images and store them in a registry and then deploy them to kubernetes at home let me know in the comments section below but anyways i'm not going to set any port mapping here i'm just going to click launch so now my workload is running but it's not exposed so the way you typically do this is by a load balancer and when you create a load balancer that creates an ingress that exposes this service inside of the cluster and if you've done this before you'll notice that you can then get to that service but you don't have ssl well actually you do have ssl but it's the kubernetes self-signed one not the one we're getting from let's encrypt but anyways i'll show you how it's done so i'm also going to open traffic in this other window so you can see what traffic's doing this is where it gets cool so if i go to add ingress let's name our website so it's techno tim live now this is the dns name or the host name that we want to route to so i'm going to put in techno10.live that's my domain and then for choose workload i'm going to choose this workload we just created which is techno10 live and then the port is 80. so my workload exposes port 80 within the docker container now we won't add any certificates or add any annotations and make sure this is the default namespace too and we'll click save now as soon as we save that we see traffic spinning up a back end pointing to a front end and it says we have http and https and my ingress is still initializing but let's go take a look at traffic so if we go back into system and then apps let's go into traffic and let's actually look at this traffic workload and then we'll go over here and go to logs and if we look in the logs we can see that traffic is actually doing stuff so the acme part of traffic which is the let's encrypt client is actually going out and validating my dns so we can see here an info message the server validated our dns validation succeeded and it responded with a certificate which means we have a staging certificate and so if we go to our domain so mine's https technotim.live we should see a certificate error and this is to be expected that's because we requested a staging certificate from let's encrypt so let's take a look at this certificate advanced view certificate and we can see right here fake le intermediate x1 and if we scroll down the qualifier and the value http cps let's encrypt.org so this is a good sign we're getting a staging certificate from let's encrypt eventually we want a production certificate so i highly recommend setting up all your workloads and validating that you get a staging certificate before moving to production the production api has rate limiting enabled and if you exceed the amount of errors within a certain time period you'll get blocked so i highly recommend setting up staging certificates before moving to production and verifying they all work but once you're ready let's get a production certificate so we'll need to do two things first we'll go back into traffic so that's cluster system and then traffic and once we're in here let's scale him down to zero scale down to zero so he should remove his pods and if we go into traffic and check it shouldn't be any pods here okay so he's good and he's shut down we're gonna have to go into the persistent volume that we created in here you'll find an acme.json file you'll want to remove this file this contains your certificates so i've had to do a sudo rm acme.json and once that's gone verified it's gone and it's gone then we'll go back into traffic and we'll upgrade this workload here in the ammo we'll want to change staging to false once you set it to false let's hit upgrade then we'll go back into traffic and now we'll need to scale him up scale them up to one let's go into this pod okay so this pods running let's look in our logs okay and we can see that validations have succeeded so that means we have a production certificate let's go check and now we can see our website loads let's check this certificate connection secure verified by let's encrypt more information and we can see our certificate so congratulations we've exposed our kubernetes cluster externally using metal lb we set up a reverse proxy with traffic we enabled let's encrypt to get certificates for us and now we can access our website securely anywhere in the world and this is all automated now i know that this tutorial is a little complex but kubernetes is complex i tried to make this as simple as possible and that's why i choose rancher to help spin up kubernetes and hopefully you're learning a lot along the way so what do you think about exposing your kubernetes cluster externally what do you think about metal lb traffic or let's encrypt did you have any problems along the way if so let me know in the comments section below and while you're in the comments don't forget to give this video a thumbs up and subscribe if you haven't already and if you have more questions you can always join my live stream i stream every tuesday thursday and saturday so if you have a question about this video or any of my other videos hop in my stream and i'd love to have you so thanks so much for watching and until next time stream on my friends what will your next video be about uh it's uh it's coming i i gave lots of hints i already gave lots of hints in this while i've been talking and uh i gave hints on the first video i i usually don't tell him that it's a huge surprise but you'll have to see and it'll be coming soon i need to finish editing it uh or it will not get out saturday morning saturday morning is my is my release day that's a i release every saturday morning it's like saturday morning cartoons back in the day except for their mediocre tech videos by me

Channel: Techno Tim

Views: 79,533

Rating: 4.9631424 out of 5

Keywords: homelab, selfhosted, self-hosted, metallb, rancher, docker, kubernetes, k3s, k8s, home lab, letsencrypt, let's encrypt, how to set up traefik, free certificates, dns, reverse proxy, self-hosting, homelab services, homelab security, ssl, certificates, kuectl, kube config, how to install metallb, open source, rancher 2, free certificate, acme, staging, production, load balancer, cloudflare, techno tim, tutorial, proxy, web server, cluster, traefik, helm, rke, micro services, https, how to install traefik

Id: pAM2GBCDGTo

Length: 16min 59sec (1019 seconds)

Published: Sat Aug 15 2020

Reddit Comments

[u/kalpol]

And here I am running apache on FreeBSD like a pleb.

[u/Techno-Tim]

Here's my latest tutorial on self-hosting services in your homelab. It includes instructions on how to set up MetalLB and Traefik for your Kubernetes cluster running on Rancher. Hope you like it!

[u/Wulfnor]

Hi Tim,

Ive tried to follow this tutorial today but i encountered some problems:

First, after doing eferything as in tutorial Traefik hang on"installing" status and throws this error in rancher ui:

Error: render error in "traefik/templates/dns-provider-secret.yaml": template: traefik/templates/dns-provider-secret.yaml:1:80: executing "traefik/templates/dns-provider-secret.yaml" at <.Values.acme.dnsProvider.name>: nil pointer evaluating interface {}.name : exit status 1       

Ive found out that this may be caused by deprecated dnsProvider variable, and should use dnsChallenge instead as in documentation

https://docs.traefik.io/v1.7/configuration/acme/#dnsprovider-deprecated

So ive changed config to:

   dnsChallenge:
   name: "cloudflare"

evrything installs but i dont get ssl cert, in pod logs im getting:

{"level":"error","msg":"Unable to obtain ACME certificate for domains \"xxxxxx\" detected thanks to rule \"Host:xxxxxx\" : cannot get ACME client unrecognized DNS provider: nil","time":"2020-09-06T23:39:18Z"}

this tells me that dns provider name is not getting set. After reading helm chart readme there is:

 option:acme.dnsProvider.name   Which DNS provider to use. See here for the list of possible values.    defaul:nil

but now... i dont know how to set everything. It seems that somewhere down the line Traefik changed config options name, but helm chart is not updated and im unable to pass proper dns provider name to pod.

Also BTW: traefik seems great for exposing docker containers to the world, but what with vms ? how to pass to traefik info about vm that run in proxmox ? I dont want to make two reverse proxy servers, one for vms and one for containers, it seems messy.

[u/Boonigan]

Your videos have been great for me. I’ve recently started getting into Homelabbing and you’ve provided some awesome content for me to go off of. I really appreciate it!

[u/kasngun]

Nice! Discovered you the other day where you showcased your home lab and also what you ran on them, thought the videos were really cool. Got yourself a new sub, looking forward to seeing more content from you :)

[u/WunderTechTutorials]

Love your videos - always super helpful content. Keep doing your thing!

[u/monkeychef]

Killing it dude. Looking forward to going through this after following your Heimdall and Rancher. Thanks!

[u/thehedgefrog]

Cool! I'll take a look tomorrow. I'm currently reworking my entire infrastructure, I took down my 2 ESXI machines (i7 gen 2 16gb RAM, i7 gen 6 32gb RAM) when I moved in May and haven't taken the time to get 2 4U enclosures to rack them so they're collecting dust, I migrated all my services to Docker using Compose on my QNAP NAS (40gb RAM). Traefik, CloudFlare, Authelia, zero-trust exposed.

I fell in love with containerized services, posted a very short time ago on how to scale (I have 2 Dell Wyse thin clients brand new in box that I paid $20 each for that I'm dying to use). I want to ease the NAS's workload and eventually use my 2 i7s and 2 thin clients along with the NAS for containers. Playing around with Docker Swarm but there are some obvious limitations, and I'd like to give k8s a try. Rancher seems cool. I'll definitely watch and try to learn.

[u/Cook1e_mr]

I'm trying to follow this guide and have followed everything. However I dont get an endpoint in traefik for the traefik.example.com. therefore cant seem to access the UI. How do I troubleshoot this?