Hi everyone and welcome to another video! Now you guys have asked me for
a long time to make a video about what i run on my home
server, in terms of software And I planned to make this
video about two weeks ago But unfortunately because of me
moving out to a different apartment it was delayed And as you can hear there's
a lot of echo in the video So i'm sorry if that's too disturbing But we're gonna have to live with that for now So let me just open the home page
of my server here at homer.box And first things first: this homepage that i have, It's called Homer it was made by Bastien Wirtz I'm gonna leave a link in
the description to the Github I'm also using my own custom
CSS for that Nord theme Which i'm also going to leave in the description So be sure to check that out if
you want to replicate my setup So i'm just going to go, I
think, you know, one by one And i'm going to tell you what
i'm running on my home server So first, this is Jellyfin. It's a media server that
lets you watch all of your Legally obtained content in a Netflix-like UI I'm sure that a lot of you
are acquainted with that So i'm not going to, you
know dwell too much on that Next up we've got Deluge This is where i download all of my
very legally obtained linux ISOs I'm obviously not going to
show you what all I have there Because it's very private linux ISOs You know, like Hannah Montana
Linux, Carly Ray Jepsen Linux, You know, all those very
secret and illegal Linux ISOs That you probably shouldn't see So next up we have Radarr This is what I use to download
(once again very legally) movies And there's also a similar
service for series, for TV shows which is called Sonarr Here I also have a bunch of, you
know, very legally obtained series Okay I'm going to stop using
the "legally obtained" phrase Because it's getting old But I also have a similar service for books which is called OpenBooks This basically acts as a IRC client Which logs you onto those,
you know pirated book channels Now, if you don't know
about this whole rabbit hole Of IRC channels where you can get free books You should definitely Google that, Because that's how you get
a lot of, you know, EPUBs And digital books very easily, Without the need to go on those,
like, fishy websites and services And infecting your PC with a bunch of viruses So yeah definitely check this out Now once again I want to say that I
am not endorsing piracy, obviously Piracy is bad. So this is all just for educational purposes, so that you know which services
you definitely shouldn't use Next up we got Nextcloud Now you all know about Nextcloud already, I'm not even going to introduce it Basically this is a self-hosted alternative to services like Google Drive and Dropbox I'm also not going to log in just yet I'm just going to show you that it does support two-factor authentication with either TOTP or, you know, something like Yubikey So that's pretty cool I'm not going to show you my Nextcloud, because I don't know if there's some kind of private stuff there or not But i'm basically using it for several things For calendar and contact synchronization, Because I want to kind of go away from iCloud I don't want to rely on that for my
contact and calendar synchronization And I'm also using it for, like, you know, general purpose file sharing,
with friends, family and stuff And that's what it's pretty good for The whole calendar and context synchronization is also the reason why I don't
use, for instance, Seafile Because it is, as far as I
know, centered on file sharing And doesn't really have any kind of contact or Calendar sharing functionality So that's why I use Nexcloud Now, next up we got PhotoPrism This is basically a self-hosted alternative to stuff like you know Picasa (Remember that one?) Or Google Photos It gives you a very nice, very
neat gallery of your pictures As you can see I... I don't really have, you know, any cool pictures in here...
Nothing private either Just a bunch of tech stuff,
some wheels for my car, Raspberry Pi gore, all that stuff So I'm not afraid of, you know, showing it But yeah, basically I have this automation where You know, my phone sends
pictures to WebDAV of Nextcloud And then those get imported to PhotoPrism Basically that replaces the
iCloud Photos on my phone And that's what I use instead This is pretty good, I would say It also has pretty cool, you
know, "AI" kind of features Where you can type like "car",
and it's going to show you What it thinks is a car, you know. Some of them are obviously
not a car but most of them are So I'd say that's pretty good So yeah, PhotoPrism. Really cool. Next up we got Bitwarden. Now Bitwarden is a password
manager which you can host yourself However, I am not hosting
the official Bitwarden image And instead I'm using something called Vaultwarden Vaultwarden is basically an open-source
implementation of Bitwarden based on Rust And it's pretty good, because all
those features that you have to pay for in the official Bitwarden server?
Here you get them for free So stuff like two-factor
authentication and, you know, a lot of other features For me mostly it's important
that, you know, a password manager has two-factor authentication,
so that's what i'm using it for So yeah it's pretty cool Obviously I'm not going to show you my passwords You can you know check it out for yourself I used to use KeePass and just, you know,
synchronize the database with Nextcloud But that doesn't work very
well with mobile clients I just... I could have never
gotten it to synchronize properly So Bitwarden is the next best thing And one of the advantages of Bitwarden
that is very often overlooked Is that it caches the passwords and, you
know, other information on the device itself So even if you don't have the access
to the instance, so for example you're sitting somewhere without the
Internet or your home server is down And you can't access your instance, You still get your passwords, you
don't lose access of your accounts Which is you know what often happens
with other cloud-based password managers So that's pretty good So let's move to the "System" category Here, first up we got UniFi controller This is pretty useless for now I do have a UniFi 6 Access Point This is what I use for my WiFi But it is kind of useless for now Because every time my server reboots
I have to reintroduce the WiFi network So you can see here...so I kind of have it here Sometimes I'm not too lazy to
adopt it, sometimes I am too lazy But it's sitting there in my
home page or, you kno, in Homer And you know I...I'm kind of too
lazy to set it up properly, I guess Next up we got Jackett. Now Jackett is what basically
acts as a torrent tracker indexer For Sonar and Radarr; those two
services that can, you know, Help you download movies and TV
series and you can just, you know, Paste some torrent trackers here that you like and it's going to index them and
provide, you know, an indexed database, I guess to those two services,
not much to explain here really Here I have my home router I'm using a FRITZ!Box and obviously,
I'm not going to show you the UI Because there's probably a
lot of private things in there But this is basically just an admin panel
for my home router, nothing interesting here And here we got PiKVM. So as you already know if you previously
watched my, you know, home server hardware video Which if you didn't, you
should definitely check it out, I have a Raspberry Pi in my home
server which acts as a poor man's KVM And as you can see, it provides me
with a display output of my home server So you can see, I have my
home server here right now I can actually reboot it
right now out of the PiKVM And i can enter the BIOS settings,
as you will see in a second So as you can see, this is the BIOS screen, I can do that without taking
the home server out of the rack, putting it on a table, connecting it
to the monitor, the keyboard, etc. This is, like, very exhausting
and every time there's an update That breaks SystemD or, you
know, breaks kernel updates, You know, you have to do it,
and it's just very annoying So PiKVM is a very cheap and a very
versatile solution that you can use It's basically the cost of Raspberry
PI itself and an HDMI capture card, Maybe, you know, a couple of cables But it is a very very awesome
thing to have if you, you know, Don't have an industrial or
enterprise server grade motherboard what has IPMI or management ports This is basically a very cheap
and very cool solution I guess So next up on our list is PiHole Now Pihole, that's something I
actually run on a separate device I have an extra Raspberry
Pi that runs PiHole and VPN So that I can VPN into my home network even
if my home server is down for some reason And PiHole also acts as a local DNS server So I'm going to show you... Let me just log in real quick So as you can see I got, you
know, all those cool stats But the reason I'm using it is local DNS So even though I'm on my home network And all of those services that I just told you
about aren't exposed to the outside network Via some kind of an HTTP
proxy or reverse proxy setup I can still have the pretty domain
names that you see right here So if I enter something like 'homer.box'
it's going to show you my Homer homepage And if I enter 'jellyfin.box'
it's going to bring me to Jellyfin So that's pretty cool, I think One more thing this PiHole setup is accomplishing is it basically acts as a recursive DNS resolver It also runs Unbound which forwards
all the DNS requests to DoT servers which I'm gonna show you later So this basically makes sure that
all of my DNS, you know, requests, All of my DNS traffic is going through
DNS over TLS, so that's pretty cool Now home automation is something that I
also have to make a separate video about Because it's a whole huge topic on its own But I'm going to kind of
show you around real quick So Phoscon-GW is just a web
interface for my ConBee II USB stick If you're not familiar with Zigbee
devices, this is, you know, once again, going to be a huge rabbit hole And I'm sure I'm going to
talk about it at some point It basically just lets me adopt new Zigbee devices As you can see I have a couple here,
and I also have thermostats and stuff But a place where the most home automation
for my house is taking place is right here It's Home Assistant, and as you
can see I got a lot of stuff here And I can just basically turn off my
lights like this, with just one click I can turn on the big lights, I can turn
them off, I can make it hotter in my room Because I also have smart thermostats I can make it colder, etc. This is all i think pretty cool, you know, a
lot of, you know, good quality of life things And the reason I'm not running some kind of, you
know, a cloud centered, off-the-shelf solution Like maybe IKEA smart home stuff, or Philips Is because I want to have full
control over my smart home devices I don't want to be left, you know,
completely stranded without heating If one of those services goes down I want it to be absolutely self-sufficient,
I want it to work without Internet Even if nothing else works, even
if i don't have any Internet, I want my heating to still
work, obviously, you know, like... just like any normal heating But i also want to have
those "nice to have" things Where, you know, if you go out
of the room, and you're like "Aw, I forgot to turn off the
light, but now I'm too lazy" And then I just do this and bam, the light is off I know it sounds like a bunch of
first world problems, you know "What are you, too lazy to
get up and turn up the light?" But honestly it's just, like,
it's been a fun project for me And I'm thinking I'm gonna try to
make more and more of my home smart Just for the sake of it, not because I'm too lazy But because it's just fun, you know So I'll definitely keep you
posted on that and, you know, You should expect a video
about Smart Home very soon Which in my channel's language is "Maybe this
year... I don't know, can't promise anything" So yeah, that's actually it, those
are all the services that are run I don't run a lot of stuff I also want to tell you about
those green and red indicators They're supposed to indicate
whether a service is up or down But because of a lot, of you know, HTTPS and
CORS shenanigans it doesn't work as expected For example Nextcloud reports that
it's down even though it's obviously up So now that we've covered the basics,
you know, what services I'm running Why I run them, etc. I want to get to the, you know, more
exciting part of my home service setup And that is the under the hood stuff So how I set my home server up, how
I provision it, how I install the OS How I, you know, set this up And how i do it all completely
automatically... well, almost Let's just get right into it
and I hope it becomes more clear Now obviously, for you, this list
of services might be very different. You might not need half of the stuff that I
run, or you might need something else entirely. Because of that, you should really take
a look at the Awesome-Selfhosted Github. It has a huge list of self-hostable
services that you can run yourself, so make sure to check it out. Now that I've told you what I run,
I want to talk about how I run it. So right off the bat, the OS I use
on my server is Ubuntu Server 20.04. I know that this choice
might make some people upset, but I've tried pretty much every mainstream
option under the sun for my server, including some obscure stuff
like OpenBSD and Gentoo, and Ubuntu just works for me. Besides, Ubuntu includes cloud-init, which is
crucial for deploying and provisioning my server. The reason I'm not using a
NAS-centric OS like TrueNAS or Unraid is because I find them too limiting
in terms of how you run things. With TrueNAS you have to run ZFS, and Unraid
expects you to do everything through GUI, So things like setting up Docker containers
or storage can't be efficiently automated. A barebones Linux distro with a command
line interface just works better for me Because it gives me more freedom and flexibility. As for storage and filesystems, my
hard drives are formatted in XFS And I'm using a MergerFS array to
combine them into one big storage pool. MergerFS lets you use drives with different
file systems and even different capacities, And you can add more drives
to the pool after the fact. I'm also using a 1TB NVME drive as a cache. MergerFS doesn't have a native
support for cache drives, But there's a cool little guide on the MergerFS
Github that outlines a workaround for that. I've set it up so that the files are moved
from the NVME drive to the hard drive array after they're not accessed for 5 days That way I can copy files to the array
really fast, thanks to the 10 gig networking, and also copy the fresh files from the server. And at the same time the NVME drive doesn't get clogged up with all kinds
of useless files over time. When it comes to redundancy, I am using SnapRAID
to store parity data on an extra 6TB drive. That way if one of the drives
fail, I won't lose all the data. MergerFS and SnapRAID is basically the best of
both worlds, since it isn't as limiting as ZFS, where you need same size drives and can't
add more drives to the array after the fact. At the same time, it provides you with redundancy and lets you add an SSD to act as
a fast cache, which is pretty neat. SnapRAID also lets you exclude some
files and folders from the parity for example I have my Downloads folder excluded, since I can always re-download
things if I lose a drive. As for running my web applications, I use
Docker containers for pretty much everything. This way I don't have to worry about
dependencies or persistent storage – I just set the containers up once and I back up my persistent storage folder to
the hard drive array weekly. Next time I set up my server I just have
to copy the persistent storage folder back and launch the Docker containers again. Now, security is a pretty big
topic when it comes to selfhosting, and a lot of you guys asked me how to properly
secure your services from unauthorized access. Around a few months ago after an
avalanche of posts on r/selfhosted It became apparent that a
lot of people simply don't know how to secure their self-hosted applications. There were posts with dashboards, admin panels, web applications hosted on VPSes
with no authentication at all. And that wasn't some kind of a
0-day issue or a hidden exploit, People simply host their stuff and don't think
about protecting it from unauthorized access. When it comes to SSH, everyone and your mom knows
that you should use a public-key authentication, lock the access for the root user, and move
your SSH away from the default port if you can. But somehow, people don't pay the same kind
of attention to their web applications. So, if you want to avoid making the
front page of r/selfhosted, first of all, You should definitely close all of
your ports except for the HTTPS, SSH, and all the other ports
that you absolutely need. You can use firewalld, UFW,
IPTables, or whatever you prefer. Instead of accessing your web applications
directly, you should use a reverse proxy setup. Personally, I use SWAG –
Secure Web Application Gateway. It automatically fetches and renews the
SSL certificates for all your domains, includes fail2ban and also comes with a
bunch of presets for common web services. Single sign-on is one more thing you
could do to improve your security. By using a service like Authelia you can
make it so that anyone accessing your web applications will see an authentication gateway. Authelia supports two-factor authentication
as well, with either TOTP or U2F, which is a great way to protect your web services. You don't necessarily need single sign-on if all of your public web services already
support two-factor authentication, such as Nextcloud or Bitwarden, but if you're especially paranoid about
unauthorized access to your services, you should definitely take a look. One other thing that you could
do for your security and privacy is using a Cloudflare proxy for your web services. If you're hosting public
facing services from home, you don't necessarily want
everyone to know your IP address. Cloudflare basically proxies your HTTP and
HTTPS traffic through their own servers So that the domain names for your public services
don't technically point to your home IP anymore. Moreover, they also don't proxy any
other ports apart from HTTP and HTTPS, So other people won't be able to access SSH
or other services using your domain name. The free Cloudflare plan is more than
enough for the proxying functionality And it pretty much requires no
additional set up on your machine. Dynamic DNS is what you need in 99% of the cases if you're have a regular
Internet connection at home And want your services to be
accessible to the outside world. In most cases your ISP assigns
you new IP every now and then, so a dynamic DNS service makes
sure that your domain name is always pointing at your home
IP address, even if it changes. The easiest way to do that if you
don't use Cloudflare proxying and don't have your own domain name is DuckDNS. You can get a free domain
name like "domain.duckdns.org" and updating your IP is as
easy as setting up a cron job. They have instructions for every setup
imaginable, so make sure to check it out. If you have your own domain name,
ddclient is probably the way to go. They support all kinds of domain name providers, and I had zero issues making
it work with Namecheap. Now I'm proxying some of my
services through Cloudflare, so I use a Docker container
called "cloudflare-ddns" by oznu. It's very simple, you just fill out the
options, run it once and forget about it. Now obviously, you don't need any of
that if you have a static IP address, But that usually costs extra unless
you're a business client, so there's that. Now I want to tell you a little bit
about how I optimize provisioning, deploying and configuring my home server. Back in the ye olden days, I used to set up
my Linux boxes well, like everyone else does. I installed Debian or Ubuntu, I
partitioned the drives, created the users, set up all the Docker containers, Samba shares
and so on, and so forth. Manually, of course. Setting up a new machine is
a pretty boring process that sometimes can take several weeks of fine-tuning, troubleshooting and googling,
but hey, once you set it up, it's done forever and you never
have to do it again!.. Right? Well, yes. But there are many ways
in which things can go wrong – Drive failure, natural disasters and just
simply borking your installation beyond repair. Happens to the best of us. And besides, I don't know about you guys, but after I install Linux on a server and use
it for some time, it kind of feels... dirty. Like I don't know what's on it anymore,
and I don't have any control over it, so I always have this itch to
reinstall it, to make it 'cleaner', less bloated, and better this time There's also another problem –
when setting up a new Linux machine I personally tend to come across issues
that are not so easy to diagnose, but once I solve them, I
immediately forget how I did it, So next time I have to reinstall,
I get the same issues again and have to go through the whole
troubleshooting process again. And I think I found a way to set up my server in a
cleaner, less bloated and an overall better way – it's called "Infrastracture as Code" Infrastracture as Code is a philosophy of
managing configurations for your machines in the same way that you would manage code. Instead of setting your box by hand, you declare the state of your
machine using a tool like Ansible, and bring your machine to that state
by running an Ansible playbook. Right now those of you who've heard
about NixOS are probably going "Wait, I've heard all of that before!" And yes, actually NixOS has
a very similar philosophy. And Ansible lets you apply the same
principle to any Linux distro you want. Just declare the desire state of your machine and bring it to that state by
executing one command. Pretty cool! Ansible is a very powerful tool
when it comes to Linux automation, and I'm not going to go too much
in depth into it in this video. But I will definitely leave some links
to videos, books and documentation that can help you learn Ansible if you want to. For now though, I want to answer
one very important question – how is Ansible better than just using a bash
script to automate your installation process? Obviously, there are many use cases
for which Ansible is a no-brainer, and using a shell script would be ridiculous. But using Ansible for setting up a single
home server seems to be overkill, right? After all, things like setting up users, partitioning and mounting hard
drives or installing packages, can definitely be achieved by a
shell script, right? So, why Ansible? Well, let's start with idempotency. Idempotency is the ability to
run a program multiple times and have it produce the same result,
regardless of the system's state. Here's a very easy example –
let's create a folder with mkdir. As you can see, running it once works
as expected – the folder is created. But if we run it again, we'll get an
error – because the folder already exists. However, if we add the "-p" flag
to the command, we'll see no error. So now, the command is idempotent. However, in other cases it might not be as easy. For instance, some package
managers will complain if you try to install a package that's already installed, and in a lot of other cases you'll have to add
error conditions, write "if" loops, and so on. However, with Ansible, idempotency is baked in – Ansible will not change the state of your
system unless there are changes to be made. Whereas with shell scripts, you'll
need to constantly reinvent the wheel, add extra checks and conditions to
make sure your code is idempotent, with Ansible you don't need to worry about it. Next up, readability. Ansible playbooks and tasks are much better in terms of readability than
your average shell script Which means, they're also easier to maintain. Ansible encourages giving your tasks
names that reflect what they're doing, and pretty much every module
has very intuitive parameters. The structure of Anible projects
can be intimidating at first, but once you read some documentation,
it definitely brings some clarity. Another reason why I prefer Ansible to
shell scripts is built-in secret management. Ansible Vault lets you create encrypted
variables that aren't stored in plain text And require a password for decryption. The password inputs can also be hashed, which prevents the encrypted
variables from appearing in plain text on their way to the targret machine. Finally, Ansible pretty much
only has one dependency – Python. You don't have to install Ansible
itself on the target machine, and Python is installed by default
on all of the major Linux distros. The only thing you need is an SSH
connection to the target machine. Thanks to modules such as "package" and "service", Ansible playbooks can also be used on a
huge variety of Linux and BSD-based distros, since many modules are distro-agnostic. So now that I've sold you
on the whole Ansible thing – let me show you the Ansible playbook
that I wrote to set up my home server And tell you about some tricks that
I've implemented to make my life easier. So here is my playbook. I know that it can be pretty confusing
if you've never dealt with Ansible, so I will leave a link to the
Github repository with the playbook So you can take a look at it
and explore at your own pace. But here are a few highlights. First, I run this file named `ssh_juggle_port` As you might know, I usually change my SSH
port to 69 to avoid automated bot attacks, but doing that also means that Ansible
won't be able to connect to the server before the port has been changed. This file includes some tasks
that try both options – 22 and 69, And can also try to connect to the target host
with the default Raspberry Pi credentials, Which is handy for setting up my Pi. Next, I set up my users. I create the login user, add him to the
sudo group and allow passwordless sudo. I also copy my public SSH key and all
the necessary groups for the user. After that's done, I run my essential role. This role takes care of the initial
system setup, sets up APT mirrors, updates the system and installs
all the neccessary packages. It also clones my dotfiles, sets the hostname
and disables the login messages in the terminal. Then, I set up docker and MSMTP, which sends me e-mail notifications
when something goes wrong with SMART, Snapraid or any other services. After that, I set up all
of my Docker containers... And then I also set up Samba and NTP. Lastly, I harden my SSH configuration, change
the default port and restrict user access. Obviously, if I were to go over each and every line in my tasks and roles and
explain how everything works, that would take the whole day. So instead,
I'm gonna show you a dangerous magic trick. Kind of like sawing a person in half
and then putting them back together. Here's what I'm gonna do – I'm going to completely erase my boot drive
and provision it from scratch using Ansible. Sounds scary? Well, I'm kind of nervous too. So let's get into it. So the first thing that we need to do is
upload the Ubuntu Server ISO to PiKVM. PiKVM will then pretend to be a USB drive and
we'll be able to boot from it to install our OS. But before that, we need to modify the Ubuntu
installation ISO for unattended installation. For that we'll use my Ansible
role called "ubuntu_autoinstall". This role will inject a
special file called 'user-data' which will be filled with the
variables that we specify, Like the serial number of bootdrive,
user, password, public SSH key, and so on. After the ISO image has been
downloaded, verified and repackaged, Ansible will upload it to
PiKVM using the official API. As you can see here, the
drive icon is now spinning, which means that the upload is in progress. Now that the iage is uploaded we can
reboot our machine by typing `sudo reboot`, and then start mashing F1 and F12
(because I'm not sure which one it is), to bring up the boot override menu. Here, we're gonna choose "UEFI: Linux USB
Gadget", press Enter, choose the first option, And that's basically all the
work that we have to do for now. The installer will now initialize
cloud-init and install Ubuntu automatically, without any manual intervention, using the
variables that we supplied in the playbook. Alright, the installation is now complete, So I will go back to the terminal and
run the ansible playbook called `run.yml` Which will set up my server from scratch. It will create the users, update
all the packages, mount the drives, Install Samba, NTP, Docker
and other necessary packages; Restore the persistent data for all the
docker containers and bring them up. It will also harden the SSH access and
change the default port from 22 to 69. As usual, I will leave a link to the
Github repository with the Ansible playbook In the description if you
want to take a closer look, And if you'd like to see a separate
video with a step-by-step walkthrough And a deep dive into my
Infrastracture as Code setup, make sure to leave a comment below. And there we go! Impressive, right? Here's where I'm supposed to say
"don't try this at home, kids", But actually – do try this at home! Take a look at my GitHub repository and see if
you can adopt Infrastructure as Code yourself! It's gonna change your life. Instead of having so called 'snowflake'
servers with unique configurations That can't be reproduced and will take a
long time to recover in case of a failure, IaC makes your configuration resilient,
idempotant, easily reproducible and portable. Now Ansible is mostly used in the
professional Linux environment To deploy multiple machines
with similar configurations. But even if you only have one machine, it definitely helps to keep your
setup tidy and easily reproducible. Besides, it's a good skill to have if you
want to get a job in Linux administration, DevOps or similar fields. So unlike ricing i3 or installing Gentoo, you'll be learning an actually valuable
skill that might get you a good job later. So that's gonna be it for this long video, and to wrap it up I also have a
long list of people that want to shout out for helping me make this video possible, both directly and indirectly. David Stephens, for his Ansible NAS project Jeff Geerling, for his amazing YouTube
videos and his book "Ansible for DevOps" Jonathan Hanson for his SSH
port juggling implementation Alex Kretzschmar and Chris Fisher from the
Self Hosted Show for their amazing podcast Tyler Alterio for the mergerfs Ansible role Jake Howard and Alex Kretzschmar
for the snapraid Ansible role. And finally, the Self-Hosted discord server And as usual,huge thanks to my Patrons!
