Automating my Homelab with Ansible

a home lab is a special thing for nerds like me from a little router and a Raspberry Pi in A Rat's Nest of cables to a giant rack full of servers with screaming fans and petabytes of storage everyone's home lab has a story this is my story at least so far and I told it at ansible Fest 2022 in Chicago after you watch I'd love to hear about your home lab in the comments below and if you want to learn about ansible check out my book ansible for devops this is my home office it's also my little YouTube studio and it's in my basement so it's nice to finally be back at ansible Fest after three years in fact ansible Fest 2019 was the last in-person conference I went to and if you look behind me that lanyard is still the last one on my little lanyard hook so it'll be nice to finally have a new one to put up there and there are some big companies at ansible Fest like Microsoft Amazon and Google like you see here they all have giant data centers and employ thousands of sysadmins but me I represent Midwestern Mac LLC a tiny little company headquartered in St Louis and I don't have a data center I have a rack so specifically I have this rack this is in my basement and yes it's sitting right next to my HVAC system with a condensate pump and humidifier right next to it that means water lines and on the other side of it is the basement floor drain which has backed up a few times that's why I bought casters from Iraq if you want more where that comes from check out my rack setup video on the gearling engineering YouTube channel my dad and I have a ton of fun over there working on some projects together maybe we have a little too much fun sometimes but I put my rack on casters because if I get a sewage backup I'd rather sacrifice the casters than the rack and thousands of dollars of rack equipment and wouldn't you know they already came in handy once earlier this year my AC condensate pump actually failed and the water alarm I setup also failed and I looked on in horror the day I came downstairs to see this luckily the water was all just going under the rack and wasn't coming out from the rack none of my servers wet themselves yet so anyway my home lab today has three main purposes first it supports the video production needs of this YouTube channel second it ensures my kids have five nines of uptime for Doritos that's what they call the media app they use to watch all the episodes of Berenstain Bears and all the other kids shows and movies I ripped onto our Nas and third it lets me have fun cosplaying as a CIS admin hey I even have a shirt for that check it out along with other favorites like my it was DNS shirt over on redshirtjeff.com but let's get to the meat of this presentation the home lab every home lab starts with humble beginnings you move into a new place what's the first thing you're going to do plug in a router get the internet water food a bed all that stuff can wait you need to get the internet going ASAP most of us start things out with a router sometimes it's built into the modem that your ISP sends you sometimes you have a separate modem and plug it into another router like I do here with my Asus Wi-Fi router but you want to start doing more you want to store stuff on the network and maybe run some apps like pie hole to block ads or home VPN or a media sharing Library so you add in a Nas but then you realize running a VPN Plex pie hole backup software and home automation all on a single consumer Nas is a bit fragile and now you're looking at Craigslist for use server Hardware UPS's and racks and your significant other starts wondering why the electric bill shut up so quickly last month so that brings me to my home lab the first version you see here was just a piece of plywood with a modem and a network switch I actually had my Wi-Fi router plugged in upstairs for better signal so you don't see it in the picture here but at least I had a little ups and an empty punch block and not sure why I put that on there I never even had a phone line at my house but I guess it looked cool but I moved everything on this board over to another part of my basement when we remodeled and I also added a 3G cell for better cell reception in the basement when I started working down there and a Raspberry Pi to run some local services and the website pidramble.com but things started getting a little bit more serious when I bought my first rack and when I bought that thing I didn't think I'd ever fill it up but I did I had the space so I grabbed a free UPS from a local business that was moving and that's when I found out that the rails for it and other servers only work in server racks not a smaller Network or AV rack like the one I bought so I modded this rack the grinder a circular saw and a tap and die set were involved but dang it I cut that UPS installed and then I filled up the rest of the thing I added on a 2.5 gig switch and a 10 gig switch added some more Raspberry Pi's I started caring about Cable Management I added a second Nest so I could have two local copies of all my data and it was quickly running out of Rackspace to the point I didn't even have room for that new server that you see they're laying on the ground so I invited my dad over and he helped me upgrade to this bigger rack the one that I'm using today and before I talk about automation I'll walk you through the entire rack starting from the top I have a patch panel with cable drops from around the house and a couple from other parts of the rack for equipment that's down below then I have a shelf for random things I can't recommend like my Philips Hue Hub my cable modem and my home assistant yellow I put those up near the top since they have radios in them and the open rack with a wood top lets them connect to iot devices without any issues then there's my 10 gig switch from microtic the core of my network goes through here so all my storage and wired computers can transfer data around one gigabyte per second in the middle I have a Poe plus switch from qnap that also has a couple 10 gig ports and I use it to power some Raspberry Pi's in my IP cameras under that is my Pi rack where I run most of my networking applications and I'll get to those in a bit then there's a drawer and I'm not going to show you the inside just like I won't open up the neat patch and show you the Rat's Nest of cables in there either under all that is my edit Nas running trueness with 40 terabytes of raw SSD storage then if you think that's Overkill just wait a second below that are my two storage nasas one with six bays and the other with four the one on the right is just a replica of the one on the left minus a little bit of data that I don't have to have a second copy of so I have at least two complete copies of all the important data in my house between those two nasas I have 200 terabytes of raw capacity and then there's the stornator this thing is from 45 drives and I actually built what I call the pedophile a single Raspberry Pi addressing 60 hard drives for a full petabyte through four broadcom hbas I did a whole video on that called the petabyte pie project so I'm not going to get into it here but if you look closely you can see the fans aren't actually spinning I'm actually running that server yet since I'm going to revamp my local storage setup later this year and I'll be dealing with that in a video so make sure you're subscribed under that at the bottom is the ups that I got for free from a local business that had closed down I actually bought a used net network card for it off eBay so I can hook it up to my network I'm integrating it into home assistant and using network UPS tools to integrate it into my other server's power management I don't have all that working quite how I like it yet but I will soon so if you subscribe you'll probably hear more about that setup next year and here in my office Studio where I'm recording I have a second rack where I have another 10 gig switch my main workstation which is a rack mounted Mac Studio and my audio processing hardware and just to be complete here's the back of the main rack where I have a pdu mounted to distribute power to all my devices and here's the back of my office rack I'm not going to win any awards for cable management but it's functional so as far as the data center goes this is nothing but for a home lab there's actually a decent amount of gear and trying to manage every device by hand in my spare time would be impossible I mean you could just throw equipment in the rack and let it sit until it's hacked or dies but if you want to have a secure useful home lab you have to do stuff you have to configure everything you have to have have backups you have to test the backups you have to have a patching workflow and you have to be able to replace and upgrade hardware and I choose to do as much as possible with ansible one of the most important things and something that you can set up in your own home lab pretty quickly is to have some sort of Maintenance and upgrade Playbook I have a home lab project that has this ansible inventory file it has all the connection details for all the Linux FreeBSD Raspberry Pi and iot devices on my network with that inventory I can run things like this upgrade Playbook to quickly upgrade and reboot all the servers for maintenance but also if there's ever any urgent patch like for heart bleed it's still a work in progress and I'm also interested in getting my two windows machines under management but I'll be publishing all this on GitHub soon the home lab project also has a Playbook that sets up the displays and Hardware on all the pies in my pyrec since I maintain a lot of open source code for each of the main parts of my home web I have individual device configurations split out into their own playbooks and GitHub repos but if I were doing this just for myself I would probably store all the playbooks in one repo the first pie the one you see here runs pie hole pie hole provides local DNS and tracker blocking on my network mitt also is running Prometheus and grafana and I use it to monitor things in my house like internet performance and some environmental sensors all that is running in a set of Docker containers set up by ansible the configurations are all ansible templates and this makes it so I can maintain this open source internet Pi Project and this project has hundreds of users and it's really cool to see how other people have expanded it and monitor even more things in their home labs and just before ansible fast I added on monitoring for my actual router so I can measure performance both at the edge and inside my network it's a little tricky with a consumer router but I was able to get it working and I have a blog post where I go into detail how I did it I've been burned by a lack of visibility into my production infrastructure so many times before and it's really helpful having this home lab environment where I can explore different tools and visualize things in different ways without risking those production systems I mean some some people would look at that and say you're just taking your work home with you but well yeah I guess maybe you could look at it that way but that doesn't make it any less fun and rewarding for the people like me who love dealing with their home Labs I'm also running a few other pies in the rack like a pie running my backup scripts another one running Pi VPN and a couple more running a Drupal website and an NVR for Network video recording for my security cameras all those pies are managed by ansible playbooks and I'll have the links down in the description as well I also have a few Scrappy playbooks for some of the other devices like the home assistant yellow here there's actually a Raspberry Pi inside but since it's managed by home assistant OS I haven't found a good way to fully integrate it into my other automation which is a little ironic because that pie is actually controlling all my home automation but since it's just running Linux under the hood I'll find a way I also have two Macs I use for my main workstations there's this Mac Studio in my office and also a MacBook Air I take on the road some people just yellow and use Cloud sync and Cloud apps to make multiple computers work for them but me I use ansible so I have this Playbook that I use to fully automate the configuration of both Macs even including things like the position of all my dock items I won't dig deep into it because I actually already did that at last year's ansible Fest and there's also a whole video about it linked below but there's a lot left you might have noticed there are a lot more devices in my tourex especially all the network switches the nasas and my router and speaking of the router earlier I showed you this picture of my internet connection and you're probably like oh Jeff has gigabit internet nice well no I didn't show you the rest of that graph I mean 40 megabytes of upload isn't nothing but it can be painful when I'm trying to ship an off-site backup of a 200 gig video project every week and you can also see a bunch of dips on there one of those was when I was running a backup but those other three are full outages I get at least one or two outages a month so I've been working on a new router setup with a 4G fallback right now I'm waffling over whether to build it with plain Linux or use open wrt we'll see but either way my Pi router project is also up on GitHub and right now there isn't any automation around it but hopefully that'll be wrapped up soon in another ansible home lab Playbook unfortunately there are some devices like my managed switches that don't have any ansible modules so I'm still trying to figure out how to manage them better right now I have to go into the UI in my browser and that's just not fun for someone who loves automation there is a little support for router OS for my main switches but nothing at all for qnap for my Poe switch if you get network equipment from the better supported vendors you can automate things a lot easier but wrapping things up a bit people who manage home Labs do it for a lot of different reasons but I think for most of us whether the home lab looks like a shrine to Computing or something that would be home in a trash dump we just love the thrill of learning and cosplaying is a sysadmin you get full control over an environment where mistakes can be made and should be made on a small scale so you can learn better ways of dealing with them on larger scales and just like with the hardware the same goes for Automation in some cases you can manage every single bit of your home lab using ansible automation from your network router to iot devices most things have an ansible module or collection ready to use if you can do it on the command line chances are you can do it with ansible my home lab is growing every day and if you want to use any of my Automation in your own home lab check out my projects on GitHub and there are links below until next time I'm Jeff gearling

Channel: Jeff Geerling

Views: 211,912

Keywords: ansible, automation, chicago, illinois, it, aap, tower, awx, homelab, servers, sysadmin, cosplaying, cosplay, tshirt, red shirt jeff, storinator, xl60, ups, apc, nas, network, switch, router, management, patching, workflow, devops, sysops, devsecops, training, computers, mikrotik, patch, ethernet, nic, nut, tools, networking, mac studio, rackmount, rack, gear, startech.com, neat patch, qnap, asustor, home assistant, iot, power

Id: yoFTL0Zm3tw

Length: 13min 18sec (798 seconds)

Published: Wed Oct 19 2022

Reddit Comments

[u/bigDottee]

Gets reported for not homelab related...

but bruh... this is literally about a home lab.

This post has mod approval.

[u/geerlingguy]

Since I know not everyone loves sitting through a video to grab some little nuggets like what is mentioned in the video, here's my blog post with the relevant links.

I may do a more formal 'homelab tour' at some point, but if you want to see just the actual rack, here's a picture of that: https://imgur.com/gallery/gLUBs2J

[Edit: Forgot to mention, I'm giving out Ansible for DevOps free today (feel free to share the link if anyone's interested!) to celebrate #AnsibleFest: https://www.jeffgeerling.com/ansible-2022 ]

[u/Badluckredditor]

Love your stuff Jeff.

I'm convinced that the REAL Jeff is actually "Red Shirt Jeff", and that the "Good Jeff" is the adopted personality. You have everyone else fooled, but I've got my eye on you...

[u/somzeFiree]

Is there a blog on topic ‘How to start with home lab’ ( I know it is kinda too general question but you know some basics for setting up your own network, servers, etc…

[u/slashbackslash]

Jeff,

As always, it's a pleasure to watch your content. You've given me some suggestions:

- UPS

Network attached, of course. I need to do something similar with HomeAssistant and Grafana, as I'd love to get more info on power draw and uptime.

​

- Ansible Update Scripts

This one I desperately need. I have too many VM's and machines running out-of-date versions of Linux. Luckily, I don't open any ports on my network. But that doesn't mean there aren't other entry vectors, of course!

Finally, I loved the bits about "why" we homelab. My homelab has been such a learning experience, and I really hope others find as much enjoyment from theirs as I do from mine.

Also, you have a cool Dad.

[u/shishcat8214]

mom i'm on TV

[u/TommyBoyChicago]

I thought the title of this video was misleading. It was a lot more of a tour/history of your home lab and next to nothing on Ansible. Was very disappointed by the time I got to the end and had learned nothing.

[u/lucho4u]

Your session was packed full before it even started and a lot of us were hoping to attend missed out, thanks for posting the video and always creating great content!