TrueNAS is an incredibly powerful open source NAS system that lets you host and share files on your network. It's targeted at enterprises, but because it's open source and runs on commodity hardware, it's a favorite among tech enthusiasts and home labbers alike. And because it's targeting enterprises, it ships with some features that might be confusing to some. I know this firsthand because I only recently figured out how a lot of this fits together. And that's when I decided to put together a guide on everything I do after setting up TrueNAS. This will pick up right after the installation and will walk through some of the most common questions people have when configuring it. We'll also be focusing on performance. Things like how should I configure my ZFS pool? What are the benefits of caching and how do I enable it? Why are file transfers so slow? What's ZIL, SLOG, and ARC and other concepts to help you understand your TrueNAS installation a little bit better so you can squeeze the most performance out of it? This will be a reasonable guide that anyone can use, but especially people at home and home labbers. Some of the recommendations I'm going to make today I might paint with a broad stroke, so feel free to veer from this where you see fit. So let's get into it. First we're going to talk about pool configuration and our disk layout to help us optimize the data on this pool. Now RAIDZ has many different RAID types. You have RAID 1, RAID 2, RAID Z1, RAID Z2, RAID Z3, so on and so forth. And most people are going to say, choose the RAID type that's best for you. But if you don't know your data access patterns and you don't know some of the limitations on expanding later, you may not know what the best one for you is. I'll talk about the best one for me and that was mirrored Vdevs. So what does that mean? That means I have pairs of mirrors. So if you see here in my devices I have a pair of mirrors here. It has two devices that have 14 terabytes each and the same goes here. Another pair of two devices and a pair of two devices and another pair of two devices and some other devices you see down there that we'll talk about. And there are upsides and downsides to choosing mirrored Vdevs. Well some of the upsides are it provides redundancy, you get some increased performance, and most importantly, at least for me, expansion is a lot easier in the future. When using mirrored Vdevs in order to expand my total capacity, I only need to add two more drives. Previously I had a six-disc RAIDZ2 and in order to expand that pool or grow my pool, I had to buy six more disks that were higher capacity. But when moving to mirrors, in order to expand my total capacity, I only need to add two more drives and then I can expand the pool. Now there are some downsides to mirrored Vdevs. Right performance can be decreased because when you write data, you actually write to all mirrors. Another downside is if you lose two of the drives in a pair, you lose all of your data. Which can be pretty likely if you have two mirrored Vdevs, but the more mirrored Vdevs you have, the less likely that is to happen. Plus you should have backups, right? We'll talk about that too. And another downside to mirrors is you lose half of your total capacity. That means if you mirror two 8 terabyte drives instead of 16 terabytes of usable space, you're only going to get 8 terabytes of usable space. But if you can stomach the 50% capacity loss, I think that mirrored V-devs is definitely the way to go, especially for home and homelab. It's a lot cheaper to expand by two than it is by six. And now on to datasets. There are a few things I configured with datasets. Most of them I apply at the pool level and let all of the individual datasets inherit those settings. For example, is encryption. Now I turn encryption on. I don't see any reason why you wouldn't turn encryption on. If you're worried about the small performance overhead of encrypting these files, that's just a drop in the bucket for modern processors. So definitely turn this on on your pool. Unfortunately, you can't turn it on later. So when you're setting up your pool, be sure you turn it on. So if we look at the details for this pool, you can see that I have sync set to standard. Now that's something we'll talk about here in a little bit. Compression level is LZ4. Now I would keep this at the default. LZ4 is a great compression algorithm. It's a good balance between performance and space. So I would keep it there unless you have some special need for something else. Enable at time. Now this is off by default. I would probably keep it off, but this will update the access date according to the date if you turn this on. In the last setting, ZFS deduplication. If you're asking if you need this, you probably don't need to turn it on. It's a pretty advanced feature that I've never needed, so I've never turned it on. So for sync, this is a setting that tells Trunas whether or not to wait for confirmations after it writes a file. So for example, with sync, it's going to wait for the acknowledgement before it continues writing. And on the converse, disabled or asynchronous or no acknowledgement at all, I guess is a better way to put it. It's just going to write the files continuously without acknowledgement. Now you can see how that can get you into trouble. On sync, while slower, you're always guaranteed that the file is written. Whereas on disabled, it's never guaranteed that the file is written, and it's written directly to the ARC. Basically bypassing the ZIL. So what is the ZIL? The ZIL is the ZFS intent log. And that's just a place in your pool where it writes these files. And you can see how that could slow you down a bit. Your pool's busy doing other things. All of a sudden it's going to get a lot of writes. You said it was a synchronous write, so it has to write to the ZIL and stop doing what it was doing, or focus on that a little bit more. Write them there, and then eventually write them to the pool. And if you disable this altogether, you're writing it directly to ARC or to RAM, which eventually writes then to disk. Much faster, not as safe. And then there's standard, which is a pretty good balance, which says, "Hey, the client that's writing should tell me whether it's a synchronous write or not." That's typically where I set mine, not only because it's the default, because it makes the most sense to me. Now this can slow down some of your writes, but we'll talk about how to optimize that in a little bit. So we talked about ARC a little bit, but what is ARC? ARC is Adaptive Replacement Cache, and that's everything you see here in this chart. This is all of the RAM that it's going to use to store files so it doesn't have to retrieve those from disk if they're requested again. And there's an algorithm to determine which files it should keep in memory once you access often, and ones it should evict, like ones you don't access often. And this ARC is the fastest way to access these files. So if we can store more in the ARC, it doesn't have to go to disk to get it, it can serve it directly from ARC. And then your pool can keep doing the thing it was doing before. So multiple benefits. And so a good rule of thumb that I've heard is to have one gigabyte of RAM for every one terabyte of disk. Now that's just a loose rule to gauge how much you need. The more the better. The more ARC in RAM you have, the more you can store in there, and the faster you can serve that data on reads. Now there is one quirk if you're using Trunab scale, but I guess it's a quirk with all Linux and ZFS. It's that Linux will only reserve 50% of your RAM that you have on the system available for ARC. Now there's a command that I learned from Tom Lawrence that you could run to use more than 50% of your RAM. You could run it once to set it in real time, and then you create a startup script to run it on each boot. If you want to do that, I'll have links to my documentation below, and I'll even link to Tom Lawrence's video where I learned that. But a general rule of thumb for tuning your ARC is to let it run for a couple of days, see how much the services are taking up of your RAM, and then use the rest of it. And then leave a little bit for padding. As you can see I did. I could probably recover some of that eight gigs. And then leave some for virtual machines if you do that. Anyways, I'll have a link below that has the documentation talking about all of that. So what's the best way to improve read speeds? Well let's add more ARC or more RAM. That is hands down going to give you the best performance. But what if you can't add any more RAM? Or like me, I work with really large files, 20, 30, 40 gigabyte files, to where the next time I access them they're not in RAM anymore. That's where L2ARC comes in. So an L2ARC is just what it sounds like. It's a level 2 ARC or a level 2 cache. And you'll want to use fast disks for this L2 cache. At least faster than what you have in your pool. In this L2ARC cache drive you can add later to your pool. As you can see I have this NVMe drive for my pool. And generally speaking with cache drives you want to strike these cache drives. They don't need to be mirrored and if they die your pool will continue to work. As you can see from these charts my cache drives aren't that busy. But when I start to edit videos where I'm pulling in 20, 30, 40, 50 gigabyte drives this starts to light up. So an L2ARC might not be for you. You might have enough RAM. But I found it helped smooth out my read speeds in some of these scenarios. So now that we know how to increase our read speeds, how do we increase our write speeds? Now earlier we talked about synchronous writes and how we could disable that if we wanted to. Although I don't suggest it. That means we still have some clients that might need to write synchronously. If you have a fast network like a 10 gigabit network you might not be getting the write speeds that you expect. Now how could that happen? So for example if you have one hard drive or one mirrored pair in your NAS it can only read and write at around 150 megabytes per second. That's roughly a little bit over a gigabit per second. So by that math you would have to have about eight drives or eight pairs in order to write at 10 gigabits per second. Better check my math. Okay nine or ten. I'll round up. How did I get that math? 150 megabytes per second times eight, eight bits in a byte, is 1200 megabits which is 1.2 gigabit and then multiply that by the number of spindles which is eight or nine spindles or ten to get you 10 gigabit. Now that's something that a lot of people don't think about. I didn't think about it when I first built my NAS. I thought hey I have six hard drives. Of course I'm going to be able to write at 10 gigabit per second. But as I found out for synchronous writes you can't because those drives can't write fast enough. So this is where a slog comes in. So there's zil and there's slog. The way that I understand this is zil, your zfs and tent log, is called a zil when it lives on your pool. But then if you move that zil off in that pool into a separate drive or a separate pair of drives it then becomes a slog or a separate log intent. I know semantics but when it's outside it's a slog. When it's on the pool it's a zil. It's the way I understand it. But anyways moving that out of the pool and then putting that on faster SSD drives can easily get you faster write speeds if you're noticing you can't write at 10 gigabit per second or whatever the network throughput is you're trying to target. And that's what I ended up doing here. You can see in my log I have my slog which is two NVMe drives that live outside of the main pool. And now this is my zfs and tent log. When I write things that are synchronous it can write them to the super fast SSD freeing up the main pool to keep doing what it was doing. And then write those at a later time versus the opposite which is write them onto the main pool in a section for zil and then write them again to the main pool. It's like double writes and so you're kind of locking up some IOPS doing that. So if you move that zil to a slog to a mirrored pair that is SSDs you then free up those IOPS and you get really fast IOPS for your write. Make sense? So one thing about doing this is that you should use a mirror. Why am I using a mirror? Well if this is your zfs and tent log this is where your writes are kind of batched up before they get written to the pool. Now if it's not a mirror and it's just a stripe or one drive you could imagine that if that drive dies you could suffer from data loss. So I've mirrored mine so that if one dies I can swap it out and I don't suffer from many data loss. Since we talked about networking a little bit earlier as we talked about throughput we should visit the network section to make sure we're getting the best throughput for our network interface. And if you look at my network interfaces you can see I have two physical interfaces. These are two 10 gig interfaces right here. These are configured in a bond so I have a linux bond to bond these two together and they're using LACP or the link aggregation protocol. So LACP is going to bond these two network interfaces together and give me two 10 gig pipes. So it's not going to give me one big 20 gig pipe it's going to give me two 10 gig pipes so that I can send traffic through kind of like that like a like a highway. There's two different lanes on the highway as I was told in the past. So why do I lag mine isn't 10 gig enough? Kind of. It depends. I like to think about this as doubling the fastest transmit I could possibly do. So for me that's one 10 gig pipe. But if I'm transferring a file at 10 gigabit and I only have one 10 gigabit NIC that leaves very little bandwidth for anyone else on the network. Especially if my backups start running. So I look at it like this. Hey two clients on my network can push or pull to my NAS at 10 gigabits per second. And again that's going to depend on your hardware obviously on your server but it's also going to depend on your switch to make sure that you can bond these NICs. And it also gives me some redundancy too. So I would recommend putting in two NICs to your server even if they're only one or 2.5 gigabit. But remember that network speed and disk speed go hand in hand. Now what do I mean by that? Well remember earlier when we were talking about 10 gigabit and how we wanted to increase our write throughput and that's why we added the slog device? Well that's a perfect example of how our network was faster really than the IOPS we had in that machine. The cache drive got us past that. But we could also have the opposite problem too. Think if we had all SSDs. Now we have all of these IOPS locked up in the machine where we don't have a network fast enough to push them. So you'll have to do a little bit of planning to make sure your network is fast enough to get the IOPS out of your server. And if not you'll either have to upgrade your network or leave them on the table. And vice versa if you have a faster network you'll have to make sure that you can bring your IOPS up enough to saturate that network. Just food for thought. So another consideration is VLAN routing. If your client's on one VLAN and your data is on another VLAN you'll have to make sure that your switch or your router can route those packets fast enough. So there are ways around that. One is buy better hardware but the other way is to create a dedicated network dedicated to data only. But then I feel like you're losing a little bit of control because then you're no longer routing your traffic through your router and applying those ACLs. But that's a trade-off. You're gonna have to decide if it's worth it. Another thing I do after configuring my pools is setting up snapshots. Now snapshots will take obviously snapshots of your data set within ZFS. Now they're really fast and they're really awesome because they're only the diff from the previous snapshot. So if you added a lot of data the first snapshot will be pretty big because it's the diff between nothing and something. But then subsequent snapshots will be fairly small because it'll be the diff between something and the something else that you just added. So most of my snapshots I set up the same way. I choose my data set. I choose a lifetime for them. Most of the time I choose two weeks. You can expand this if you want depending on how much disk space you have. And then I'm sure to set it recursive so it will recurse all of those folders within that data set. And then I don't allow it to take empty snapshots. I disable that mainly because I don't want to scroll through a whole bunch of empty snapshots. If I try to restore something there's no point of them being there. Set my schedule and then obviously enable it and save. Then you have these snapshots where you can roll things back or apply them to a new data set if you need to restore a whole bunch of this data somewhere else. And then as far as backups go you should probably back up this data somewhere else. I know that's tough to do especially in home or small businesses because you have to have yet another server of the same size to be able to do that. So in a perfect world I would have another server that had about the same amount of disk space and I would rsync or copy these snapshots over to it on a task. Now we know that's hard to do on a budget and it's hard for me to recommend this when I'm not doing this myself. But my plan in the future is to build another pool and then synchronize my snapshots over to that pool. Another thing that I set up every time I'm setting up a TrueNet server is to make sure that my alerts are working. The alerts can be the difference between a failing pool and a dead pool. So you want to make sure that your alerts set up so you get notified if a pool gets degraded or a drive dies so that you can replace that as soon as possible. So it's super easy to do but easy to overlook. And the last thing I recommend doing is just getting a UPS. I know this seems obvious to some but I highly recommend getting a UPS as soon as possible to put your NAS on. Doesn't have to be much. You can do it on a budget but just enough to shut it down safely if the power is down for more than a couple of minutes. A UPS along with your ZIL or maybe your slog can help prevent data loss if you lose power. Now over the last couple of weeks while rebuilding my TrueNet server I learned so much about ZIL, about slog, about ARC, about networking throughput, and I hope you learned something too. And remember if you found anything in this video helpful don't forget to like and subscribe. Thanks for watching.
