Taking a look at RAIDZ expansion

Linn County Republicans
18 followers
8 Video Views·Dec 4, 2024
In this video, I explore the process of expanding ZFS pools by adding a single drive to a RAIDZ vdev. This method simplifies the expansion of ZFS pools, making ZFS expansion easier than before. I look into the steps involved, the underlying mechanics, and the performance speeds during the expansion.
Here is a PDF that goes over how ZFS expansion works under the hood: https://openzfs.org/w/images/5/5e/RAIDZ_Expansion_2023.pdf
Today, I'm going to take a look at a feature that I think a
lot of you have been waiting for,
and that's ZFS RAID-Z expansion.
You can finally take an existing RAID-Z array of, let's
say, four drives and just add a fifth drive
and get the additional capacity of
that fifth drive added to your pool.
Overall, it's a pretty painless process, but today I'm
going to take a look at how it actually performs
and works on my test benches, go through
the interface of how it can be performed,
do a little bit of performance testing to see how long it
takes to add that additional drive,
and then do a little bit of an explanation
of what's going on under the hood in ZFS.
ZFS, since its creation, has had a
limitation that once you make a RAID-Z,
which is roughly equivalent to a RAID 5 or RAID 6 array of,
let's say, four drives, it's fixed.
You can't just add a fifth or sixth drive to the array. You
can't convert its type. It's just there.
You could delete it and make a new one,
but for a lot of home server users like me,
buying another set of drives to store your data and copy it
to is a lot of additional costs that
many people don't want to have. And while it may be
recommended to people to just buy the bigger
array that they might think they want to need, again, with
home users with limited budgets,
paying for more drives and more ports to use those drives
can be a lot of additional money.
So being able to just add an additional drive later on is a
really nice feature for many,
especially home and low budget users. And
it's really nice to see that they finally
released this feature and it's getting into stable releases
now. Now, a few caveats before
starting this process is that your array needs to be in a
nice stable online state before doing
anything. If you have drives that are offline or having
issues, you need to replace or fix those
problems before you can start doing any RAID-Z expansion.
The other thing is you likely need to
update your ZFS software and upgrade your pool to be able
to support RAID-Z expansion. ZFS expansion
is basically only supported in the latest releases as of
now, like TrueNAS 24.10 Electric Eel. So you
need to update your TrueNAS install and then go and run the
Z-pool, upgrade your pool name to add
those additional features to the pool. And that'll upgrade
the metadata to the latest version so that
you can perform the expansion process. This shouldn't
result in any issues or any downtime,
but the one disadvantage it means is now that your ZFS pool
is now a newer version, it can't be
mounted on older systems. So if you want to recover the
data or mount it on another system, it has to
be running that same version of ZFS on the other system or
a newer release to be able to take
advantage to it. And now that you have an upgraded ZFS pool
and everything is online and ready to go,
you can use the zpool attach drive in the command line to
just add additional drives to the pool.
So the command would look like Z-pool attach pool name, the
RAID-Z name. So if you run something
like ZFS status, you'll see something like RAID-Z 1-0-1 for
example, and so on depending on how large
your pool is. So that's telling it what actual RAID-Z VDev
the drive should be added to. And then you
run the device name that you want to add to the pool. So
for example, something like slash dev slash
disk dash by ID, and then the full device of the name is
typically the recommended path to add the
drive to the pool. I'd also typically suggest running this
with dash in first, which means dry run.
So to show it, it'll show you what the pool will look like
before running the actual command. So
you don't accidentally add it to the wrong RAID-Z or do
something weird and can kind of double check
your work before it's actually written to the pool. And
then once it's written, it's going to
start moving the data around in the pool. So you have more
additional space on all of the drives.
If you're using something like TrueNAS, you can go under the
pools, see a list of RAID-Zs on it,
click on one of those, say extend, and then select from one
of the unused drives on your system.
I did this just fine in TrueNAS using the GUI and the
command line, but it seems like there's a bug
in the current version of TrueNAS where it will just be
stuck on the prompt of adding, but it
shows it as added in the command prompt. And if I reload
the webpage, it works just fine with the
device added. Now, when it comes to the actual performance
amount of time taken to add the drive,
it was a lot longer than I'd expect using purely sequential
data copies. For all of my testing,
I was using WD-RED four terabyte drives, which run at a
maximum of about 100 to 200 megabytes
per second in sequential workloads. And I was seeing about
40 megabytes using the extend command
adding more drives. And as I'll explain later, this is
likely due to the fact that ZFS is using
a lot of non-sequential workloads when it adds more drives,
causing mechanical hard drives to do
this much slower than made otherwise be capable of in a
truly sequential workflow. So in my testing,
I took a four drive RAID Z1 using those four terabyte
WD Red drives with about 540 gigabytes
of data allocated and added a fifth drive. That took about
one hour and 20 minutes. I then added
a sixth drive. It was a little bit faster at one hour 15
minutes. Added a seventh drive at about one
hour 10 minutes. And adding an eighth drive took about one
hour total. So those speed estimates
are definitely a bit slower than what the drives are
capable of at the max sequential speeds,
but far from slow enough to cause any substantial issues.
Unfortunately though, this means if you
are using drives that have performance issues with a lot of
random IO like SMR drives, this is likely
going to cause major performance issues and will take much
longer. I also want to note that while
I didn't have any issues performing ZFS RAIDZ expanding, I
generally like to recommend to make
a backup of your system first or have a plan for the worst
case scenario or something like
file system corruption. Anytime you're doing a RAID
expansion or major changes to how your disks
are laid out, just in case. I like to take it the safe side
so I'm ready for the worst case scenario
even though that is going to likely be very
unlikely with any of these expansion jobs.
One disadvantage with how RAIDZ expansion works is that it
doesn't rewrite the parity for its
existing data blocks. It just spreads out those data and
parity blocks across the additional drive.
And what this means in practice is you will have a little
bit less usable space if you take a RAID Z
pool and extend it than if you were to start with that
original larger RAID Z pool or use something
like RAID 5 and RAID 6 expansion which will typically give
you the full available amount of space.
As a quick example, if I had four drives of let's say one
terabyte each that were completely full,
I would have three terabytes of data if an RAID Z1
configuration and one terabyte of parity. If I
then decided to go add an additional one terabyte drive to
make it a five drive RAID Z1 that has been
expanded, I will still have three terabytes of data and one
terabyte of parity. It's just now
going to be spread across five drives instead of four. So
now instead of each drive being full with
one terabyte of data each drive will now have 0.8 terabytes
of data used. In comparison, if I was to
rewrite all that data or use a more traditional RAID 5 or
RAID 6 expansion that would use about 0.75
terabytes of drive data on each drive. So you are using
losing a little bit of space on existing data.
One workaround for this is if you move data away from it
and then copy new data to that now expanded
RAID Z pool, it will copy it to it using the full width and
taking the most advantage of the drive
space you have. There are a lot of cool white papers online
of how Open ZFS has implemented it,
but I'm going to try to explain it a little bit on my
Blackboard to get a better feel of what's going
on under the hood. So in my example here, I have a very
small array of let's say four drives here,
each of which has drive data like one, two, and three. And
they have the parity trunks here and
the rest of these A, B, and C trunks are going to be data.
So in this case, I am going to go and add a
fifth drive right here. A more traditional RAID setup would
effectively move the parity to a new
drive and then find some other data to make this 1D of
data. But RAID Z expansion doesn't do that.
It keeps this 1P exactly the same and then it moves 2A down
here and nudges all of this data
forward. So like 2B would go here and then 2P would get
moved forward and then 2D is going to
stay here and move forward. So now there's two free chonks
on these last two drives and 3A
and 3B will get moved down, giving you more additional
space on those additional drives
as everything gets nudged over. And what this means in
practice is my three available chunks
of data instead of being on that new drive are now spread
out across all the drives so I can write
more data to all of it. Check out the white paper. It's a
nice read if you want to find out more of
what's going on under the hood here. But I think it's
interesting to kind of know what the data
solutions are doing under the hood to find out why they
have these limitations of slightly less
usable space. And I did my testing expanding a four drive
RAID Z1 all the way up to eight drives
and I saw exactly my expected amount of data and parity
utilization on all of those drives.
In general I would say this is not a huge
limitation for most use cases as it's not
that much more space with most reasonable
configurations of RAID Z, RAID Z2 and RAID Z3.
One other slight limitation that there is in ZFS is that it
isn't really rewriting the metadata but
it's more so making a pointer to the new locations after
it's expanded the data. Now I didn't see any
performance issues in my four drives that has been expanded
three times into eight but it's a
technically not optimal setup in terms of how it has to
handle metadata but practically it probably
doesn't matter in most home situations. And I think this
brings me to the conclusion of this
video. I think it's really awesome to see ZFS adding any of
these new features and especially
RAID Z expansion as it's one thing I've heard multiple
people say I don't want to use ZFS because
of it and now it's here. So if you want if you had the
previously debate between using ZFS features
and having the ability to expand parity array now you no
longer have to worry about that.
There are some technical limitations but I don't think
they're a big issue for most home users
where being able to just add an additional drive is going
to be a much bigger feature add
than the slight possible limitations. If I have the budget
and resources I'd still get the big
array and just immediately start with it if I can but I
know that's not a thing for everyone and
it's great to see they've added software here. Let me know
if you've expanded your array already
or if you're planning to in the future or any other
comments or thoughts you have with RAID Z
expansion in the comments below. I'm looking forward to
taking a read to them and thanks for watching this video.
Also I'm giving the YouTube membership thing a try so check
the join button down below if you
want to take a look at some BTS content help me decide on
what future videos are going to be about
and watch some of these new videos a little bit early.
Thanks for watching and supporting the channel.
