Account for missing space as reported by ZFS list
up vote
3
down vote
favorite
This is a question regarding the available space as reported by zfs list that does not meet user expectations. I have looked at similar questions, but I still don't understand the zfs list output. Let me explain why:
I have a simple setup with just 4x WD30EFRX disks. Each disk is reported by Linux to have the following capacity:
5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
That is, each drive has a capacity of 3000592982016 bytes. According to the math, the aggregated raw space for the four drives is 3000592982016*4 = 12002371928064 bytes (10.9 TiB), and zpool list seems to agree:
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
nas 10.9T 1.15M 10.9T - 0% 0% 1.00x ONLINE -
So good so far.
Now, I put those 4 drives into a RAIDZ1, meaning 25% of the total capacity should be used for parity. I would expect zfs list to report something close to 2.72 TiB *3 = 8.16 TiB. Instead, I'm getting only 7.65 TiB.
zfs list nas
NAME USED AVAIL REFER MOUNTPOINT
nas 558K 7.65T 140K /nas
zfs list nas -p
NAME USED AVAIL REFER MOUNTPOINT
nas 571392 8416155617280 142848 /nas
I also expect ZFS to use up some space for metadata and its own internal data structures and whatnot, but I'm missing over half a Terabyte of data:
Available space according to zfs list -p is
(8416155617280/1024^3) GiB = 7838 GiB
Available space after parity (3 drives of 5860533168 512-byte sectors) should be close to
5860533168*512/(1024^3)*3 GiB = 8383 GiB
8383 GiB - 7838 GiB = 545 GiB
Over half a terabyte, close to 5% of the aggregated raw disk space, is unaccounted for. Where did all that space go?
Note: I'm running Debian Jessie, ZFS 0.6.5.2-2. I did check for the quota, reservation, refquota and refreservation properties, which are all set to none. I just created this pool and there is no data in it, and of course no snapshots.
Here's my full zfs properties:
zfs get all nas
NAME PROPERTY VALUE SOURCE
nas type filesystem -
nas creation Tue Feb 7 17:28 2017 -
nas used 558K -
nas available 7.65T -
nas referenced 140K -
nas compressratio 1.00x -
nas mounted yes -
nas quota none default
nas reservation none default
nas recordsize 128K default
nas mountpoint /nas default
nas sharenfs off default
nas checksum on default
nas compression off default
nas atime on default
nas devices on default
nas exec on default
nas setuid on default
nas readonly off default
nas zoned off default
nas snapdir hidden default
nas aclinherit restricted default
nas canmount on default
nas xattr on default
nas copies 1 default
nas version 5 -
nas utf8only off -
nas normalization none -
nas casesensitivity sensitive -
nas vscan off default
nas nbmand off default
nas sharesmb off default
nas refquota none default
nas refreservation none default
nas primarycache all default
nas secondarycache all default
nas usedbysnapshots 0 -
nas usedbydataset 140K -
nas usedbychildren 418K -
nas usedbyrefreservation 0 -
nas logbias latency default
nas dedup off default
nas mlslabel none default
nas sync standard default
nas refcompressratio 1.00x -
nas written 140K -
nas logicalused 120K -
nas logicalreferenced 40K -
nas filesystem_limit none default
nas snapshot_limit none default
nas filesystem_count none default
nas snapshot_count none default
nas snapdev hidden default
nas acltype off default
nas context none default
nas fscontext none default
nas defcontext none default
nas rootcontext none default
nas relatime on temporary
nas redundant_metadata all default
nas overlay off default
Thank you!
linux raid zfs nas freenas
add a comment |
up vote
3
down vote
favorite
This is a question regarding the available space as reported by zfs list that does not meet user expectations. I have looked at similar questions, but I still don't understand the zfs list output. Let me explain why:
I have a simple setup with just 4x WD30EFRX disks. Each disk is reported by Linux to have the following capacity:
5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
That is, each drive has a capacity of 3000592982016 bytes. According to the math, the aggregated raw space for the four drives is 3000592982016*4 = 12002371928064 bytes (10.9 TiB), and zpool list seems to agree:
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
nas 10.9T 1.15M 10.9T - 0% 0% 1.00x ONLINE -
So good so far.
Now, I put those 4 drives into a RAIDZ1, meaning 25% of the total capacity should be used for parity. I would expect zfs list to report something close to 2.72 TiB *3 = 8.16 TiB. Instead, I'm getting only 7.65 TiB.
zfs list nas
NAME USED AVAIL REFER MOUNTPOINT
nas 558K 7.65T 140K /nas
zfs list nas -p
NAME USED AVAIL REFER MOUNTPOINT
nas 571392 8416155617280 142848 /nas
I also expect ZFS to use up some space for metadata and its own internal data structures and whatnot, but I'm missing over half a Terabyte of data:
Available space according to zfs list -p is
(8416155617280/1024^3) GiB = 7838 GiB
Available space after parity (3 drives of 5860533168 512-byte sectors) should be close to
5860533168*512/(1024^3)*3 GiB = 8383 GiB
8383 GiB - 7838 GiB = 545 GiB
Over half a terabyte, close to 5% of the aggregated raw disk space, is unaccounted for. Where did all that space go?
Note: I'm running Debian Jessie, ZFS 0.6.5.2-2. I did check for the quota, reservation, refquota and refreservation properties, which are all set to none. I just created this pool and there is no data in it, and of course no snapshots.
Here's my full zfs properties:
zfs get all nas
NAME PROPERTY VALUE SOURCE
nas type filesystem -
nas creation Tue Feb 7 17:28 2017 -
nas used 558K -
nas available 7.65T -
nas referenced 140K -
nas compressratio 1.00x -
nas mounted yes -
nas quota none default
nas reservation none default
nas recordsize 128K default
nas mountpoint /nas default
nas sharenfs off default
nas checksum on default
nas compression off default
nas atime on default
nas devices on default
nas exec on default
nas setuid on default
nas readonly off default
nas zoned off default
nas snapdir hidden default
nas aclinherit restricted default
nas canmount on default
nas xattr on default
nas copies 1 default
nas version 5 -
nas utf8only off -
nas normalization none -
nas casesensitivity sensitive -
nas vscan off default
nas nbmand off default
nas sharesmb off default
nas refquota none default
nas refreservation none default
nas primarycache all default
nas secondarycache all default
nas usedbysnapshots 0 -
nas usedbydataset 140K -
nas usedbychildren 418K -
nas usedbyrefreservation 0 -
nas logbias latency default
nas dedup off default
nas mlslabel none default
nas sync standard default
nas refcompressratio 1.00x -
nas written 140K -
nas logicalused 120K -
nas logicalreferenced 40K -
nas filesystem_limit none default
nas snapshot_limit none default
nas filesystem_count none default
nas snapshot_count none default
nas snapdev hidden default
nas acltype off default
nas context none default
nas fscontext none default
nas defcontext none default
nas rootcontext none default
nas relatime on temporary
nas redundant_metadata all default
nas overlay off default
Thank you!
linux raid zfs nas freenas
1
What is yourashiftsetting? Have a look at louwrentius.com/… , I have not checked your numbers correctly, but the difference is in similar size range.
– user121391
Feb 8 '17 at 13:28
That's an interesting comment, I didn't know about ashift! My ashift is 12 even though my HDDs seem to have 512 byte sectors. I tried rebuilding the pool with -o ashift=9 and I get a little bit of space back (7.88 TiB for ashift=9). Previously it was 7.65 TiB for ashift=12, which is the default. Even for ashift=9, there is still over 300GB of unaccounted space.
– Pablo Llopis
Feb 8 '17 at 14:59
Correction: parted reportsSector size (logical/physical): 512B/4096B. Nevertheless, I'm skeptical zfs needs to allocate for so much metadata than I'm losing over 300GB.
– Pablo Llopis
Feb 8 '17 at 15:06
add a comment |
up vote
3
down vote
favorite
up vote
3
down vote
favorite
This is a question regarding the available space as reported by zfs list that does not meet user expectations. I have looked at similar questions, but I still don't understand the zfs list output. Let me explain why:
I have a simple setup with just 4x WD30EFRX disks. Each disk is reported by Linux to have the following capacity:
5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
That is, each drive has a capacity of 3000592982016 bytes. According to the math, the aggregated raw space for the four drives is 3000592982016*4 = 12002371928064 bytes (10.9 TiB), and zpool list seems to agree:
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
nas 10.9T 1.15M 10.9T - 0% 0% 1.00x ONLINE -
So good so far.
Now, I put those 4 drives into a RAIDZ1, meaning 25% of the total capacity should be used for parity. I would expect zfs list to report something close to 2.72 TiB *3 = 8.16 TiB. Instead, I'm getting only 7.65 TiB.
zfs list nas
NAME USED AVAIL REFER MOUNTPOINT
nas 558K 7.65T 140K /nas
zfs list nas -p
NAME USED AVAIL REFER MOUNTPOINT
nas 571392 8416155617280 142848 /nas
I also expect ZFS to use up some space for metadata and its own internal data structures and whatnot, but I'm missing over half a Terabyte of data:
Available space according to zfs list -p is
(8416155617280/1024^3) GiB = 7838 GiB
Available space after parity (3 drives of 5860533168 512-byte sectors) should be close to
5860533168*512/(1024^3)*3 GiB = 8383 GiB
8383 GiB - 7838 GiB = 545 GiB
Over half a terabyte, close to 5% of the aggregated raw disk space, is unaccounted for. Where did all that space go?
Note: I'm running Debian Jessie, ZFS 0.6.5.2-2. I did check for the quota, reservation, refquota and refreservation properties, which are all set to none. I just created this pool and there is no data in it, and of course no snapshots.
Here's my full zfs properties:
zfs get all nas
NAME PROPERTY VALUE SOURCE
nas type filesystem -
nas creation Tue Feb 7 17:28 2017 -
nas used 558K -
nas available 7.65T -
nas referenced 140K -
nas compressratio 1.00x -
nas mounted yes -
nas quota none default
nas reservation none default
nas recordsize 128K default
nas mountpoint /nas default
nas sharenfs off default
nas checksum on default
nas compression off default
nas atime on default
nas devices on default
nas exec on default
nas setuid on default
nas readonly off default
nas zoned off default
nas snapdir hidden default
nas aclinherit restricted default
nas canmount on default
nas xattr on default
nas copies 1 default
nas version 5 -
nas utf8only off -
nas normalization none -
nas casesensitivity sensitive -
nas vscan off default
nas nbmand off default
nas sharesmb off default
nas refquota none default
nas refreservation none default
nas primarycache all default
nas secondarycache all default
nas usedbysnapshots 0 -
nas usedbydataset 140K -
nas usedbychildren 418K -
nas usedbyrefreservation 0 -
nas logbias latency default
nas dedup off default
nas mlslabel none default
nas sync standard default
nas refcompressratio 1.00x -
nas written 140K -
nas logicalused 120K -
nas logicalreferenced 40K -
nas filesystem_limit none default
nas snapshot_limit none default
nas filesystem_count none default
nas snapshot_count none default
nas snapdev hidden default
nas acltype off default
nas context none default
nas fscontext none default
nas defcontext none default
nas rootcontext none default
nas relatime on temporary
nas redundant_metadata all default
nas overlay off default
Thank you!
linux raid zfs nas freenas
This is a question regarding the available space as reported by zfs list that does not meet user expectations. I have looked at similar questions, but I still don't understand the zfs list output. Let me explain why:
I have a simple setup with just 4x WD30EFRX disks. Each disk is reported by Linux to have the following capacity:
5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
That is, each drive has a capacity of 3000592982016 bytes. According to the math, the aggregated raw space for the four drives is 3000592982016*4 = 12002371928064 bytes (10.9 TiB), and zpool list seems to agree:
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
nas 10.9T 1.15M 10.9T - 0% 0% 1.00x ONLINE -
So good so far.
Now, I put those 4 drives into a RAIDZ1, meaning 25% of the total capacity should be used for parity. I would expect zfs list to report something close to 2.72 TiB *3 = 8.16 TiB. Instead, I'm getting only 7.65 TiB.
zfs list nas
NAME USED AVAIL REFER MOUNTPOINT
nas 558K 7.65T 140K /nas
zfs list nas -p
NAME USED AVAIL REFER MOUNTPOINT
nas 571392 8416155617280 142848 /nas
I also expect ZFS to use up some space for metadata and its own internal data structures and whatnot, but I'm missing over half a Terabyte of data:
Available space according to zfs list -p is
(8416155617280/1024^3) GiB = 7838 GiB
Available space after parity (3 drives of 5860533168 512-byte sectors) should be close to
5860533168*512/(1024^3)*3 GiB = 8383 GiB
8383 GiB - 7838 GiB = 545 GiB
Over half a terabyte, close to 5% of the aggregated raw disk space, is unaccounted for. Where did all that space go?
Note: I'm running Debian Jessie, ZFS 0.6.5.2-2. I did check for the quota, reservation, refquota and refreservation properties, which are all set to none. I just created this pool and there is no data in it, and of course no snapshots.
Here's my full zfs properties:
zfs get all nas
NAME PROPERTY VALUE SOURCE
nas type filesystem -
nas creation Tue Feb 7 17:28 2017 -
nas used 558K -
nas available 7.65T -
nas referenced 140K -
nas compressratio 1.00x -
nas mounted yes -
nas quota none default
nas reservation none default
nas recordsize 128K default
nas mountpoint /nas default
nas sharenfs off default
nas checksum on default
nas compression off default
nas atime on default
nas devices on default
nas exec on default
nas setuid on default
nas readonly off default
nas zoned off default
nas snapdir hidden default
nas aclinherit restricted default
nas canmount on default
nas xattr on default
nas copies 1 default
nas version 5 -
nas utf8only off -
nas normalization none -
nas casesensitivity sensitive -
nas vscan off default
nas nbmand off default
nas sharesmb off default
nas refquota none default
nas refreservation none default
nas primarycache all default
nas secondarycache all default
nas usedbysnapshots 0 -
nas usedbydataset 140K -
nas usedbychildren 418K -
nas usedbyrefreservation 0 -
nas logbias latency default
nas dedup off default
nas mlslabel none default
nas sync standard default
nas refcompressratio 1.00x -
nas written 140K -
nas logicalused 120K -
nas logicalreferenced 40K -
nas filesystem_limit none default
nas snapshot_limit none default
nas filesystem_count none default
nas snapshot_count none default
nas snapdev hidden default
nas acltype off default
nas context none default
nas fscontext none default
nas defcontext none default
nas rootcontext none default
nas relatime on temporary
nas redundant_metadata all default
nas overlay off default
Thank you!
linux raid zfs nas freenas
linux raid zfs nas freenas
edited Feb 9 '17 at 21:09
asked Feb 7 '17 at 23:30
Pablo Llopis
162
162
1
What is yourashiftsetting? Have a look at louwrentius.com/… , I have not checked your numbers correctly, but the difference is in similar size range.
– user121391
Feb 8 '17 at 13:28
That's an interesting comment, I didn't know about ashift! My ashift is 12 even though my HDDs seem to have 512 byte sectors. I tried rebuilding the pool with -o ashift=9 and I get a little bit of space back (7.88 TiB for ashift=9). Previously it was 7.65 TiB for ashift=12, which is the default. Even for ashift=9, there is still over 300GB of unaccounted space.
– Pablo Llopis
Feb 8 '17 at 14:59
Correction: parted reportsSector size (logical/physical): 512B/4096B. Nevertheless, I'm skeptical zfs needs to allocate for so much metadata than I'm losing over 300GB.
– Pablo Llopis
Feb 8 '17 at 15:06
add a comment |
1
What is yourashiftsetting? Have a look at louwrentius.com/… , I have not checked your numbers correctly, but the difference is in similar size range.
– user121391
Feb 8 '17 at 13:28
That's an interesting comment, I didn't know about ashift! My ashift is 12 even though my HDDs seem to have 512 byte sectors. I tried rebuilding the pool with -o ashift=9 and I get a little bit of space back (7.88 TiB for ashift=9). Previously it was 7.65 TiB for ashift=12, which is the default. Even for ashift=9, there is still over 300GB of unaccounted space.
– Pablo Llopis
Feb 8 '17 at 14:59
Correction: parted reportsSector size (logical/physical): 512B/4096B. Nevertheless, I'm skeptical zfs needs to allocate for so much metadata than I'm losing over 300GB.
– Pablo Llopis
Feb 8 '17 at 15:06
1
1
What is your
ashift setting? Have a look at louwrentius.com/… , I have not checked your numbers correctly, but the difference is in similar size range.– user121391
Feb 8 '17 at 13:28
What is your
ashift setting? Have a look at louwrentius.com/… , I have not checked your numbers correctly, but the difference is in similar size range.– user121391
Feb 8 '17 at 13:28
That's an interesting comment, I didn't know about ashift! My ashift is 12 even though my HDDs seem to have 512 byte sectors. I tried rebuilding the pool with -o ashift=9 and I get a little bit of space back (7.88 TiB for ashift=9). Previously it was 7.65 TiB for ashift=12, which is the default. Even for ashift=9, there is still over 300GB of unaccounted space.
– Pablo Llopis
Feb 8 '17 at 14:59
That's an interesting comment, I didn't know about ashift! My ashift is 12 even though my HDDs seem to have 512 byte sectors. I tried rebuilding the pool with -o ashift=9 and I get a little bit of space back (7.88 TiB for ashift=9). Previously it was 7.65 TiB for ashift=12, which is the default. Even for ashift=9, there is still over 300GB of unaccounted space.
– Pablo Llopis
Feb 8 '17 at 14:59
Correction: parted reports
Sector size (logical/physical): 512B/4096B. Nevertheless, I'm skeptical zfs needs to allocate for so much metadata than I'm losing over 300GB.– Pablo Llopis
Feb 8 '17 at 15:06
Correction: parted reports
Sector size (logical/physical): 512B/4096B. Nevertheless, I'm skeptical zfs needs to allocate for so much metadata than I'm losing over 300GB.– Pablo Llopis
Feb 8 '17 at 15:06
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f343312%2faccount-for-missing-space-as-reported-by-zfs-list%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
What is your
ashiftsetting? Have a look at louwrentius.com/… , I have not checked your numbers correctly, but the difference is in similar size range.– user121391
Feb 8 '17 at 13:28
That's an interesting comment, I didn't know about ashift! My ashift is 12 even though my HDDs seem to have 512 byte sectors. I tried rebuilding the pool with -o ashift=9 and I get a little bit of space back (7.88 TiB for ashift=9). Previously it was 7.65 TiB for ashift=12, which is the default. Even for ashift=9, there is still over 300GB of unaccounted space.
– Pablo Llopis
Feb 8 '17 at 14:59
Correction: parted reports
Sector size (logical/physical): 512B/4096B. Nevertheless, I'm skeptical zfs needs to allocate for so much metadata than I'm losing over 300GB.– Pablo Llopis
Feb 8 '17 at 15:06