Account for missing space as reported by ZFS list











up vote
3
down vote

favorite
1












This is a question regarding the available space as reported by zfs list that does not meet user expectations. I have looked at similar questions, but I still don't understand the zfs list output. Let me explain why:



I have a simple setup with just 4x WD30EFRX disks. Each disk is reported by Linux to have the following capacity:



5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)


That is, each drive has a capacity of 3000592982016 bytes. According to the math, the aggregated raw space for the four drives is 3000592982016*4 = 12002371928064 bytes (10.9 TiB), and zpool list seems to agree:



NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH ALTROOT
nas 10.9T 1.15M 10.9T - 0% 0% 1.00x ONLINE -


So good so far.
Now, I put those 4 drives into a RAIDZ1, meaning 25% of the total capacity should be used for parity. I would expect zfs list to report something close to 2.72 TiB *3 = 8.16 TiB. Instead, I'm getting only 7.65 TiB.



zfs list nas
NAME USED AVAIL REFER MOUNTPOINT
nas 558K 7.65T 140K /nas

zfs list nas -p
NAME USED AVAIL REFER MOUNTPOINT
nas 571392 8416155617280 142848 /nas


I also expect ZFS to use up some space for metadata and its own internal data structures and whatnot, but I'm missing over half a Terabyte of data:



Available space according to zfs list -p is



(8416155617280/1024^3) GiB = 7838 GiB


Available space after parity (3 drives of 5860533168 512-byte sectors) should be close to



5860533168*512/(1024^3)*3 GiB = 8383 GiB


8383 GiB - 7838 GiB = 545 GiB



Over half a terabyte, close to 5% of the aggregated raw disk space, is unaccounted for. Where did all that space go?



Note: I'm running Debian Jessie, ZFS 0.6.5.2-2. I did check for the quota, reservation, refquota and refreservation properties, which are all set to none. I just created this pool and there is no data in it, and of course no snapshots.
Here's my full zfs properties:



zfs get all nas                                                                            
NAME PROPERTY VALUE SOURCE
nas type filesystem -
nas creation Tue Feb 7 17:28 2017 -
nas used 558K -
nas available 7.65T -
nas referenced 140K -
nas compressratio 1.00x -
nas mounted yes -
nas quota none default
nas reservation none default
nas recordsize 128K default
nas mountpoint /nas default
nas sharenfs off default
nas checksum on default
nas compression off default
nas atime on default
nas devices on default
nas exec on default
nas setuid on default
nas readonly off default
nas zoned off default
nas snapdir hidden default
nas aclinherit restricted default
nas canmount on default
nas xattr on default
nas copies 1 default
nas version 5 -
nas utf8only off -
nas normalization none -
nas casesensitivity sensitive -
nas vscan off default
nas nbmand off default
nas sharesmb off default
nas refquota none default
nas refreservation none default
nas primarycache all default
nas secondarycache all default
nas usedbysnapshots 0 -
nas usedbydataset 140K -
nas usedbychildren 418K -
nas usedbyrefreservation 0 -
nas logbias latency default
nas dedup off default
nas mlslabel none default
nas sync standard default
nas refcompressratio 1.00x -
nas written 140K -
nas logicalused 120K -
nas logicalreferenced 40K -
nas filesystem_limit none default
nas snapshot_limit none default
nas filesystem_count none default
nas snapshot_count none default
nas snapdev hidden default
nas acltype off default
nas context none default
nas fscontext none default
nas defcontext none default
nas rootcontext none default
nas relatime on temporary
nas redundant_metadata all default
nas overlay off default


Thank you!










share|improve this question




















  • 1




    What is your ashift setting? Have a look at louwrentius.com/… , I have not checked your numbers correctly, but the difference is in similar size range.
    – user121391
    Feb 8 '17 at 13:28










  • That's an interesting comment, I didn't know about ashift! My ashift is 12 even though my HDDs seem to have 512 byte sectors. I tried rebuilding the pool with -o ashift=9 and I get a little bit of space back (7.88 TiB for ashift=9). Previously it was 7.65 TiB for ashift=12, which is the default. Even for ashift=9, there is still over 300GB of unaccounted space.
    – Pablo Llopis
    Feb 8 '17 at 14:59












  • Correction: parted reports Sector size (logical/physical): 512B/4096B. Nevertheless, I'm skeptical zfs needs to allocate for so much metadata than I'm losing over 300GB.
    – Pablo Llopis
    Feb 8 '17 at 15:06















up vote
3
down vote

favorite
1












This is a question regarding the available space as reported by zfs list that does not meet user expectations. I have looked at similar questions, but I still don't understand the zfs list output. Let me explain why:



I have a simple setup with just 4x WD30EFRX disks. Each disk is reported by Linux to have the following capacity:



5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)


That is, each drive has a capacity of 3000592982016 bytes. According to the math, the aggregated raw space for the four drives is 3000592982016*4 = 12002371928064 bytes (10.9 TiB), and zpool list seems to agree:



NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH ALTROOT
nas 10.9T 1.15M 10.9T - 0% 0% 1.00x ONLINE -


So good so far.
Now, I put those 4 drives into a RAIDZ1, meaning 25% of the total capacity should be used for parity. I would expect zfs list to report something close to 2.72 TiB *3 = 8.16 TiB. Instead, I'm getting only 7.65 TiB.



zfs list nas
NAME USED AVAIL REFER MOUNTPOINT
nas 558K 7.65T 140K /nas

zfs list nas -p
NAME USED AVAIL REFER MOUNTPOINT
nas 571392 8416155617280 142848 /nas


I also expect ZFS to use up some space for metadata and its own internal data structures and whatnot, but I'm missing over half a Terabyte of data:



Available space according to zfs list -p is



(8416155617280/1024^3) GiB = 7838 GiB


Available space after parity (3 drives of 5860533168 512-byte sectors) should be close to



5860533168*512/(1024^3)*3 GiB = 8383 GiB


8383 GiB - 7838 GiB = 545 GiB



Over half a terabyte, close to 5% of the aggregated raw disk space, is unaccounted for. Where did all that space go?



Note: I'm running Debian Jessie, ZFS 0.6.5.2-2. I did check for the quota, reservation, refquota and refreservation properties, which are all set to none. I just created this pool and there is no data in it, and of course no snapshots.
Here's my full zfs properties:



zfs get all nas                                                                            
NAME PROPERTY VALUE SOURCE
nas type filesystem -
nas creation Tue Feb 7 17:28 2017 -
nas used 558K -
nas available 7.65T -
nas referenced 140K -
nas compressratio 1.00x -
nas mounted yes -
nas quota none default
nas reservation none default
nas recordsize 128K default
nas mountpoint /nas default
nas sharenfs off default
nas checksum on default
nas compression off default
nas atime on default
nas devices on default
nas exec on default
nas setuid on default
nas readonly off default
nas zoned off default
nas snapdir hidden default
nas aclinherit restricted default
nas canmount on default
nas xattr on default
nas copies 1 default
nas version 5 -
nas utf8only off -
nas normalization none -
nas casesensitivity sensitive -
nas vscan off default
nas nbmand off default
nas sharesmb off default
nas refquota none default
nas refreservation none default
nas primarycache all default
nas secondarycache all default
nas usedbysnapshots 0 -
nas usedbydataset 140K -
nas usedbychildren 418K -
nas usedbyrefreservation 0 -
nas logbias latency default
nas dedup off default
nas mlslabel none default
nas sync standard default
nas refcompressratio 1.00x -
nas written 140K -
nas logicalused 120K -
nas logicalreferenced 40K -
nas filesystem_limit none default
nas snapshot_limit none default
nas filesystem_count none default
nas snapshot_count none default
nas snapdev hidden default
nas acltype off default
nas context none default
nas fscontext none default
nas defcontext none default
nas rootcontext none default
nas relatime on temporary
nas redundant_metadata all default
nas overlay off default


Thank you!










share|improve this question




















  • 1




    What is your ashift setting? Have a look at louwrentius.com/… , I have not checked your numbers correctly, but the difference is in similar size range.
    – user121391
    Feb 8 '17 at 13:28










  • That's an interesting comment, I didn't know about ashift! My ashift is 12 even though my HDDs seem to have 512 byte sectors. I tried rebuilding the pool with -o ashift=9 and I get a little bit of space back (7.88 TiB for ashift=9). Previously it was 7.65 TiB for ashift=12, which is the default. Even for ashift=9, there is still over 300GB of unaccounted space.
    – Pablo Llopis
    Feb 8 '17 at 14:59












  • Correction: parted reports Sector size (logical/physical): 512B/4096B. Nevertheless, I'm skeptical zfs needs to allocate for so much metadata than I'm losing over 300GB.
    – Pablo Llopis
    Feb 8 '17 at 15:06













up vote
3
down vote

favorite
1









up vote
3
down vote

favorite
1






1





This is a question regarding the available space as reported by zfs list that does not meet user expectations. I have looked at similar questions, but I still don't understand the zfs list output. Let me explain why:



I have a simple setup with just 4x WD30EFRX disks. Each disk is reported by Linux to have the following capacity:



5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)


That is, each drive has a capacity of 3000592982016 bytes. According to the math, the aggregated raw space for the four drives is 3000592982016*4 = 12002371928064 bytes (10.9 TiB), and zpool list seems to agree:



NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH ALTROOT
nas 10.9T 1.15M 10.9T - 0% 0% 1.00x ONLINE -


So good so far.
Now, I put those 4 drives into a RAIDZ1, meaning 25% of the total capacity should be used for parity. I would expect zfs list to report something close to 2.72 TiB *3 = 8.16 TiB. Instead, I'm getting only 7.65 TiB.



zfs list nas
NAME USED AVAIL REFER MOUNTPOINT
nas 558K 7.65T 140K /nas

zfs list nas -p
NAME USED AVAIL REFER MOUNTPOINT
nas 571392 8416155617280 142848 /nas


I also expect ZFS to use up some space for metadata and its own internal data structures and whatnot, but I'm missing over half a Terabyte of data:



Available space according to zfs list -p is



(8416155617280/1024^3) GiB = 7838 GiB


Available space after parity (3 drives of 5860533168 512-byte sectors) should be close to



5860533168*512/(1024^3)*3 GiB = 8383 GiB


8383 GiB - 7838 GiB = 545 GiB



Over half a terabyte, close to 5% of the aggregated raw disk space, is unaccounted for. Where did all that space go?



Note: I'm running Debian Jessie, ZFS 0.6.5.2-2. I did check for the quota, reservation, refquota and refreservation properties, which are all set to none. I just created this pool and there is no data in it, and of course no snapshots.
Here's my full zfs properties:



zfs get all nas                                                                            
NAME PROPERTY VALUE SOURCE
nas type filesystem -
nas creation Tue Feb 7 17:28 2017 -
nas used 558K -
nas available 7.65T -
nas referenced 140K -
nas compressratio 1.00x -
nas mounted yes -
nas quota none default
nas reservation none default
nas recordsize 128K default
nas mountpoint /nas default
nas sharenfs off default
nas checksum on default
nas compression off default
nas atime on default
nas devices on default
nas exec on default
nas setuid on default
nas readonly off default
nas zoned off default
nas snapdir hidden default
nas aclinherit restricted default
nas canmount on default
nas xattr on default
nas copies 1 default
nas version 5 -
nas utf8only off -
nas normalization none -
nas casesensitivity sensitive -
nas vscan off default
nas nbmand off default
nas sharesmb off default
nas refquota none default
nas refreservation none default
nas primarycache all default
nas secondarycache all default
nas usedbysnapshots 0 -
nas usedbydataset 140K -
nas usedbychildren 418K -
nas usedbyrefreservation 0 -
nas logbias latency default
nas dedup off default
nas mlslabel none default
nas sync standard default
nas refcompressratio 1.00x -
nas written 140K -
nas logicalused 120K -
nas logicalreferenced 40K -
nas filesystem_limit none default
nas snapshot_limit none default
nas filesystem_count none default
nas snapshot_count none default
nas snapdev hidden default
nas acltype off default
nas context none default
nas fscontext none default
nas defcontext none default
nas rootcontext none default
nas relatime on temporary
nas redundant_metadata all default
nas overlay off default


Thank you!










share|improve this question















This is a question regarding the available space as reported by zfs list that does not meet user expectations. I have looked at similar questions, but I still don't understand the zfs list output. Let me explain why:



I have a simple setup with just 4x WD30EFRX disks. Each disk is reported by Linux to have the following capacity:



5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)


That is, each drive has a capacity of 3000592982016 bytes. According to the math, the aggregated raw space for the four drives is 3000592982016*4 = 12002371928064 bytes (10.9 TiB), and zpool list seems to agree:



NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH ALTROOT
nas 10.9T 1.15M 10.9T - 0% 0% 1.00x ONLINE -


So good so far.
Now, I put those 4 drives into a RAIDZ1, meaning 25% of the total capacity should be used for parity. I would expect zfs list to report something close to 2.72 TiB *3 = 8.16 TiB. Instead, I'm getting only 7.65 TiB.



zfs list nas
NAME USED AVAIL REFER MOUNTPOINT
nas 558K 7.65T 140K /nas

zfs list nas -p
NAME USED AVAIL REFER MOUNTPOINT
nas 571392 8416155617280 142848 /nas


I also expect ZFS to use up some space for metadata and its own internal data structures and whatnot, but I'm missing over half a Terabyte of data:



Available space according to zfs list -p is



(8416155617280/1024^3) GiB = 7838 GiB


Available space after parity (3 drives of 5860533168 512-byte sectors) should be close to



5860533168*512/(1024^3)*3 GiB = 8383 GiB


8383 GiB - 7838 GiB = 545 GiB



Over half a terabyte, close to 5% of the aggregated raw disk space, is unaccounted for. Where did all that space go?



Note: I'm running Debian Jessie, ZFS 0.6.5.2-2. I did check for the quota, reservation, refquota and refreservation properties, which are all set to none. I just created this pool and there is no data in it, and of course no snapshots.
Here's my full zfs properties:



zfs get all nas                                                                            
NAME PROPERTY VALUE SOURCE
nas type filesystem -
nas creation Tue Feb 7 17:28 2017 -
nas used 558K -
nas available 7.65T -
nas referenced 140K -
nas compressratio 1.00x -
nas mounted yes -
nas quota none default
nas reservation none default
nas recordsize 128K default
nas mountpoint /nas default
nas sharenfs off default
nas checksum on default
nas compression off default
nas atime on default
nas devices on default
nas exec on default
nas setuid on default
nas readonly off default
nas zoned off default
nas snapdir hidden default
nas aclinherit restricted default
nas canmount on default
nas xattr on default
nas copies 1 default
nas version 5 -
nas utf8only off -
nas normalization none -
nas casesensitivity sensitive -
nas vscan off default
nas nbmand off default
nas sharesmb off default
nas refquota none default
nas refreservation none default
nas primarycache all default
nas secondarycache all default
nas usedbysnapshots 0 -
nas usedbydataset 140K -
nas usedbychildren 418K -
nas usedbyrefreservation 0 -
nas logbias latency default
nas dedup off default
nas mlslabel none default
nas sync standard default
nas refcompressratio 1.00x -
nas written 140K -
nas logicalused 120K -
nas logicalreferenced 40K -
nas filesystem_limit none default
nas snapshot_limit none default
nas filesystem_count none default
nas snapshot_count none default
nas snapdev hidden default
nas acltype off default
nas context none default
nas fscontext none default
nas defcontext none default
nas rootcontext none default
nas relatime on temporary
nas redundant_metadata all default
nas overlay off default


Thank you!







linux raid zfs nas freenas






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Feb 9 '17 at 21:09

























asked Feb 7 '17 at 23:30









Pablo Llopis

162




162








  • 1




    What is your ashift setting? Have a look at louwrentius.com/… , I have not checked your numbers correctly, but the difference is in similar size range.
    – user121391
    Feb 8 '17 at 13:28










  • That's an interesting comment, I didn't know about ashift! My ashift is 12 even though my HDDs seem to have 512 byte sectors. I tried rebuilding the pool with -o ashift=9 and I get a little bit of space back (7.88 TiB for ashift=9). Previously it was 7.65 TiB for ashift=12, which is the default. Even for ashift=9, there is still over 300GB of unaccounted space.
    – Pablo Llopis
    Feb 8 '17 at 14:59












  • Correction: parted reports Sector size (logical/physical): 512B/4096B. Nevertheless, I'm skeptical zfs needs to allocate for so much metadata than I'm losing over 300GB.
    – Pablo Llopis
    Feb 8 '17 at 15:06














  • 1




    What is your ashift setting? Have a look at louwrentius.com/… , I have not checked your numbers correctly, but the difference is in similar size range.
    – user121391
    Feb 8 '17 at 13:28










  • That's an interesting comment, I didn't know about ashift! My ashift is 12 even though my HDDs seem to have 512 byte sectors. I tried rebuilding the pool with -o ashift=9 and I get a little bit of space back (7.88 TiB for ashift=9). Previously it was 7.65 TiB for ashift=12, which is the default. Even for ashift=9, there is still over 300GB of unaccounted space.
    – Pablo Llopis
    Feb 8 '17 at 14:59












  • Correction: parted reports Sector size (logical/physical): 512B/4096B. Nevertheless, I'm skeptical zfs needs to allocate for so much metadata than I'm losing over 300GB.
    – Pablo Llopis
    Feb 8 '17 at 15:06








1




1




What is your ashift setting? Have a look at louwrentius.com/… , I have not checked your numbers correctly, but the difference is in similar size range.
– user121391
Feb 8 '17 at 13:28




What is your ashift setting? Have a look at louwrentius.com/… , I have not checked your numbers correctly, but the difference is in similar size range.
– user121391
Feb 8 '17 at 13:28












That's an interesting comment, I didn't know about ashift! My ashift is 12 even though my HDDs seem to have 512 byte sectors. I tried rebuilding the pool with -o ashift=9 and I get a little bit of space back (7.88 TiB for ashift=9). Previously it was 7.65 TiB for ashift=12, which is the default. Even for ashift=9, there is still over 300GB of unaccounted space.
– Pablo Llopis
Feb 8 '17 at 14:59






That's an interesting comment, I didn't know about ashift! My ashift is 12 even though my HDDs seem to have 512 byte sectors. I tried rebuilding the pool with -o ashift=9 and I get a little bit of space back (7.88 TiB for ashift=9). Previously it was 7.65 TiB for ashift=12, which is the default. Even for ashift=9, there is still over 300GB of unaccounted space.
– Pablo Llopis
Feb 8 '17 at 14:59














Correction: parted reports Sector size (logical/physical): 512B/4096B. Nevertheless, I'm skeptical zfs needs to allocate for so much metadata than I'm losing over 300GB.
– Pablo Llopis
Feb 8 '17 at 15:06




Correction: parted reports Sector size (logical/physical): 512B/4096B. Nevertheless, I'm skeptical zfs needs to allocate for so much metadata than I'm losing over 300GB.
– Pablo Llopis
Feb 8 '17 at 15:06















active

oldest

votes











Your Answer








StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f343312%2faccount-for-missing-space-as-reported-by-zfs-list%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown






























active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes
















draft saved

draft discarded




















































Thanks for contributing an answer to Unix & Linux Stack Exchange!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.





Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


Please pay close attention to the following guidance:


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f343312%2faccount-for-missing-space-as-reported-by-zfs-list%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Entries order in /etc/network/interfaces

新発田市

Grub takes very long (several minutes) to open Menu (in Multi-Boot-System)