Bug 203330 - ZFS free space bug or HDD driver bug
Summary: ZFS free space bug or HDD driver bug
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 10.2-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-fs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-09-25 10:20 UTC by Pekka Järvinen
Modified: 2015-10-25 10:18 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Pekka Järvinen 2015-09-25 10:20:02 UTC
I'm running Nas4Free 10.2.0.2 - Prester (revision 1814) which is based on FreeBSD 10.2-RELEASE-p2 #0 r287260M: Fri Aug 28 18:38:18 CEST 2015. System has 24 GB ECC memory. Motherboard is SuperMicro X8ST3-F. I've tried with ashift=9 and 12 without difference.

I only get 14.5T of free space.

I've tried OpenIndiana's ZFS implementation with same machine and setup and it gives 28.5T free and lists drives as 8001.49 GB. So this might be a bug in FreeBSD's ZFS implementation or FreeBSD HDD driver bug?

Disks:

ada0 at ata2 bus 0 scbus2 target 0 lun 0
ada0: <ST8000AS0002-1NA17Z AR13> ACS-2 ATA SATA 3.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada0: 7630885MB (15628053168 512 byte sectors: 16H 63S/T 16383C)
ada1 at ata2 bus 0 scbus2 target 1 lun 0
ada1: <ST8000AS0002-1NA17Z AR13> ACS-2 ATA SATA 3.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada1: 7630885MB (15628053168 512 byte sectors: 16H 63S/T 16383C)
ada2 at ata3 bus 0 scbus3 target 0 lun 0
ada2: <ST8000AS0002-1NA17Z AR13> ACS-2 ATA SATA 3.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada2: 7630885MB (15628053168 512 byte sectors: 16H 63S/T 16383C)
ada3 at ata3 bus 0 scbus3 target 1 lun 0
ada3: <ST8000AS0002-1NA17Z AR13> ACS-2 ATA SATA 3.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada3: 7630885MB (15628053168 512 byte sectors: 16H 63S/T 16383C)
ada4 at ata4 bus 0 scbus4 target 0 lun 0
ada4: <ST8000AS0002-1NA17Z AR13> ACS-2 ATA SATA 3.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada4: 7630885MB (15628053168 512 byte sectors: 16H 63S/T 16383C)
ada5 at ata5 bus 0 scbus5 target 0 lun 0
ada5: <ST8000AS0002-1NA17Z AR13> ACS-2 ATA SATA 3.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada5: 7630885MB (15628053168 512 byte sectors: 16H 63S/T 16383C)


Create new pool:

storage: /# zpool create storage2 raidz2 ada0 ada1 ada2 ada3 ada4 ada5
zfs list:

storage: /# zfs list
NAME       USED  AVAIL  REFER  MOUNTPOINT
storage2   432K  14.5T   192K  /storage2
iostat:

storage: /# zpool iostat storage2
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
storage2     900K  43.5T      0      1      0  7.84K
zpool list:

storage: /# zpool list
NAME       SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
storage2  43.5T   900K  43.5T         -     0%     0%  1.00x  ONLINE  -
zpool status:

storage: /# zpool status
  pool: storage2
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage2    ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            ada0    ONLINE       0     0     0
            ada1    ONLINE       0     0     0
            ada2    ONLINE       0     0     0
            ada3    ONLINE       0     0     0
            ada4    ONLINE       0     0     0
            ada5    ONLINE       0     0     0

errors: No known data errors
zfs list:

storage: /# zfs list -t all
NAME       USED  AVAIL  REFER  MOUNTPOINT
storage2   480K  14.5T   192K  /storage2
Properties:

# zfs get all storage2
NAME      PROPERTY              VALUE                  SOURCE
storage2  type                  filesystem             -
storage2  creation              Thu Sep 24 18:48 2015  -
storage2  used                  623K                   -
storage2  available             14.5T                  -
storage2  referenced            192K                   -
storage2  compressratio         1.00x                  -
storage2  mounted               yes                    -
storage2  quota                 none                   default
storage2  reservation           none                   default
storage2  recordsize            128K                   default
storage2  mountpoint            /storage2              default
storage2  sharenfs              off                    default
storage2  checksum              on                     default
storage2  compression           off                    default
storage2  atime                 on                     default
storage2  devices               on                     default
storage2  exec                  on                     default
storage2  setuid                on                     default
storage2  readonly              off                    default
storage2  jailed                off                    default
storage2  snapdir               hidden                 default
storage2  aclmode               discard                default
storage2  aclinherit            restricted             default
storage2  canmount              on                     default
storage2  xattr                 off                    temporary
storage2  copies                1                      default
storage2  version               5                      -
storage2  utf8only              off                    -
storage2  normalization         none                   -
storage2  casesensitivity       sensitive              -
storage2  vscan                 off                    default
storage2  nbmand                off                    default
storage2  sharesmb              off                    default
storage2  refquota              none                   default
storage2  refreservation        none                   default
storage2  primarycache          all                    default
storage2  secondarycache        all                    default
storage2  usedbysnapshots       0                      -
storage2  usedbydataset         192K                   -
storage2  usedbychildren        432K                   -
storage2  usedbyrefreservation  0                      -
storage2  logbias               latency                default
storage2  dedup                 off                    default
storage2  mlslabel                                     -
storage2  sync                  standard               default
storage2  refcompressratio      1.00x                  -
storage2  written               192K                   -
storage2  logicalused           80K                    -
storage2  logicalreferenced     9.50K                  -
storage2  volmode               default                default
storage2  filesystem_limit      none                   default
storage2  snapshot_limit        none                   default
storage2  filesystem_count      none                   default
storage2  snapshot_count        none                   default
storage2  redundant_metadata    all                    default
 
 
 
# zdb storage2
 
Cached configuration:
        version: 5000
        name: 'storage2'
        state: 0
        txg: 4
        pool_guid: 10414888771172952257
        hostid: 1452759488
        hostname: 'storage.home'
        vdev_children: 1
        vdev_tree:
            type: 'root'
            id: 0
            guid: 10414888771172952257
            create_txg: 4
            children[0]:
                type: 'raidz'
                id: 0
                guid: 2681305684837460215
                nparity: 2
                metaslab_array: 34
                metaslab_shift: 38
                ashift: 12
                asize: 48009350479872
                is_log: 0
                create_txg: 4
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 14137175296949034209
                    path: '/dev/ada0'
                    phys_path: '/dev/ada0'
                    whole_disk: 1
                    create_txg: 4
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 5793672826394463181
                    path: '/dev/ada1'
                    phys_path: '/dev/ada1'
                    whole_disk: 1
                    create_txg: 4
                children[2]:
                    type: 'disk'
                    id: 2
                    guid: 426194980956466479
                    path: '/dev/ada2'
                    phys_path: '/dev/ada2'
                    whole_disk: 1
                    create_txg: 4
                children[3]:
                    type: 'disk'
                    id: 3
                    guid: 5113566757325497587
                    path: '/dev/ada3'
                    phys_path: '/dev/ada3'
                    whole_disk: 1
                    create_txg: 4
                children[4]:
                    type: 'disk'
                    id: 4
                    guid: 16297810626525192704
                    path: '/dev/ada4'
                    phys_path: '/dev/ada4'
                    whole_disk: 1
                    create_txg: 4
                children[5]:
                    type: 'disk'
                    id: 5
                    guid: 16307112257360078181
                    path: '/dev/ada5'
                    phys_path: '/dev/ada5'
                    whole_disk: 1
                    create_txg: 4
        features_for_read:
            com.delphix:hole_birth
            com.delphix:embedded_data
 
MOS Configuration:
        version: 5000
        name: 'storage2'
        state: 0
        txg: 4
        pool_guid: 10414888771172952257
        hostid: 1452759488
        hostname: 'storage.home'
        vdev_children: 1
        vdev_tree:
            type: 'root'
            id: 0
            guid: 10414888771172952257
            create_txg: 4
            children[0]:
                type: 'raidz'
                id: 0
                guid: 2681305684837460215
                nparity: 2
                metaslab_array: 34
                metaslab_shift: 38
                ashift: 12
                asize: 48009350479872
                is_log: 0
                create_txg: 4
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 14137175296949034209
                    path: '/dev/ada0'
                    phys_path: '/dev/ada0'
                    whole_disk: 1
                    create_txg: 4
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 5793672826394463181
                    path: '/dev/ada1'
                    phys_path: '/dev/ada1'
                    whole_disk: 1
                    create_txg: 4
                children[2]:
                    type: 'disk'
                    id: 2
                    guid: 426194980956466479
                    path: '/dev/ada2'
                    phys_path: '/dev/ada2'
                    whole_disk: 1
                    create_txg: 4
                children[3]:
                    type: 'disk'
                    id: 3
                    guid: 5113566757325497587
                    path: '/dev/ada3'
                    phys_path: '/dev/ada3'
                    whole_disk: 1
                    create_txg: 4
                children[4]:
                    type: 'disk'
                    id: 4
                    guid: 16297810626525192704
                    path: '/dev/ada4'
                    phys_path: '/dev/ada4'
                    whole_disk: 1
                    create_txg: 4
                children[5]:
                    type: 'disk'
                    id: 5
                    guid: 16307112257360078181
                    path: '/dev/ada5'
                    phys_path: '/dev/ada5'
                    whole_disk: 1
                    create_txg: 4
        features_for_read:
            com.delphix:hole_birth
            com.delphix:embedded_data
 
Uberblock:
        magic = 0000000000bab10c
        version = 5000
        txg = 5
        guid_sum = 15831494980392489965
        timestamp = 1443109719 UTC = Thu Sep 24 18:48:39 2015
 
All DDTs are empty
 
Metaslabs:
        vdev          0
        metaslabs   174   offset                spacemap          free
        ---------------   -------------------   ---------------   -------------
        metaslab      0   offset            0   spacemap     37   free     256G
        On-disk histogram:              fragmentation 0
                         37:      1 *
        metaslab      1   offset   4000000000   spacemap      0   free     256G
        metaslab      2   offset   8000000000   spacemap      0   free     256G
        metaslab      3   offset   c000000000   spacemap      0   free     256G
        metaslab      4   offset  10000000000   spacemap      0   free     256G
        metaslab      5   offset  14000000000   spacemap      0   free     256G
        metaslab      6   offset  18000000000   spacemap      0   free     256G
        metaslab      7   offset  1c000000000   spacemap      0   free     256G
        metaslab      8   offset  20000000000   spacemap      0   free     256G
        metaslab      9   offset  24000000000   spacemap      0   free     256G
        metaslab     10   offset  28000000000   spacemap      0   free     256G
        metaslab     11   offset  2c000000000   spacemap      0   free     256G
        metaslab     12   offset  30000000000   spacemap      0   free     256G
        metaslab     13   offset  34000000000   spacemap      0   free     256G
        metaslab     14   offset  38000000000   spacemap      0   free     256G
        metaslab     15   offset  3c000000000   spacemap      0   free     256G
        metaslab     16   offset  40000000000   spacemap      0   free     256G
        metaslab     17   offset  44000000000   spacemap      0   free     256G
        metaslab     18   offset  48000000000   spacemap      0   free     256G
        metaslab     19   offset  4c000000000   spacemap      0   free     256G
        metaslab     20   offset  50000000000   spacemap      0   free     256G
        metaslab     21   offset  54000000000   spacemap      0   free     256G
        metaslab     22   offset  58000000000   spacemap      0   free     256G
        metaslab     23   offset  5c000000000   spacemap      0   free     256G
        metaslab     24   offset  60000000000   spacemap      0   free     256G
        metaslab     25   offset  64000000000   spacemap      0   free     256G
        metaslab     26   offset  68000000000   spacemap      0   free     256G
        metaslab     27   offset  6c000000000   spacemap      0   free     256G
        metaslab     28   offset  70000000000   spacemap      0   free     256G
        metaslab     29   offset  74000000000   spacemap      0   free     256G
        metaslab     30   offset  78000000000   spacemap      0   free     256G
        metaslab     31   offset  7c000000000   spacemap      0   free     256G
        metaslab     32   offset  80000000000   spacemap      0   free     256G
        metaslab     33   offset  84000000000   spacemap     36   free     256G
        On-disk histogram:              fragmentation 0
                         37:      1 *
        metaslab     34   offset  88000000000   spacemap      0   free     256G
        metaslab     35   offset  8c000000000   spacemap      0   free     256G
        metaslab     36   offset  90000000000   spacemap      0   free     256G
        metaslab     37   offset  94000000000   spacemap      0   free     256G
        metaslab     38   offset  98000000000   spacemap      0   free     256G
        metaslab     39   offset  9c000000000   spacemap      0   free     256G
        metaslab     40   offset  a0000000000   spacemap      0   free     256G
        metaslab     41   offset  a4000000000   spacemap      0   free     256G
        metaslab     42   offset  a8000000000   spacemap      0   free     256G
        metaslab     43   offset  ac000000000   spacemap      0   free     256G
        metaslab     44   offset  b0000000000   spacemap      0   free     256G
        metaslab     45   offset  b4000000000   spacemap      0   free     256G
        metaslab     46   offset  b8000000000   spacemap      0   free     256G
        metaslab     47   offset  bc000000000   spacemap      0   free     256G
        metaslab     48   offset  c0000000000   spacemap      0   free     256G
        metaslab     49   offset  c4000000000   spacemap      0   free     256G
        metaslab     50   offset  c8000000000   spacemap      0   free     256G
        metaslab     51   offset  cc000000000   spacemap      0   free     256G
        metaslab     52   offset  d0000000000   spacemap      0   free     256G
        metaslab     53   offset  d4000000000   spacemap      0   free     256G
        metaslab     54   offset  d8000000000   spacemap      0   free     256G
        metaslab     55   offset  dc000000000   spacemap      0   free     256G
        metaslab     56   offset  e0000000000   spacemap      0   free     256G
        metaslab     57   offset  e4000000000   spacemap      0   free     256G
        metaslab     58   offset  e8000000000   spacemap      0   free     256G
        metaslab     59   offset  ec000000000   spacemap      0   free     256G
        metaslab     60   offset  f0000000000   spacemap      0   free     256G
        metaslab     61   offset  f4000000000   spacemap      0   free     256G
        metaslab     62   offset  f8000000000   spacemap      0   free     256G
        metaslab     63   offset  fc000000000   spacemap      0   free     256G
        metaslab     64   offset 100000000000   spacemap      0   free     256G
        metaslab     65   offset 104000000000   spacemap      0   free     256G
        metaslab     66   offset 108000000000   spacemap     35   free     256G
        On-disk histogram:              fragmentation 0
                         37:      1 *
        metaslab     67   offset 10c000000000   spacemap      0   free     256G
        metaslab     68   offset 110000000000   spacemap      0   free     256G
        metaslab     69   offset 114000000000   spacemap      0   free     256G
        metaslab     70   offset 118000000000   spacemap      0   free     256G
        metaslab     71   offset 11c000000000   spacemap      0   free     256G
        metaslab     72   offset 120000000000   spacemap      0   free     256G
        metaslab     73   offset 124000000000   spacemap      0   free     256G
        metaslab     74   offset 128000000000   spacemap      0   free     256G
        metaslab     75   offset 12c000000000   spacemap      0   free     256G
        metaslab     76   offset 130000000000   spacemap      0   free     256G
        metaslab     77   offset 134000000000   spacemap      0   free     256G
        metaslab     78   offset 138000000000   spacemap      0   free     256G
        metaslab     79   offset 13c000000000   spacemap      0   free     256G
        metaslab     80   offset 140000000000   spacemap      0   free     256G
        metaslab     81   offset 144000000000   spacemap      0   free     256G
        metaslab     82   offset 148000000000   spacemap      0   free     256G
        metaslab     83   offset 14c000000000   spacemap      0   free     256G
        metaslab     84   offset 150000000000   spacemap      0   free     256G
        metaslab     85   offset 154000000000   spacemap      0   free     256G
        metaslab     86   offset 158000000000   spacemap      0   free     256G
        metaslab     87   offset 15c000000000   spacemap      0   free     256G
        metaslab     88   offset 160000000000   spacemap      0   free     256G
        metaslab     89   offset 164000000000   spacemap      0   free     256G
        metaslab     90   offset 168000000000   spacemap      0   free     256G
        metaslab     91   offset 16c000000000   spacemap      0   free     256G
        metaslab     92   offset 170000000000   spacemap      0   free     256G
        metaslab     93   offset 174000000000   spacemap      0   free     256G
        metaslab     94   offset 178000000000   spacemap      0   free     256G
        metaslab     95   offset 17c000000000   spacemap      0   free     256G
        metaslab     96   offset 180000000000   spacemap      0   free     256G
        metaslab     97   offset 184000000000   spacemap      0   free     256G
        metaslab     98   offset 188000000000   spacemap      0   free     256G
        metaslab     99   offset 18c000000000   spacemap      0   free     256G
        metaslab    100   offset 190000000000   spacemap      0   free     256G
        metaslab    101   offset 194000000000   spacemap      0   free     256G
        metaslab    102   offset 198000000000   spacemap      0   free     256G
        metaslab    103   offset 19c000000000   spacemap      0   free     256G
        metaslab    104   offset 1a0000000000   spacemap      0   free     256G
        metaslab    105   offset 1a4000000000   spacemap      0   free     256G
        metaslab    106   offset 1a8000000000   spacemap      0   free     256G
        metaslab    107   offset 1ac000000000   spacemap      0   free     256G
        metaslab    108   offset 1b0000000000   spacemap      0   free     256G
        metaslab    109   offset 1b4000000000   spacemap      0   free     256G
        metaslab    110   offset 1b8000000000   spacemap      0   free     256G
        metaslab    111   offset 1bc000000000   spacemap      0   free     256G
        metaslab    112   offset 1c0000000000   spacemap      0   free     256G
        metaslab    113   offset 1c4000000000   spacemap      0   free     256G
        metaslab    114   offset 1c8000000000   spacemap      0   free     256G
        metaslab    115   offset 1cc000000000   spacemap      0   free     256G
        metaslab    116   offset 1d0000000000   spacemap      0   free     256G
        metaslab    117   offset 1d4000000000   spacemap      0   free     256G
        metaslab    118   offset 1d8000000000   spacemap      0   free     256G
        metaslab    119   offset 1dc000000000   spacemap      0   free     256G
        metaslab    120   offset 1e0000000000   spacemap      0   free     256G
        metaslab    121   offset 1e4000000000   spacemap      0   free     256G
        metaslab    122   offset 1e8000000000   spacemap      0   free     256G
        metaslab    123   offset 1ec000000000   spacemap      0   free     256G
        metaslab    124   offset 1f0000000000   spacemap      0   free     256G
        metaslab    125   offset 1f4000000000   spacemap      0   free     256G
        metaslab    126   offset 1f8000000000   spacemap      0   free     256G
        metaslab    127   offset 1fc000000000   spacemap      0   free     256G
        metaslab    128   offset 200000000000   spacemap      0   free     256G
        metaslab    129   offset 204000000000   spacemap      0   free     256G
        metaslab    130   offset 208000000000   spacemap      0   free     256G
        metaslab    131   offset 20c000000000   spacemap      0   free     256G
        metaslab    132   offset 210000000000   spacemap      0   free     256G
        metaslab    133   offset 214000000000   spacemap      0   free     256G
        metaslab    134   offset 218000000000   spacemap      0   free     256G
        metaslab    135   offset 21c000000000   spacemap      0   free     256G
        metaslab    136   offset 220000000000   spacemap      0   free     256G
        metaslab    137   offset 224000000000   spacemap      0   free     256G
        metaslab    138   offset 228000000000   spacemap      0   free     256G
        metaslab    139   offset 22c000000000   spacemap      0   free     256G
        metaslab    140   offset 230000000000   spacemap      0   free     256G
        metaslab    141   offset 234000000000   spacemap      0   free     256G
        metaslab    142   offset 238000000000   spacemap      0   free     256G
        metaslab    143   offset 23c000000000   spacemap      0   free     256G
        metaslab    144   offset 240000000000   spacemap      0   free     256G
        metaslab    145   offset 244000000000   spacemap      0   free     256G
        metaslab    146   offset 248000000000   spacemap      0   free     256G
        metaslab    147   offset 24c000000000   spacemap      0   free     256G
        metaslab    148   offset 250000000000   spacemap      0   free     256G
        metaslab    149   offset 254000000000   spacemap      0   free     256G
        metaslab    150   offset 258000000000   spacemap      0   free     256G
        metaslab    151   offset 25c000000000   spacemap      0   free     256G
        metaslab    152   offset 260000000000   spacemap      0   free     256G
        metaslab    153   offset 264000000000   spacemap      0   free     256G
        metaslab    154   offset 268000000000   spacemap      0   free     256G
        metaslab    155   offset 26c000000000   spacemap      0   free     256G
        metaslab    156   offset 270000000000   spacemap      0   free     256G
        metaslab    157   offset 274000000000   spacemap      0   free     256G
        metaslab    158   offset 278000000000   spacemap      0   free     256G
        metaslab    159   offset 27c000000000   spacemap      0   free     256G
        metaslab    160   offset 280000000000   spacemap      0   free     256G
        metaslab    161   offset 284000000000   spacemap      0   free     256G
        metaslab    162   offset 288000000000   spacemap      0   free     256G
        metaslab    163   offset 28c000000000   spacemap      0   free     256G
        metaslab    164   offset 290000000000   spacemap      0   free     256G
        metaslab    165   offset 294000000000   spacemap      0   free     256G
        metaslab    166   offset 298000000000   spacemap      0   free     256G
        metaslab    167   offset 29c000000000   spacemap      0   free     256G
        metaslab    168   offset 2a0000000000   spacemap      0   free     256G
        metaslab    169   offset 2a4000000000   spacemap      0   free     256G
        metaslab    170   offset 2a8000000000   spacemap      0   free     256G
        metaslab    171   offset 2ac000000000   spacemap      0   free     256G
        metaslab    172   offset 2b0000000000   spacemap      0   free     256G
        metaslab    173   offset 2b4000000000   spacemap      0   free     256G
 
        vdev          0         metaslabs  174          fragmentation  0%
                         37:      3 ***
        pool storage2   fragmentation     0%
                         37:      3 ***
Dataset mos [META], ID 0, cr_txg 4, 432K, 37 objects
 
    Object  lvl   iblk   dblk  dsize  lsize   %full  type
         0    1    16K    16K  96.0K    32K   57.81  DMU dnode
         1    1    16K     1K  24.0K     1K  100.00  object directory
         2    1    16K    512      0    512    0.00  DSL directory
         3    1    16K    512      0    512  100.00  DSL props
         4    1    16K    512      0    512  100.00  DSL directory child map
         5    1    16K    512      0    512    0.00  DSL directory
         6    1    16K    512      0    512  100.00  DSL props
         7    1    16K    512      0    512  100.00  DSL directory child map
         8    1    16K    512      0    512    0.00  DSL directory
         9    1    16K    512      0    512  100.00  DSL props
        10    1    16K    512      0    512  100.00  DSL directory child map
        11    1    16K   128K      0   128K    0.00  bpobj
        12    1    16K    512      0    512    0.00  DSL directory
        13    1    16K    512      0    512  100.00  DSL props
        14    1    16K    512      0    512  100.00  DSL directory child map
        15    1    16K    512      0    512    0.00  DSL dataset
        16    1    16K    512      0    512  100.00  DSL dataset snap map
        17    1    16K    512      0    512  100.00  DSL deadlist map
        18    1    16K    512      0    512    0.00  DSL dataset
        19    1    16K    512      0    512  100.00  DSL deadlist map
        20    1    16K   128K      0   128K    0.00  bpobj
        21    1    16K    512      0    512    0.00  DSL dataset
        22    1    16K    512      0    512  100.00  DSL dataset snap map
        23    1    16K    512      0    512  100.00  DSL deadlist map
        24    1    16K   128K      0   128K    0.00  bpobj
        25    1    16K    512      0    512  100.00  DSL dataset next clones
        26    1    16K    512      0    512  100.00  DSL dir clones
        27    1    16K    16K  24.0K    16K  100.00  packed nvlist
        28    1    16K    512  24.0K    512  100.00  zap
        29    1    16K    512  24.0K    512  100.00  zap
        30    1    16K    16K  48.0K    32K  100.00  zap
        31    1    16K    16K  48.0K    16K  100.00  bpobj (Z=uncompressed)
        32    1    16K   128K  24.0K   128K  100.00  SPA history
        33    1    16K    512  24.0K    512  100.00  zap
        34    1    16K    512      0     1K  100.00  object array
        35    1    16K     4K  24.0K     4K  100.00  SPA space map
        36    1    16K     4K  24.0K     4K  100.00  SPA space map
        37    1    16K     4K  24.0K     4K  100.00  SPA space map
 
Dataset storage2 [ZPL], ID 21, cr_txg 1, 192K, 7 objects
 
    Object  lvl   iblk   dblk  dsize  lsize   %full  type
         0    7    16K    16K   112K    16K   21.88  DMU dnode
        -1    1    16K    512      0    512  100.00  ZFS user/group used
        -2    1    16K    512      0    512  100.00  ZFS user/group used
         1    1    16K     1K    16K     1K  100.00  ZFS master node
         2    1    16K    512      0    512  100.00  SA master node
         3    1    16K    512      0    512  100.00  ZFS delete queue
         4    1    16K    512      0    512  100.00  ZFS directory
         5    1    16K  1.50K    16K  1.50K  100.00  SA attr registration
         6    1    16K    16K    32K    32K  100.00  SA attr layouts
         7    1    16K    512      0    512  100.00  ZFS directory
 
Verified large_blocks feature refcount is correct (0)
 
Traversing all blocks to verify checksums and verify nothing leaked ...
 
loading space map for vdev 0 of 1, metaslab 66 of 174 ...
 
        No leaks (block sum matches space maps exactly)
 
        bp count:              60
        ganged count:           0
        bp logical:        545280      avg:   9088
        bp physical:        92160      avg:   1536     compression:   5.92
        bp allocated:     1327104      avg:  22118     compression:   0.41
        bp deduped:             0    ref>1:      0   deduplication:   1.00
        SPA allocated:    1327104     used:  0.00%
 
        additional, non-pointer bps of type 0:         23
        Dittoed blocks on same vdev: 37
 
Blocks  LSIZE   PSIZE   ASIZE     avg    comp   %Total  Type
     -      -       -       -       -       -        -  unallocated
     1     1K     512   36.0K   36.0K    2.00     2.78  object directory
     -      -       -       -       -       -        -  object array
     1    16K      1K   36.0K   36.0K   16.00     2.78  packed nvlist
     -      -       -       -       -       -        -  packed nvlist size
     1    16K     16K   72.0K   72.0K    1.00     5.56  bpobj
     -      -       -       -       -       -        -  bpobj header
     -      -       -       -       -       -        -  SPA space map header
     3  12.0K   12.0K    108K   36.0K    1.00     8.33  SPA space map
     -      -       -       -       -       -        -  ZIL intent log
     9   144K   36.0K    312K   34.7K    4.00    24.07  DMU dnode
     2     4K   2.50K   60.0K   30.0K    1.60     4.63  DMU objset
     -      -       -       -       -       -        -  DSL directory
     -      -       -       -       -       -        -  DSL directory child map
     -      -       -       -       -       -        -  DSL dataset snap map
     -      -       -       -       -       -        -  DSL props
     -      -       -       -       -       -        -  DSL dataset
     -      -       -       -       -       -        -  ZFS znode
     -      -       -       -       -       -        -  ZFS V0 ACL
     -      -       -       -       -       -        -  ZFS plain file
     -      -       -       -       -       -        -  ZFS directory
     1     1K     512   24.0K   24.0K    2.00     1.85  ZFS master node
     -      -       -       -       -       -        -  ZFS delete queue
     -      -       -       -       -       -        -  zvol object
     -      -       -       -       -       -        -  zvol prop
     -      -       -       -       -       -        -  other uint8[]
     -      -       -       -       -       -        -  other uint64[]
     -      -       -       -       -       -        -  other ZAP
     -      -       -       -       -       -        -  persistent error log
     1   128K      1K   36.0K   36.0K   128.00    2.78  SPA history
     -      -       -       -       -       -        -  SPA history offsets
     -      -       -       -       -       -        -  Pool properties
     -      -       -       -       -       -        -  DSL permissions
     -      -       -       -       -       -        -  ZFS ACL
     -      -       -       -       -       -        -  ZFS SYSACL
     -      -       -       -       -       -        -  FUID table
     -      -       -       -       -       -        -  FUID table size
     -      -       -       -       -       -        -  DSL dataset next clones
     -      -       -       -       -       -        -  scan work queue
     -      -       -       -       -       -        -  ZFS user/group used
     -      -       -       -       -       -        -  ZFS user/group quota
     -      -       -       -       -       -        -  snapshot refcount tags
     -      -       -       -       -       -        -  DDT ZAP algorithm
     -      -       -       -       -       -        -  DDT statistics
     -      -       -       -       -       -        -  System attributes
     -      -       -       -       -       -        -  SA master node
     1  1.50K     512   24.0K   24.0K    3.00     1.85  SA attr registration
     2    32K      4K   48.0K   24.0K    8.00     3.70  SA attr layouts
     -      -       -       -       -       -        -  scan translations
     -      -       -       -       -       -        -  deduplicated block
     -      -       -       -       -       -        -  DSL deadlist map
     -      -       -       -       -       -        -  DSL deadlist map hdr
     -      -       -       -       -       -        -  DSL dir clones
     -      -       -       -       -       -        -  bpobj subobj
    10   132K   10.0K    360K   36.0K   13.20    27.78  deferred free
     -      -       -       -       -       -        -  dedup ditto
     5  33.5K   6.00K    180K   36.0K    5.58    13.89  other
    60   533K   90.0K   1.27M   21.6K    5.92   100.00  Total
 
                            capacity   operations   bandwidth  ---- errors ----
description                used avail  read write  read write  read write cksum
storage2                  1.27M 43.5T   169     0  928K     0     0     0     0
  raidz2                  1.27M 43.5T   169     0  928K     0     0     0     0
    /dev/ada0                           100     0 1.25M     0     0     0     0
    /dev/ada1                            96     0  828K     0     0     0     0
    /dev/ada2                           111     0  888K     0     0     0     0
    /dev/ada3                            93     0  816K     0     0     0     0
    /dev/ada4                            93     0  816K     0     0     0     0
    /dev/ada5                           109     0  880K     0     0     0     0
 
History:
unrecognized record:
  history internal str: 'pool version 5000; software version 5000/5; uts storage.home 10.2-RELEASE-p2 1002000 amd64'
  internal_name: 'create'
  history txg: 5
  history time: 1443109719
  history hostname: 'storage.home'
2015-09-24.18:48:39 zpool create storage2 raidz2 ada0 ada1 ada2 ada3 ada4 ada5
  history command: 'zpool create storage2 raidz2 ada0 ada1 ada2 ada3 ada4 ada5'
  history who: 0
  history time: 1443109719
  history hostname: 'storage.home'
Comment 1 Pekka Järvinen 2015-09-25 13:47:37 UTC
Tried with FreeBSD-10.2-STABLE-amd64-20150917-r287929-memstick.img.

Gives 28.1T free. So it seems fixed.

ada0 at ata2 bus 0 scbus2 target 0 lun 0
ada0: <ST8000AS0002-1NA17Z AR13> ACS-2 ATA SATA 3.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada0: 7630885MB (15628053168 512 byte sectors: 16H 63S/T 16383C)
ada1 at ata2 bus 0 scbus2 target 1 lun 0
ada1: <ST8000AS0002-1NA17Z AR13> ACS-2 ATA SATA 3.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada1: 7630885MB (15628053168 512 byte sectors: 16H 63S/T 16383C)
ada2 at ata3 bus 0 scbus3 target 0 lun 0
ada2: <ST8000AS0002-1NA17Z AR13> ACS-2 ATA SATA 3.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada2: 7630885MB (15628053168 512 byte sectors: 16H 63S/T 16383C)
ada3 at ata3 bus 0 scbus3 target 1 lun 0
ada3: <ST8000AS0002-1NA17Z AR13> ACS-2 ATA SATA 3.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada3: 7630885MB (15628053168 512 byte sectors: 16H 63S/T 16383C)
ada4 at ata4 bus 0 scbus4 target 0 lun 0
ada4: <ST8000AS0002-1NA17Z AR13> ACS-2 ATA SATA 3.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada4: 7630885MB (15628053168 512 byte sectors: 16H 63S/T 16383C)
ada5 at ata5 bus 0 scbus5 target 0 lun 0
ada5: <ST8000AS0002-1NA17Z AR13> ACS-2 ATA SATA 3.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada5: 7630885MB (15628053168 512 byte sectors: 16H 63S/T 16383C)


# zpool iostat storage2
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
storage2    1.41M  43.5T      0      9      0  40.1K

# zpool status
  pool: storage2
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage2    ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            ada0    ONLINE       0     0     0
            ada1    ONLINE       0     0     0
            ada2    ONLINE       0     0     0
            ada3    ONLINE       0     0     0
            ada4    ONLINE       0     0     0
            ada5    ONLINE       0     0     0

errors: No known data errors

# zfs list -t all
NAME       USED  AVAIL  REFER  MOUNTPOINT
storage2   767K  28.1T   192K  /storage2


# zfs get all storage2
NAME      PROPERTY              VALUE                  SOURCE
storage2  type                  filesystem             -
storage2  creation              Fri Sep 25 13:27 2015  -
storage2  used                  767K                   -
storage2  available             28.1T                  -
storage2  referenced            192K                   -
storage2  compressratio         1.00x                  -
storage2  mounted               no                     -
storage2  quota                 none                   default
storage2  reservation           none                   default
storage2  recordsize            128K                   default
storage2  mountpoint            /storage2              default
storage2  sharenfs              off                    default
storage2  checksum              on                     default
storage2  compression           off                    default
storage2  atime                 on                     default
storage2  devices               on                     default
storage2  exec                  on                     default
storage2  setuid                on                     default
storage2  readonly              off                    default
storage2  jailed                off                    default
storage2  snapdir               hidden                 default
storage2  aclmode               discard                default
storage2  aclinherit            restricted             default
storage2  canmount              on                     default
storage2  xattr                 on                     default
storage2  copies                1                      default
storage2  version               5                      -
storage2  utf8only              off                    -
storage2  normalization         none                   -
storage2  casesensitivity       sensitive              -
storage2  vscan                 off                    default
storage2  nbmand                off                    default
storage2  sharesmb              off                    default
storage2  refquota              none                   default
storage2  refreservation        none                   default
storage2  primarycache          all                    default
storage2  secondarycache        all                    default
storage2  usedbysnapshots       0                      -
storage2  usedbydataset         192K                   -
storage2  usedbychildren        575K                   -
storage2  usedbyrefreservation  0                      -
storage2  logbias               latency                default
storage2  dedup                 off                    default
storage2  mlslabel                                     -
storage2  sync                  standard               default
storage2  refcompressratio      1.00x                  -
storage2  written               192K                   -
storage2  logicalused           41K                    -
storage2  logicalreferenced     9.50K                  -
storage2  volmode               default                default
storage2  filesystem_limit      none                   default
storage2  snapshot_limit        none                   default
storage2  filesystem_count      none                   default
storage2  snapshot_count        none                   default
storage2  redundant_metadata    all                    default


# zdb storage2

Cached configuration:
        version: 5000
        name: 'storage2'
        state: 0
        txg: 349
        pool_guid: 16228745912224404975
        hostname: ''
        vdev_children: 1
        vdev_tree:
            type: 'root'
            id: 0
            guid: 16228745912224404975
            create_txg: 4
            children[0]:
                type: 'raidz'
                id: 0
                guid: 1307452363419103148
                nparity: 2
                metaslab_array: 34
                metaslab_shift: 38
                ashift: 12
                asize: 48009350479872
                is_log: 0
                create_txg: 4
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 5260468480490112749
                    path: '/dev/ada0'
                    phys_path: '/dev/ada0'
                    whole_disk: 1
                    create_txg: 4
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 4598430704244637354
                    path: '/dev/ada1'
                    phys_path: '/dev/ada1'
                    whole_disk: 1
                    create_txg: 4
                children[2]:
                    type: 'disk'
                    id: 2
                    guid: 16647901076271785395
                    path: '/dev/ada2'
                    phys_path: '/dev/ada2'
                    whole_disk: 1
                    create_txg: 4
                children[3]:
                    type: 'disk'
                    id: 3
                    guid: 6043875109471819040
                    path: '/dev/ada3'
                    phys_path: '/dev/ada3'
                    whole_disk: 1
                    create_txg: 4
                children[4]:
                    type: 'disk'
                    id: 4
                    guid: 7329569051287925227
                    path: '/dev/ada4'
                    phys_path: '/dev/ada4'
                    whole_disk: 1
                    create_txg: 4
                children[5]:
                    type: 'disk'
                    id: 5
                    guid: 3532287443299098846
                    path: '/dev/ada5'
                    phys_path: '/dev/ada5'
                    whole_disk: 1
                    create_txg: 4
        features_for_read:
            com.delphix:hole_birth
            com.delphix:embedded_data

MOS Configuration:
        version: 5000
        name: 'storage2'
        state: 0
        txg: 349
        pool_guid: 16228745912224404975
        hostname: ''
        vdev_children: 1
        vdev_tree:
            type: 'root'
            id: 0
            guid: 16228745912224404975
            create_txg: 4
            children[0]:
                type: 'raidz'
                id: 0
                guid: 1307452363419103148
                nparity: 2
                metaslab_array: 34
                metaslab_shift: 38
                ashift: 12
                asize: 48009350479872
                is_log: 0
                create_txg: 4
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 5260468480490112749
                    path: '/dev/ada0'
                    phys_path: '/dev/ada0'
                    whole_disk: 1
                    create_txg: 4
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 4598430704244637354
                    path: '/dev/ada1'
                    phys_path: '/dev/ada1'
                    whole_disk: 1
                    create_txg: 4
                children[2]:
                    type: 'disk'
                    id: 2
                    guid: 16647901076271785395
                    path: '/dev/ada2'
                    phys_path: '/dev/ada2'
                    whole_disk: 1
                    create_txg: 4
                children[3]:
                    type: 'disk'
                    id: 3
                    guid: 6043875109471819040
                    path: '/dev/ada3'
                    phys_path: '/dev/ada3'
                    whole_disk: 1
                    create_txg: 4
                children[4]:
                    type: 'disk'
                    id: 4
                    guid: 7329569051287925227
                    path: '/dev/ada4'
                    phys_path: '/dev/ada4'
                    whole_disk: 1
                    create_txg: 4
                children[5]:
                    type: 'disk'
                    id: 5
                    guid: 3532287443299098846
                    path: '/dev/ada5'
                    phys_path: '/dev/ada5'
                    whole_disk: 1
                    create_txg: 4
        features_for_read:
            com.delphix:hole_birth
            com.delphix:embedded_data

Uberblock:
        magic = 0000000000bab10c
        version = 5000
        txg = 349
        guid_sum = 5608497919580231886
        timestamp = 1443188614 UTC = Fri Sep 25 13:43:34 2015

All DDTs are empty

Metaslabs:
        vdev          0
        metaslabs   174   offset                spacemap          free
        ---------------   -------------------   ---------------   -------------
        metaslab      0   offset            0   spacemap     37   free     256G
        On-disk histogram:              fragmentation 0
                         13:      4 ****
                         14:      0
                         15:      1 *
                         16:      0
                         17:      0
                         18:      0
                         19:      0
                         20:      1 *
                         21:      0
                         22:      2 **
                         23:      0
                         24:      0
                         25:      0
                         26:      0
                         27:      0
                         28:      0
                         29:      0
                         30:      0
                         31:      0
                         32:      0
                         33:      0
                         34:      0
                         35:      0
                         36:      0
                         37:      1 *
        metaslab      1   offset   4000000000   spacemap      0   free     256G
        metaslab      2   offset   8000000000   spacemap      0   free     256G
        metaslab      3   offset   c000000000   spacemap      0   free     256G
        metaslab      4   offset  10000000000   spacemap      0   free     256G
        metaslab      5   offset  14000000000   spacemap      0   free     256G
        metaslab      6   offset  18000000000   spacemap      0   free     256G
        metaslab      7   offset  1c000000000   spacemap      0   free     256G
        metaslab      8   offset  20000000000   spacemap      0   free     256G
        metaslab      9   offset  24000000000   spacemap      0   free     256G
        metaslab     10   offset  28000000000   spacemap      0   free     256G
        metaslab     11   offset  2c000000000   spacemap      0   free     256G
        metaslab     12   offset  30000000000   spacemap      0   free     256G
        metaslab     13   offset  34000000000   spacemap      0   free     256G
        metaslab     14   offset  38000000000   spacemap      0   free     256G
        metaslab     15   offset  3c000000000   spacemap      0   free     256G
        metaslab     16   offset  40000000000   spacemap      0   free     256G
        metaslab     17   offset  44000000000   spacemap      0   free     256G
        metaslab     18   offset  48000000000   spacemap      0   free     256G
        metaslab     19   offset  4c000000000   spacemap      0   free     256G
        metaslab     20   offset  50000000000   spacemap      0   free     256G
        metaslab     21   offset  54000000000   spacemap      0   free     256G
        metaslab     22   offset  58000000000   spacemap      0   free     256G
        metaslab     23   offset  5c000000000   spacemap      0   free     256G
        metaslab     24   offset  60000000000   spacemap      0   free     256G
        metaslab     25   offset  64000000000   spacemap      0   free     256G
        metaslab     26   offset  68000000000   spacemap      0   free     256G
        metaslab     27   offset  6c000000000   spacemap      0   free     256G
        metaslab     28   offset  70000000000   spacemap      0   free     256G
        metaslab     29   offset  74000000000   spacemap      0   free     256G
        metaslab     30   offset  78000000000   spacemap      0   free     256G
        metaslab     31   offset  7c000000000   spacemap      0   free     256G
        metaslab     32   offset  80000000000   spacemap      0   free     256G
        metaslab     33   offset  84000000000   spacemap     36   free     256G
        On-disk histogram:              fragmentation 0
                         13:      4 ****
                         14:      0
                         15:      1 *
                         16:      0
                         17:      0
                         18:      0
                         19:      0
                         20:      1 *
                         21:      0
                         22:      2 **
                         23:      0
                         24:      0
                         25:      0
                         26:      0
                         27:      0
                         28:      0
                         29:      0
                         30:      0
                         31:      0
                         32:      0
                         33:      0
                         34:      0
                         35:      0
                         36:      0
                         37:      1 *
        metaslab     34   offset  88000000000   spacemap      0   free     256G
        metaslab     35   offset  8c000000000   spacemap      0   free     256G
        metaslab     36   offset  90000000000   spacemap      0   free     256G
        metaslab     37   offset  94000000000   spacemap      0   free     256G
        metaslab     38   offset  98000000000   spacemap      0   free     256G
        metaslab     39   offset  9c000000000   spacemap      0   free     256G
        metaslab     40   offset  a0000000000   spacemap      0   free     256G
        metaslab     41   offset  a4000000000   spacemap      0   free     256G
        metaslab     42   offset  a8000000000   spacemap      0   free     256G
        metaslab     43   offset  ac000000000   spacemap      0   free     256G
        metaslab     44   offset  b0000000000   spacemap      0   free     256G
        metaslab     45   offset  b4000000000   spacemap      0   free     256G
        metaslab     46   offset  b8000000000   spacemap      0   free     256G
        metaslab     47   offset  bc000000000   spacemap      0   free     256G
        metaslab     48   offset  c0000000000   spacemap      0   free     256G
        metaslab     49   offset  c4000000000   spacemap      0   free     256G
        metaslab     50   offset  c8000000000   spacemap      0   free     256G
        metaslab     51   offset  cc000000000   spacemap      0   free     256G
        metaslab     52   offset  d0000000000   spacemap      0   free     256G
        metaslab     53   offset  d4000000000   spacemap      0   free     256G
        metaslab     54   offset  d8000000000   spacemap      0   free     256G
        metaslab     55   offset  dc000000000   spacemap      0   free     256G
        metaslab     56   offset  e0000000000   spacemap      0   free     256G
        metaslab     57   offset  e4000000000   spacemap      0   free     256G
        metaslab     58   offset  e8000000000   spacemap      0   free     256G
        metaslab     59   offset  ec000000000   spacemap      0   free     256G
        metaslab     60   offset  f0000000000   spacemap      0   free     256G
        metaslab     61   offset  f4000000000   spacemap      0   free     256G
        metaslab     62   offset  f8000000000   spacemap      0   free     256G
        metaslab     63   offset  fc000000000   spacemap      0   free     256G
        metaslab     64   offset 100000000000   spacemap      0   free     256G
        metaslab     65   offset 104000000000   spacemap      0   free     256G
        metaslab     66   offset 108000000000   spacemap     35   free     256G
        On-disk histogram:              fragmentation 0
                         13:      3 ***
                         14:      0
                         15:      1 *
                         16:      0
                         17:      0
                         18:      0
                         19:      0
                         20:      1 *
                         21:      0
                         22:      2 **
                         23:      0
                         24:      0
                         25:      0
                         26:      0
                         27:      0
                         28:      0
                         29:      0
                         30:      0
                         31:      0
                         32:      0
                         33:      0
                         34:      0
                         35:      0
                         36:      0
                         37:      1 *
        metaslab     67   offset 10c000000000   spacemap      0   free     256G
        metaslab     68   offset 110000000000   spacemap      0   free     256G
        metaslab     69   offset 114000000000   spacemap      0   free     256G
        metaslab     70   offset 118000000000   spacemap      0   free     256G
        metaslab     71   offset 11c000000000   spacemap      0   free     256G
        metaslab     72   offset 120000000000   spacemap      0   free     256G
        metaslab     73   offset 124000000000   spacemap      0   free     256G
        metaslab     74   offset 128000000000   spacemap      0   free     256G
        metaslab     75   offset 12c000000000   spacemap      0   free     256G
        metaslab     76   offset 130000000000   spacemap      0   free     256G
        metaslab     77   offset 134000000000   spacemap      0   free     256G
        metaslab     78   offset 138000000000   spacemap      0   free     256G
        metaslab     79   offset 13c000000000   spacemap      0   free     256G
        metaslab     80   offset 140000000000   spacemap      0   free     256G
        metaslab     81   offset 144000000000   spacemap      0   free     256G
        metaslab     82   offset 148000000000   spacemap      0   free     256G
        metaslab     83   offset 14c000000000   spacemap      0   free     256G
        metaslab     84   offset 150000000000   spacemap      0   free     256G
        metaslab     85   offset 154000000000   spacemap      0   free     256G
        metaslab     86   offset 158000000000   spacemap      0   free     256G
        metaslab     87   offset 15c000000000   spacemap      0   free     256G
        metaslab     88   offset 160000000000   spacemap      0   free     256G
        metaslab     89   offset 164000000000   spacemap      0   free     256G
        metaslab     90   offset 168000000000   spacemap      0   free     256G
        metaslab     91   offset 16c000000000   spacemap      0   free     256G
        metaslab     92   offset 170000000000   spacemap      0   free     256G
        metaslab     93   offset 174000000000   spacemap      0   free     256G
        metaslab     94   offset 178000000000   spacemap      0   free     256G
        metaslab     95   offset 17c000000000   spacemap      0   free     256G
        metaslab     96   offset 180000000000   spacemap      0   free     256G
        metaslab     97   offset 184000000000   spacemap      0   free     256G
        metaslab     98   offset 188000000000   spacemap      0   free     256G
        metaslab     99   offset 18c000000000   spacemap      0   free     256G
        metaslab    100   offset 190000000000   spacemap      0   free     256G
        metaslab    101   offset 194000000000   spacemap      0   free     256G
        metaslab    102   offset 198000000000   spacemap      0   free     256G
        metaslab    103   offset 19c000000000   spacemap      0   free     256G
        metaslab    104   offset 1a0000000000   spacemap      0   free     256G
        metaslab    105   offset 1a4000000000   spacemap      0   free     256G
        metaslab    106   offset 1a8000000000   spacemap      0   free     256G
        metaslab    107   offset 1ac000000000   spacemap      0   free     256G
        metaslab    108   offset 1b0000000000   spacemap      0   free     256G
        metaslab    109   offset 1b4000000000   spacemap      0   free     256G
        metaslab    110   offset 1b8000000000   spacemap      0   free     256G
        metaslab    111   offset 1bc000000000   spacemap      0   free     256G
        metaslab    112   offset 1c0000000000   spacemap      0   free     256G
        metaslab    113   offset 1c4000000000   spacemap      0   free     256G
        metaslab    114   offset 1c8000000000   spacemap      0   free     256G
        metaslab    115   offset 1cc000000000   spacemap      0   free     256G
        metaslab    116   offset 1d0000000000   spacemap      0   free     256G
        metaslab    117   offset 1d4000000000   spacemap      0   free     256G
        metaslab    118   offset 1d8000000000   spacemap      0   free     256G
        metaslab    119   offset 1dc000000000   spacemap      0   free     256G
        metaslab    120   offset 1e0000000000   spacemap      0   free     256G
        metaslab    121   offset 1e4000000000   spacemap      0   free     256G
        metaslab    122   offset 1e8000000000   spacemap      0   free     256G
        metaslab    123   offset 1ec000000000   spacemap      0   free     256G
        metaslab    124   offset 1f0000000000   spacemap      0   free     256G
        metaslab    125   offset 1f4000000000   spacemap      0   free     256G
        metaslab    126   offset 1f8000000000   spacemap      0   free     256G
        metaslab    127   offset 1fc000000000   spacemap      0   free     256G
        metaslab    128   offset 200000000000   spacemap      0   free     256G
        metaslab    129   offset 204000000000   spacemap      0   free     256G
        metaslab    130   offset 208000000000   spacemap      0   free     256G
        metaslab    131   offset 20c000000000   spacemap      0   free     256G
        metaslab    132   offset 210000000000   spacemap      0   free     256G
        metaslab    133   offset 214000000000   spacemap      0   free     256G
        metaslab    134   offset 218000000000   spacemap      0   free     256G
        metaslab    135   offset 21c000000000   spacemap      0   free     256G
        metaslab    136   offset 220000000000   spacemap      0   free     256G
        metaslab    137   offset 224000000000   spacemap      0   free     256G
        metaslab    138   offset 228000000000   spacemap      0   free     256G
        metaslab    139   offset 22c000000000   spacemap      0   free     256G
        metaslab    140   offset 230000000000   spacemap      0   free     256G
        metaslab    141   offset 234000000000   spacemap      0   free     256G
        metaslab    142   offset 238000000000   spacemap      0   free     256G
        metaslab    143   offset 23c000000000   spacemap      0   free     256G
        metaslab    144   offset 240000000000   spacemap      0   free     256G
        metaslab    145   offset 244000000000   spacemap      0   free     256G
        metaslab    146   offset 248000000000   spacemap      0   free     256G
        metaslab    147   offset 24c000000000   spacemap      0   free     256G
        metaslab    148   offset 250000000000   spacemap      0   free     256G
        metaslab    149   offset 254000000000   spacemap      0   free     256G
        metaslab    150   offset 258000000000   spacemap      0   free     256G
        metaslab    151   offset 25c000000000   spacemap      0   free     256G
        metaslab    152   offset 260000000000   spacemap      0   free     256G
        metaslab    153   offset 264000000000   spacemap      0   free     256G
        metaslab    154   offset 268000000000   spacemap      0   free     256G
        metaslab    155   offset 26c000000000   spacemap      0   free     256G
        metaslab    156   offset 270000000000   spacemap      0   free     256G
        metaslab    157   offset 274000000000   spacemap      0   free     256G
        metaslab    158   offset 278000000000   spacemap      0   free     256G
        metaslab    159   offset 27c000000000   spacemap      0   free     256G
        metaslab    160   offset 280000000000   spacemap      0   free     256G
        metaslab    161   offset 284000000000   spacemap      0   free     256G
        metaslab    162   offset 288000000000   spacemap      0   free     256G
        metaslab    163   offset 28c000000000   spacemap      0   free     256G
        metaslab    164   offset 290000000000   spacemap      0   free     256G
        metaslab    165   offset 294000000000   spacemap      0   free     256G
        metaslab    166   offset 298000000000   spacemap      0   free     256G
        metaslab    167   offset 29c000000000   spacemap      0   free     256G
        metaslab    168   offset 2a0000000000   spacemap      0   free     256G
        metaslab    169   offset 2a4000000000   spacemap      0   free     256G
        metaslab    170   offset 2a8000000000   spacemap      0   free     256G
        metaslab    171   offset 2ac000000000   spacemap      0   free     256G
        metaslab    172   offset 2b0000000000   spacemap      0   free     256G
        metaslab    173   offset 2b4000000000   spacemap      0   free     256G

        vdev          0         metaslabs  174          fragmentation  0%
                         13:     11 ***********
                         14:      0
                         15:      3 ***
                         16:      0
                         17:      0
                         18:      0
                         19:      0
                         20:      3 ***
                         21:      0
                         22:      6 ******
                         23:      0
                         24:      0
                         25:      0
                         26:      0
                         27:      0
                         28:      0
                         29:      0
                         30:      0
                         31:      0
                         32:      0
                         33:      0
                         34:      0
                         35:      0
                         36:      0
                         37:      3 ***
        pool storage2   fragmentation     0%
                         13:     11 ***********
                         14:      0
                         15:      3 ***
                         16:      0
                         17:      0
                         18:      0
                         19:      0
                         20:      3 ***
                         21:      0
                         22:      6 ******
                         23:      0
                         24:      0
                         25:      0
                         26:      0
                         27:      0
                         28:      0
                         29:      0
                         30:      0
                         31:      0
                         32:      0
                         33:      0
                         34:      0
                         35:      0
                         36:      0
                         37:      3 ***
Dataset mos [META], ID 0, cr_txg 4, 575K, 37 objects

    Object  lvl   iblk   dblk  dsize  lsize   %full  type
         0    1    16K    16K  48.0K    32K   57.81  DMU dnode
         1    1    16K     1K  24.0K     1K  100.00  object directory
         2    1    16K    512      0    512    0.00  DSL directory
         3    1    16K    512      0    512  100.00  DSL props
         4    1    16K    512      0    512  100.00  DSL directory child map
         5    1    16K    512      0    512    0.00  DSL directory
         6    1    16K    512      0    512  100.00  DSL props
         7    1    16K    512      0    512  100.00  DSL directory child map
         8    1    16K    512      0    512    0.00  DSL directory
         9    1    16K    512      0    512  100.00  DSL props
        10    1    16K    512      0    512  100.00  DSL directory child map
        11    1    16K   128K      0   128K    0.00  bpobj
        12    1    16K    512      0    512    0.00  DSL directory
        13    1    16K    512      0    512  100.00  DSL props
        14    1    16K    512      0    512  100.00  DSL directory child map
        15    1    16K    512      0    512    0.00  DSL dataset
        16    1    16K    512      0    512  100.00  DSL dataset snap map
        17    1    16K    512      0    512  100.00  DSL deadlist map
        18    1    16K    512      0    512    0.00  DSL dataset
        19    1    16K    512      0    512  100.00  DSL deadlist map
        20    1    16K   128K      0   128K    0.00  bpobj
        21    1    16K    512      0    512    0.00  DSL dataset
        22    1    16K    512      0    512  100.00  DSL dataset snap map
        23    1    16K    512      0    512  100.00  DSL deadlist map
        24    1    16K   128K      0   128K    0.00  bpobj
        25    1    16K    512      0    512  100.00  DSL dataset next clones
        26    1    16K    512      0    512  100.00  DSL dir clones
        27    1    16K    16K  24.0K    16K  100.00  packed nvlist
        28    1    16K    512  24.0K    512  100.00  zap
        29    1    16K    512  24.0K    512  100.00  zap
        30    1    16K    16K  48.0K    32K  100.00  zap
        31    1    16K    16K  24.0K    16K  100.00  bpobj (Z=uncompressed)
        32    1    16K   128K  24.0K   128K  100.00  SPA history
        33    1    16K    512  24.0K    512  100.00  zap
        34    1    16K    512      0     1K  100.00  object array
        35    2    16K     4K  96.0K  12.0K  100.00  SPA space map
        36    2    16K     4K  96.0K  12.0K  100.00  SPA space map
        37    2    16K     4K  96.0K  12.0K  100.00  SPA space map

Dataset storage2 [ZPL], ID 21, cr_txg 1, 192K, 7 objects

    Object  lvl   iblk   dblk  dsize  lsize   %full  type
         0    7    16K    16K   112K    16K   21.88  DMU dnode
        -1    1    16K    512      0    512  100.00  ZFS user/group used
        -2    1    16K    512      0    512  100.00  ZFS user/group used
         1    1    16K     1K    16K     1K  100.00  ZFS master node
         2    1    16K    512      0    512  100.00  SA master node
         3    1    16K    512      0    512  100.00  ZFS delete queue
         4    1    16K    512      0    512  100.00  ZFS directory
         5    1    16K  1.50K    16K  1.50K  100.00  SA attr registration
         6    1    16K    16K    32K    32K  100.00  SA attr layouts
         7    1    16K    512      0    512  100.00  ZFS directory

Verified large_blocks feature refcount is correct (0)

Traversing all blocks to verify checksums and verify nothing leaked ...

loading space map for vdev 0 of 1, metaslab 66 of 174 ...

        No leaks (block sum matches space maps exactly)

        bp count:              67
        ganged count:           0
        bp logical:        578048      avg:   8627
        bp physical:        48128      avg:    718     compression:  12.01
        bp allocated:     1474560      avg:  22008     compression:   0.39
        bp deduped:             0    ref>1:      0   deduplication:   1.00
        SPA allocated:    1474560     used:  0.00%

        additional, non-pointer bps of type 0:         23
        Dittoed blocks on same vdev: 44

Blocks  LSIZE   PSIZE   ASIZE     avg    comp   %Total  Type
     -      -       -       -       -       -        -  unallocated
     1     1K     512   36.0K   36.0K    2.00     2.50  object directory
     -      -       -       -       -       -        -  object array
     1    16K      1K   36.0K   36.0K   16.00     2.50  packed nvlist
     -      -       -       -       -       -        -  packed nvlist size
     1    16K     512   36.0K   36.0K   32.00     2.50  bpobj
     -      -       -       -       -       -        -  bpobj header
     -      -       -       -       -       -        -  SPA space map header
    12  84.0K   19.5K    432K   36.0K    4.31    30.00  SPA space map
     -      -       -       -       -       -        -  ZIL intent log
     9   144K   6.50K    240K   26.7K   22.15    16.67  DMU dnode
     2     4K      1K   60.0K   30.0K    4.00     4.17  DMU objset
     -      -       -       -       -       -        -  DSL directory
     -      -       -       -       -       -        -  DSL directory child map
     -      -       -       -       -       -        -  DSL dataset snap map
     -      -       -       -       -       -        -  DSL props
     -      -       -       -       -       -        -  DSL dataset
     -      -       -       -       -       -        -  ZFS znode
     -      -       -       -       -       -        -  ZFS V0 ACL
     -      -       -       -       -       -        -  ZFS plain file
     -      -       -       -       -       -        -  ZFS directory
     1     1K     512   24.0K   24.0K    2.00     1.67  ZFS master node
     -      -       -       -       -       -        -  ZFS delete queue
     -      -       -       -       -       -        -  zvol object
     -      -       -       -       -       -        -  zvol prop
     -      -       -       -       -       -        -  other uint8[]
     -      -       -       -       -       -        -  other uint64[]
     -      -       -       -       -       -        -  other ZAP
     -      -       -       -       -       -        -  persistent error log
     1   128K      1K   36.0K   36.0K   128.00    2.50  SPA history
     -      -       -       -       -       -        -  SPA history offsets
     -      -       -       -       -       -        -  Pool properties
     -      -       -       -       -       -        -  DSL permissions
     -      -       -       -       -       -        -  ZFS ACL
     -      -       -       -       -       -        -  ZFS SYSACL
     -      -       -       -       -       -        -  FUID table
     -      -       -       -       -       -        -  FUID table size
     -      -       -       -       -       -        -  DSL dataset next clones
     -      -       -       -       -       -        -  scan work queue
     -      -       -       -       -       -        -  ZFS user/group used
     -      -       -       -       -       -        -  ZFS user/group quota
     -      -       -       -       -       -        -  snapshot refcount tags
     -      -       -       -       -       -        -  DDT ZAP algorithm
     -      -       -       -       -       -        -  DDT statistics
     -      -       -       -       -       -        -  System attributes
     -      -       -       -       -       -        -  SA master node
     1  1.50K     512   24.0K   24.0K    3.00     1.67  SA attr registration
     2    32K      4K   48.0K   24.0K    8.00     3.33  SA attr layouts
     -      -       -       -       -       -        -  scan translations
     -      -       -       -       -       -        -  deduplicated block
     -      -       -       -       -       -        -  DSL deadlist map
     -      -       -       -       -       -        -  DSL deadlist map hdr
     -      -       -       -       -       -        -  DSL dir clones
     -      -       -       -       -       -        -  bpobj subobj
     8  92.0K   6.00K    288K   36.0K   15.33    20.00  deferred free
     -      -       -       -       -       -        -  dedup ditto
     5  33.5K   6.00K    180K   36.0K    5.58    12.50  other
    67   565K   47.0K   1.41M   21.5K   12.01   100.00  Total

                            capacity   operations   bandwidth  ---- errors ----
description                used avail  read write  read write  read write cksum
storage2                  1.41M 43.5T   169     0  676K     0     0     0     0
  raidz2                  1.41M 43.5T   169     0  676K     0     0     0     0
    /dev/ada0                           207     0 1.66M     0     0     0     0
    /dev/ada1                           203     0 1.23M     0     0     0     0
    /dev/ada2                           225     0 1.31M     0     0     0     0
    /dev/ada3                           196     0 1.20M     0     0     0     0
    /dev/ada4                           196     0 1.20M     0     0     0     0
    /dev/ada5                           214     0 1.27M     0     0     0     0

History:
unrecognized record:
  history internal str: 'pool version 5000; software version 5000/5; uts  10.2-STABLE 1002500 amd64'
  internal_name: 'create'
  history txg: 5
  history time: 1443187631
  history hostname: ''
Comment 2 Steven Hartland freebsd_committer freebsd_triage 2015-09-27 17:06:09 UTC
Could you try a current / head kernel + world to see if this has already been addressed?
Comment 3 Pekka Järvinen 2015-10-25 10:18:31 UTC
I can't try current / head kernel + world for now.

It seems to be fixed somewhere between r287260M and r287929 AFAIK.