Bug 274605 - sysutils/arc_summary Error in sys.excepthook: ... KeyError: 'meta'
Summary: sysutils/arc_summary Error in sys.excepthook: ... KeyError: 'meta'
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Joseph Mingrone
URL:
Keywords:
: 275732 (view as bug list)
Depends on:
Blocks:
 
Reported: 2023-10-20 08:58 UTC by Vladyslav V. Prodan
Modified: 2023-12-15 13:33 UTC (History)
2 users (show)

See Also:
bugzilla: maintainer-feedback? (jrm)


Attachments
truss output for failing arc_summary on 13.2-STABLE (1.17 KB, text/plain)
2023-11-24 20:05 UTC, John F. Carr
no flags Details
truss output from failing system (259.82 KB, text/plain)
2023-11-24 20:07 UTC, John F. Carr
no flags Details
value of arc_stats variable in function section_arc (4.02 KB, text/plain)
2023-11-25 15:14 UTC, John F. Carr
no flags Details
output of arc_summary on 13.1 (29.01 KB, text/plain)
2023-12-15 08:47 UTC, Emanuel Haupt
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Vladyslav V. Prodan 2023-10-20 08:58:23 UTC
# arc_summary -d

------------------------------------------------------------------------
ZFS Subsystem Report                            Fri Oct 06 21:35:38 2023
FreeBSD 13.2-STABLE                                        zpl version 5
Machine: core.domain.com (amd64)                        spa version 5000

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/local/bin/arc_summary", line 1034, in <module>
    main()
  File "/usr/local/bin/arc_summary", line 1028, in main
    section_calls[section](kstats)
  File "/usr/local/bin/arc_summary", line 554, in section_arc
    meta = arc_stats['meta']
KeyError: 'meta'

Original exception was:
Traceback (most recent call last):
  File "/usr/local/bin/arc_summary", line 1034, in <module>
    main()
  File "/usr/local/bin/arc_summary", line 1028, in main
    section_calls[section](kstats)
  File "/usr/local/bin/arc_summary", line 554, in section_arc
    meta = arc_stats['meta']
KeyError: 'meta'


# uname -aKU
FreeBSD core.domain.com 13.2-STABLE FreeBSD 13.2-STABLE 74614fa93 SUPPORT-13-2-0 amd64 1302507 1302507
Comment 1 Joseph Mingrone freebsd_committer freebsd_triage 2023-10-29 13:35:05 UTC
Hi,

Could you share the arc_summary version here?  Some fixes were pulled in with the update to 20230609.

So far, I haven't been able to reproduce any issues in either 13.2-RELEASE or 14.0-RC1 jails.
Comment 2 Vladyslav V. Prodan 2023-10-30 10:49:02 UTC
(In reply to Joseph Mingrone from comment #1)

I downloaded the latest version of the arc_summary file.

# cd /usr/local/bin/ && wget https://raw.githubusercontent.com/openzfs/zfs/master/cmd/arc_summary && chmod +x arc_summary

# diff /usr/local/bin/arc_summary /usr/local/bin/arc_summary.orig
1c1
< #!/usr/bin/env python3
---
> #!/usr/local/bin/python3.9
714c714
<     prt_2('ARC prefetch data accesses:', f_perc(pd_total, all_accesses),
---
>     prt_2('ARC prefetch metadata accesses:', f_perc(pd_total, all_accesses),

And run:

# arc_summary -d

------------------------------------------------------------------------
ZFS Subsystem Report                            Mon Oct 30 12:45:45 2023
FreeBSD 13.2-STABLE                                        zpl version 5
Machine: core.domain.com (amd64)                        spa version 5000

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/local/bin/arc_summary", line 1034, in <module>
    main()
  File "/usr/local/bin/arc_summary", line 1028, in main
    section_calls[section](kstats)
  File "/usr/local/bin/arc_summary", line 554, in section_arc
    meta = arc_stats['meta']
KeyError: 'meta'

Original exception was:
Traceback (most recent call last):
  File "/usr/local/bin/arc_summary", line 1034, in <module>
    main()
  File "/usr/local/bin/arc_summary", line 1028, in main
    section_calls[section](kstats)
  File "/usr/local/bin/arc_summary", line 554, in section_arc
    meta = arc_stats['meta']
KeyError: 'meta'
Comment 3 Vladyslav V. Prodan 2023-10-30 10:50:21 UTC
(In reply to Vladyslav V. Prodan from comment #2)

# python -V
Python 3.9.18

# python3 -V
Python 3.9.18
Comment 4 Joseph Mingrone freebsd_committer freebsd_triage 2023-11-24 14:14:58 UTC
I'm still not able to reproduce your problem whether I install the FreeBSD package or fetch the script as you did.  Could you open an upstream issue?

root@13amd64-default:~ # uname -a
FreeBSD 13amd64-default 13.2-RELEASE-p2 FreeBSD 13.2-RELEASE-p2 amd64

root@13amd64-default:~ # pkg install arc_summary
Updating local repository catalogue...
[13amd64-default] Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
[13amd64-default] Fetching packagesite.pkg: 100%  113 KiB 115.4kB/s    00:01
Processing entries: 100%
local repository update completed. 421 packages processed.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 9 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        arc_summary: 20231023
        gettext-runtime: 0.22.3
        indexinfo: 0.3.1
        libffi: 3.4.4
        mpdecimal: 2.5.1
        py39-setuptools: 63.1.0_1
        py39-sysctl: 0.3.3.20210928
        python39: 3.9.18
        readline: 8.2.1

Number of packages to be installed: 9

The process will require 133 MiB more space.

Proceed with this action? [y/N]: y
[13amd64-default] [1/9] Installing indexinfo-0.3.1...
[13amd64-default] [1/9] Extracting indexinfo-0.3.1: 100%
[13amd64-default] [2/9] Installing mpdecimal-2.5.1...
[13amd64-default] [2/9] Extracting mpdecimal-2.5.1: 100%
[13amd64-default] [3/9] Installing libffi-3.4.4...
[13amd64-default] [3/9] Extracting libffi-3.4.4: 100%
[13amd64-default] [4/9] Installing readline-8.2.1...
[13amd64-default] [4/9] Extracting readline-8.2.1: 100%
[13amd64-default] [5/9] Installing gettext-runtime-0.22.3...
[13amd64-default] [5/9] Extracting gettext-runtime-0.22.3: 100%
[13amd64-default] [6/9] Installing python39-3.9.18...
[13amd64-default] [6/9] Extracting python39-3.9.18: 100%
[13amd64-default] [7/9] Installing py39-setuptools-63.1.0_1...
[13amd64-default] [7/9] Extracting py39-setuptools-63.1.0_1: 100%
[13amd64-default] [8/9] Installing py39-sysctl-0.3.3.20210928...
[13amd64-default] [8/9] Extracting py39-sysctl-0.3.3.20210928: 100%
[13amd64-default] [9/9] Installing arc_summary-20231023...
[13amd64-default] [9/9] Extracting arc_summary-20231023: 100%
=====
Message from python39-3.9.18:

--
Note that some standard Python modules are provided as separate ports
as they require additional dependencies. They are available as:

py39-gdbm       databases/py-gdbm@py39
py39-sqlite3    databases/py-sqlite3@py39
py39-tkinter    x11-toolkits/py-tkinter@py39

root@13amd64-default:~ # arc_summary

------------------------------------------------------------------------
ZFS Subsystem Report                            Fri Nov 24 14:10:20 2023
FreeBSD 13.2-RELEASE-p2                                    zpl version 5
Machine: 13amd64-default (amd64)                        spa version 5000

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    46.8 %   29.3 GiB
        Target size (adaptive):                        53.0 %   33.2 GiB
        Min size (hard limit):                          3.2 %    2.0 GiB
        Max size (high water):                           31:1   62.7 GiB
        Anonymous data size:                            0.0 %    0 Bytes
        Anonymous metadata size:                      < 0.1 %  384.0 KiB
        MFU data target:                               45.2 %   12.0 GiB
        MFU data size:                                  8.1 %    2.1 GiB
        MFU ghost data size:                                    28.1 MiB
        MFU metadata target:                           32.1 %    8.5 GiB
        MFU metadata size:                             28.8 %    7.7 GiB
        MFU ghost metadata size:                                 0 Bytes
        MRU data target:                               10.3 %    2.7 GiB
        MRU data size:                                 62.0 %   16.5 GiB
        MRU ghost data size:                                     1.2 GiB
        MRU metadata target:                           12.5 %    3.3 GiB
        MRU metadata size:                              1.1 %  308.8 MiB
        MRU ghost metadata size:                                 1.1 MiB
        Uncached data size:                             0.0 %    0 Bytes
        Uncached metadata size:                         0.0 %    0 Bytes
        Bonus size:                                     1.5 %  449.4 MiB
        Dnode cache target:                            10.0 %    6.3 GiB
        Dnode cache size:                              18.0 %    1.1 GiB
        Dbuf size:                                      1.5 %  462.0 MiB
        Header size:                                    1.5 %  453.7 MiB
        L2 header size:                                 0.0 %    0 Bytes
        ABD chunk waste size:                           0.9 %  275.8 MiB

ARC hash breakdown:
        Elements max:                                               2.8M
        Elements current:                              74.0 %       2.0M
        Collisions:                                                 6.2M
        Chain max:                                                     6
        Chains:                                                   210.1k

ARC misc:
        Deleted:                                                    5.9M
        Mutex misses:                                               2.2k
        Eviction skips:                                               57
        Eviction skips due to L2 writes:                               0
        L2 cached evictions:                                     0 Bytes
        L2 eligible evictions:                                 213.9 GiB
        L2 eligible MFU evictions:                     49.9 %  106.8 GiB
        L2 eligible MRU evictions:                     50.1 %  107.1 GiB
        L2 ineligible evictions:                               148.5 MiB

ARC total accesses:                                                 2.3G
        Total hits:                                    99.6 %       2.3G
        Total I/O hits:                               < 0.1 %     842.7k
        Total misses:                                   0.3 %       7.5M

ARC demand data accesses:                              17.2 %     396.8M
        Demand data hits:                              99.5 %     394.9M
        Demand data I/O hits:                         < 0.1 %       5.3k
        Demand data misses:                             0.5 %       1.9M

ARC demand metadata accesses:                          79.3 %       1.8G
        Demand metadata hits:                          99.9 %       1.8G
        Demand metadata I/O hits:                     < 0.1 %     117.2k
        Demand metadata misses:                         0.1 %     925.7k

ARC prefetch data accesses:                           < 0.1 %     655.3k
        Prefetch data hits:                             4.7 %      30.7k
        Prefetch data I/O hits:                         0.0 %          0
        Prefetch data misses:                          95.3 %     624.6k

ARC prefetch metadata accesses:                         3.4 %      78.4M
        Prefetch metadata hits:                        93.9 %      73.6M
        Prefetch metadata I/O hits:                     0.9 %     720.3k
        Prefetch metadata misses:                       5.2 %       4.1M

ARC predictive prefetches:                             99.1 %      78.4M
        Demand hits after predictive:                  89.4 %      70.1M
        Demand I/O hits after predictive:               0.1 %     115.2k
        Never demanded after predictive:               10.4 %       8.2M

ARC prescient prefetches:                               0.9 %     688.9k
        Demand hits after prescient:                   99.3 %     684.4k
        Demand I/O hits after prescient:                0.6 %       4.4k
        Never demanded after prescient:               < 0.1 %         93

ARC states hits of all accesses:
        Most frequently used (MFU):                    93.9 %       2.2G
        Most recently used (MRU):                       5.8 %     132.7M
        Most frequently used (MFU) ghost:             < 0.1 %     922.0k
        Most recently used (MRU) ghost:               < 0.1 %     107.5k
        Uncached:                                       0.0 %          0

DMU predictive prefetcher calls:                                    9.5M
        Stream hits:                                   56.1 %       5.3M
        Stream misses:                                 43.9 %       4.2M
        Streams limit reached:                         89.7 %       3.7M
        Prefetches issued                                         656.4k

L2ARC not detected, skipping section

Tunables:
        abd_scatter_enabled                                            1
        abd_scatter_min_size                                        4097
        abort_size                                                131072
        active_allocator                                         dynamic
        allow_redacted_dataset_mount                                   0
        anon_data_esize                                                0
        anon_metadata_esize                                            0
        anon_size                                                 393216
        arc.average_blocksize                                       8192
        arc.dnode_limit                                                0
        arc.dnode_limit_percent                                       10
        arc.dnode_reduce_percent                                      10
        arc.evict_batch_limit                                         10
        arc.eviction_pct                                             200
        arc.free_target                                           346186
        arc.grow_retry                                                 0
        arc.lotsfree_percent                                          10
        arc.max                                                        0
        arc.meta_balance                                             500
        arc.min                                                        0
        arc.min_prefetch_ms                                            0
        arc.min_prescient_prefetch_ms                                  0
        arc.no_grow_shift                                              5
        arc.pc_percent                                                 0
        arc.prune_task_threads                                         1
        arc.shrink_shift                                               0
        arc.sys_free                                                   0
        arc_free_target                                           346186
        arc_max                                                        0
        arc_min                                                        0
        arc_no_grow_shift                                              5
        async_block_max_blocks                      18446744073709551615
        autoimport_disable                                             1
        bclone_enabled                                                 1
        blake3_impl             cycle [fastest] generic sse2 sse41 avx2
        brt.prefetch                                                   1
        btree_verify_intensity                                         0
        ccw_retry_interval                                           300
        checksum_events_per_second                                    20
        commit_timeout_pct                                             5
        compressed_arc_enabled                                         1
        condense.indirect_commit_entry_delay_ms                        0
        condense.indirect_obsolete_pct                                25
        condense.indirect_vdevs_enable                                 1
        condense.max_obsolete_bytes                           1073741824
        condense.min_mapping_bytes                                131072
        condense_pct                                                 200
        crypt_sessions                                                 0
        dbgmsg_enable                                                  1
        dbgmsg_maxsize                                           4194304
        dbuf.cache_shift                                               5
        dbuf.metadata_cache_max_bytes               18446744073709551615
        dbuf.metadata_cache_shift                                      6
        dbuf.mutex_cache_shift                                         0
        dbuf_cache.hiwater_pct                                        10
        dbuf_cache.lowater_pct                                        10
        dbuf_cache.max_bytes                        18446744073709551615
        dbuf_state_index                                               0
        ddt_data_is_special                                            1
        deadman.checktime_ms                                       60000
        deadman.enabled                                                1
        deadman.failmode                                            wait
        deadman.synctime_ms                                       600000
        deadman.ziotime_ms                                        300000
        debug                                                          0
        debugflags                                                     0
        dedup.ddt_zap_default_bs                                      15
        dedup.ddt_zap_default_ibs                                     15
        dedup.prefetch                                                 0
        default_bs                                                     9
        default_ibs                                                   17
        delay_min_dirty_percent                                       60
        delay_scale                                               500000
        dirty_data_max                                        4294967296
        dirty_data_max_max                                    4294967296
        dirty_data_max_max_percent                                    25
        dirty_data_max_percent                                        10
        dirty_data_sync_percent                                       20
        disable_ivset_guid_check                                       0
        dmu_object_alloc_chunk_shift                                   7
        dmu_offset_next_sync                                           1
        dmu_prefetch_max                                       134217728
        dtl_sm_blksz                                                4096
        earlyabort_pass                                                1
        embedded_slog_min_ms                                          64
        flags                                                          0
        fletcher_4_impl [fastest] scalar superscalar superscalar4 sse2 ssse3 avx2
        free_bpobj_enabled                                             1
        free_leak_on_eio                                               0
        free_min_time_ms                                            1000
        history_output_max                                       1048576
        immediate_write_sz                                         32768
        initialize_chunk_size                                    1048576
        initialize_value                            16045690984833335022
        keep_log_spacemaps_at_export                                   0
        l2arc.exclude_special                                          0
        l2arc.feed_again                                               1
        l2arc.feed_min_ms                                            200
        l2arc.feed_secs                                                1
        l2arc.headroom                                                 2
        l2arc.headroom_boost                                         200
        l2arc.meta_percent                                            33
        l2arc.mfuonly                                                  0
        l2arc.noprefetch                                               1
        l2arc.norw                                                     0
        l2arc.rebuild_blocks_min_l2size                       1073741824
        l2arc.rebuild_enabled                                          1
        l2arc.trim_ahead                                               0
        l2arc.write_boost                                        8388608
        l2arc.write_max                                          8388608
        l2arc_feed_again                                               1
        l2arc_feed_min_ms                                            200
        l2arc_feed_secs                                                1
        l2arc_headroom                                                 2
        l2arc_headroom_boost                                         200
        l2arc_noprefetch                                               1
        l2arc_norw                                                     0
        l2arc_write_boost                                        8388608
        l2arc_write_max                                          8388608
        l2c_only_size                                                  0
        livelist.condense.new_alloc                                    0
        livelist.condense.sync_cancel                                  0
        livelist.condense.sync_pause                                   0
        livelist.condense.zthr_cancel                                  0
        livelist.condense.zthr_pause                                   0
        livelist.max_entries                                      500000
        livelist.min_percent_shared                                   75
        lua.max_instrlimit                                     100000000
        lua.max_memlimit                                       104857600
        max_async_dedup_frees                                     100000
        max_auto_ashift                                               14
        max_dataset_nesting                                           50
        max_log_walking                                                5
        max_logsm_summary_length                                      10
        max_missing_tvds                                               0
        max_missing_tvds_cachefile                                     2
        max_missing_tvds_scan                                          0
        max_nvlist_src_size                                            0
        max_recordsize                                          16777216
        metaslab.aliquot                                         1048576
        metaslab.bias_enabled                                          1
        metaslab.debug_load                                            0
        metaslab.debug_unload                                          0
        metaslab.df_alloc_threshold                               131072
        metaslab.df_free_pct                                           4
        metaslab.df_max_search                                  16777216
        metaslab.df_use_largest_segment                                0
        metaslab.find_max_tries                                      100
        metaslab.force_ganging                                  16777217
        metaslab.force_ganging_pct                                     3
        metaslab.fragmentation_factor_enabled                          1
        metaslab.fragmentation_threshold                              70
        metaslab.lba_weighting_enabled                                 1
        metaslab.max_size_cache_sec                                 3600
        metaslab.mem_limit                                            25
        metaslab.preload_enabled                                       1
        metaslab.preload_limit                                        10
        metaslab.preload_pct                                          50
        metaslab.segment_weight_enabled                                1
        metaslab.sm_blksz_no_log                                   16384
        metaslab.sm_blksz_with_log                                131072
        metaslab.switch_threshold                                      2
        metaslab.try_hard_before_gang                                  0
        metaslab.unload_delay                                         32
        metaslab.unload_delay_ms                                  600000
        mfu_data_esize                                        2121855488
        mfu_ghost_data_esize                                    29432320
        mfu_ghost_metadata_esize                                       0
        mfu_ghost_size                                          29432320
        mfu_metadata_esize                                    6428061696
        mfu_size                                             10537568768
        mg.fragmentation_threshold                                    95
        mg.noalloc_threshold                                           0
        min_auto_ashift                                               12
        min_metaslabs_to_flush                                         1
        mru_data_esize                                       16038855680
        mru_ghost_data_esize                                  1332094976
        mru_ghost_metadata_esize                                 1146880
        mru_ghost_size                                        1333241856
        mru_metadata_esize                                     163487744
        mru_size                                             18015454208
        multihost.fail_intervals                                      10
        multihost.history                                              0
        multihost.import_intervals                                    20
        multihost.interval                                          1000
        multilist_num_sublists                                         0
        no_scrub_io                                                    0
        no_scrub_prefetch                                              0
        nocacheflush                                                   0
        nopwrite_enabled                                               1
        obsolete_min_time_ms                                         500
        pd_bytes_max                                            52428800
        per_txg_dirty_frees_percent                                   30
        prefetch.disable                                               0
        prefetch.max_distance                                   67108864
        prefetch.max_idistance                                  67108864
        prefetch.max_sec_reap                                          2
        prefetch.max_streams                                           8
        prefetch.min_distance                                    4194304
        prefetch.min_sec_reap                                          1
        read_history                                                   0
        read_history_hits                                              0
        rebuild_max_segment                                      1048576
        rebuild_scrub_enabled                                          1
        rebuild_vdev_limit                                      67108864
        reconstruct.indirect_combinations_max                       4096
        recover                                                        0
        recv.best_effort_corrective                                    0
        recv.queue_ff                                                 20
        recv.queue_length                                       16777216
        recv.write_batch_size                                    1048576
        removal_suspend_progress                                       0
        remove_max_segment                                      16777216
        resilver_disable_defer                                         0
        resilver_min_time_ms                                        3000
        scan_blkstats                                                  0
        scan_checkpoint_intval                                      7200
        scan_fill_weight                                               3
        scan_ignore_errors                                             0
        scan_issue_strategy                                            0
        scan_legacy                                                    0
        scan_max_ext_gap                                         2097152
        scan_mem_lim_fact                                             20
        scan_mem_lim_soft_fact                                        20
        scan_report_txgs                                               0
        scan_strict_mem_lim                                            0
        scan_suspend_progress                                          0
        scan_vdev_limit                                         16777216
        scrub_error_blocks_per_txg                                  4096
        scrub_min_time_ms                                           1000
        send.corrupt_data                                              0
        send.no_prefetch_queue_ff                                     20
        send.no_prefetch_queue_length                            1048576
        send.override_estimate_recordsize                              0
        send.queue_ff                                                 20
        send.queue_length                                       16777216
        send.unmodified_spill_blocks                                   1
        send_holes_without_birth_time                                  1
        sha256_impl          cycle [fastest] generic x64 ssse3 avx avx2
        sha512_impl                cycle [fastest] generic x64 avx avx2
        slow_io_events_per_second                                     20
        snapshot_history_enabled                                       1
        spa.asize_inflation                                           24
        spa.discard_memory_limit                                16777216
        spa.load_print_vdev_tree                                       0
        spa.load_verify_data                                           1
        spa.load_verify_metadata                                       1
        spa.load_verify_shift                                          4
        spa.slop_shift                                                 5
        spa.upgrade_errlog_limit                                       0
        space_map_ibs                                                 14
        special_class_metadata_reserve_pct                            25
        standard_sm_blksz                                         131072
        super_owner                                                    0
        sync_pass_deferred_free                                        2
        sync_pass_dont_compress                                        8
        sync_pass_rewrite                                              2
        sync_taskq_batch_pct                                          75
        top_maxinflight                                             1000
        traverse_indirect_prefetch_limit                              32
        trim.extent_bytes_max                                  134217728
        trim.extent_bytes_min                                      32768
        trim.metaslab_skip                                             0
        trim.queue_limit                                              10
        trim.txg_batch                                                32
        txg.history                                                  100
        txg.timeout                                                    5
        uncached_data_esize                                            0
        uncached_metadata_esize                                        0
        uncached_size                                                  0
        unflushed_log_block_max                                   131072
        unflushed_log_block_min                                     1000
        unflushed_log_block_pct                                      400
        unflushed_log_txg_max                                       1000
        unflushed_max_mem_amt                                 1073741824
        unflushed_max_mem_ppm                                       1000
        user_indirect_is_special                                       1
        validate_skip                                                  0
        vdev.aggregation_limit                                   1048576
        vdev.aggregation_limit_non_rotating                       131072
        vdev.async_read_max_active                                     3
        vdev.async_read_min_active                                     1
        vdev.async_write_active_max_dirty_percent                     60
        vdev.async_write_active_min_dirty_percent                     30
        vdev.async_write_max_active                                   10
        vdev.async_write_min_active                                    2
        vdev.bio_delete_disable                                        0
        vdev.bio_flush_disable                                         0
        vdev.def_queue_depth                                          32
        vdev.default_ms_count                                        200
        vdev.default_ms_shift                                         29
        vdev.file.logical_ashift                                       9
        vdev.file.physical_ashift                                      9
        vdev.initializing_max_active                                   1
        vdev.initializing_min_active                                   1
        vdev.max_active                                             1000
        vdev.max_auto_ashift                                          14
        vdev.max_ms_shift                                             34
        vdev.min_auto_ashift                                          12
        vdev.min_ms_count                                             16
        vdev.mirror.non_rotating_inc                                   0
        vdev.mirror.non_rotating_seek_inc                              1
        vdev.mirror.rotating_inc                                       0
        vdev.mirror.rotating_seek_inc                                  5
        vdev.mirror.rotating_seek_offset                         1048576
        vdev.ms_count_limit                                       131072
        vdev.nia_credit                                                5
        vdev.nia_delay                                                 5
        vdev.queue_depth_pct                                        1000
        vdev.read_gap_limit                                        32768
        vdev.rebuild_max_active                                        3
        vdev.rebuild_min_active                                        1
        vdev.removal_ignore_errors                                     0
        vdev.removal_max_active                                        2
        vdev.removal_max_span                                      32768
        vdev.removal_min_active                                        1
        vdev.removal_suspend_progress                                  0
        vdev.remove_max_segment                                 16777216
        vdev.scrub_max_active                                          3
        vdev.scrub_min_active                                          1
        vdev.sync_read_max_active                                     10
        vdev.sync_read_min_active                                     10
        vdev.sync_write_max_active                                    10
        vdev.sync_write_min_active                                    10
        vdev.trim_max_active                                           2
        vdev.trim_min_active                                           1
        vdev.validate_skip                                             0
        vdev.write_gap_limit                                        4096
        version.acl                                                    1
        version.ioctl                                                 15
        version.module                         2.2.99-FreeBSD_g66b81b349
        version.spa                                                 5000
        version.zpl                                                    5
        vnops.read_chunk_size                                    1048576
        vol.mode                                                       1
        vol.recursive                                                  0
        vol.unmap_enabled                                              1
        wrlog_data_max                                        8589934592
        xattr_compat                                                   1
        zap_iterate_prefetch                                           1
        zap_micro_max_size                                        131072
        zevent.len_max                                               512
        zevent.retain_expire_secs                                    900
        zevent.retain_max                                           2000
        zfetch.max_distance                                     67108864
        zfetch.max_idistance                                    67108864
        zil.clean_taskq_maxalloc                                 1048576
        zil.clean_taskq_minalloc                                    1024
        zil.clean_taskq_nthr_pct                                     100
        zil.maxblocksize                                          131072
        zil.maxcopied                                               7680
        zil.min_commit_timeout                                      5000
        zil.nocacheflush                                               0
        zil.replay_disable                                             0
        zil.slog_bulk                                             786432
        zil_saxattr                                                    1
        zio.deadman_log_all                                            0
        zio.dva_throttle_enabled                                       1
        zio.exclude_metadata                                           0
        zio.requeue_io_start_cut_in_line                               1
        zio.slow_io_ms                                             30000
        zio.taskq_batch_pct                                           80
        zio.taskq_batch_tpq                                            0
        zvol_enforce_quotas                                            1

ZIL committed transactions:                                       122.5k
        Commit requests:                                            4.6k
        Flushes to stable storage:                                  4.6k
        Transactions to SLOG storage pool:            0 Bytes          0
        Transactions to non-SLOG storage pool:      148.1 MiB       4.2k
Comment 5 John F. Carr 2023-11-24 17:31:07 UTC
I have the same error on 13.2-STABLE (two systems) but not 14 or CURRENT.
Comment 6 John F. Carr 2023-11-24 20:05:19 UTC
Created attachment 246550 [details]
truss output for failing arc_summary on 13.2-STABLE
Comment 7 John F. Carr 2023-11-24 20:07:27 UTC
Created attachment 246551 [details]
truss output from failing system
Comment 8 John F. Carr 2023-11-24 20:07:46 UTC
In addition to running a different OS version than the systems where arc_summary works, all of my failing 13.2-STABLE systems have a cache partition on SSD linked to a raidz2 pool on spinning disk.  The failing truss output is from a system that looks like

# zpool list -v
NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
data        36.4T  20.9T  15.5T        -         -    36%    57%  1.17x    ONLINE  -
  raidz2-0  36.4T  20.9T  15.5T        -         -    36%  57.3%      -    ONLINE
    ada0    9.10T      -      -        -         -      -      -      -    ONLINE
    ada1    9.10T      -      -        -         -      -      -      -    ONLINE
    ada2    9.10T      -      -        -         -      -      -      -    ONLINE
    ada3    9.10T      -      -        -         -      -      -      -    ONLINE
cache           -      -      -        -         -      -      -      -  -
  ada4p5     150G   150G   368M        -         -     0%  99.8%      -    ONLINE
zroot        528G  56.7G   471G        -         -     9%    10%  1.00x    ONLINE  -
  ada4p4     531G  56.7G   471G        -         -     9%  10.7%      -    ONLINE
Comment 9 John F. Carr 2023-11-25 15:14:44 UTC
Created attachment 246563 [details]
value of arc_stats variable in function section_arc
Comment 10 Vladyslav V. Prodan 2023-11-27 15:06:22 UTC
My ZFS pool:

# zpool list -v
NAME                SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zroot                14G  7.41G  6.59G        -         -    17%    52%  1.00x    ONLINE  -
  gpt/system-ada0  14.5G  7.41G  6.59G        -         -    17%  53.0%      -    ONLINE
Comment 11 Emanuel Haupt freebsd_committer freebsd_triage 2023-12-13 10:07:39 UTC
I have the same error on 13.2-RELEASE-p8. See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275732.
Comment 12 Emanuel Haupt freebsd_committer freebsd_triage 2023-12-13 10:09:02 UTC
*** Bug 275732 has been marked as a duplicate of this bug. ***
Comment 13 Emanuel Haupt freebsd_committer freebsd_triage 2023-12-13 22:42:41 UTC
It's 100% reproducible:

--- use case start ---
sh
cat << EOF > Vagrantfile
Vagrant.configure("2") do |config|
  config.vm.box = "freebsd/FreeBSD-13.2-RELEASE"
  config.vm.box_version = "2023.04.07"
end

vagrant up
vagrant ssh

mkdir -p /usr/local/etc/pkg/repos/
echo 'FreeBSD: { url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest" }' > \
        /usr/local/etc/pkg/repos/FreeBSD.conf

pkg update -f
pkg install -y arc_summary

dd if=/dev/zero of=/zfs.disk bs=1M count=200
mdconfig -a -t vnode -f /zfs.disk
zpool create data /dev/md0

arc_summary

------------------------------------------------------------------------
ZFS Subsystem Report                            Wed Dec 13 22:41:52 2023
FreeBSD 13.2-RELEASE-p8                                    zpl version 5
Machine: freebsd (amd64)                                spa version 5000

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/local/bin/arc_summary", line 1034, in <module>
    main()
  File "/usr/local/bin/arc_summary", line 1028, in main
    section_calls[section](kstats)
  File "/usr/local/bin/arc_summary", line 554, in section_arc
    meta = arc_stats['meta']
KeyError: 'meta'

Original exception was:
Traceback (most recent call last):
  File "/usr/local/bin/arc_summary", line 1034, in <module>
    main()
  File "/usr/local/bin/arc_summary", line 1028, in main
    section_calls[section](kstats)
  File "/usr/local/bin/arc_summary", line 554, in section_arc
    meta = arc_stats['meta']
KeyError: 'meta'
--- use case stop ---
Comment 14 Joseph Mingrone freebsd_committer freebsd_triage 2023-12-15 01:12:27 UTC
I posted a review that hopefully fixes the problem.

https://reviews.freebsd.org/D43070

Could you try with the diff there and let me know if we're good?

https://reviews.freebsd.org/file/data/yvrzgysicyc3br5zvn5v/PHID-FILE-kx2vsrkftufkkkkccigk/D43070.diff
Comment 15 Emanuel Haupt freebsd_committer freebsd_triage 2023-12-15 08:47:57 UTC
Created attachment 247051 [details]
output of arc_summary on 13.1

The patch looks good to me. The patch solves the issue for me. See attached output.
Comment 16 commit-hook freebsd_committer freebsd_triage 2023-12-15 13:30:53 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=99767f704b5d493ce2d04365a42f70bdadb31bf7

commit 99767f704b5d493ce2d04365a42f70bdadb31bf7
Author:     Joseph Mingrone <jrm@FreeBSD.org>
AuthorDate: 2023-12-15 00:55:29 +0000
Commit:     Joseph Mingrone <jrm@FreeBSD.org>
CommitDate: 2023-12-15 12:59:06 +0000

    sysutils/arc_summary: Match script to FreeBSD version

    The latest arc_summary from the OpenZFS repository fails to run on
    FreeBSD 13.2.  Resolve this by installing arc_summary from a branch of
    our src repository that matches the major version of the target system.

    PR:             274605
    Sponsored by:   The FreeBSD Foundation
    Differential Revision:  https://reviews.freebsd.org/D43070

 sysutils/arc_summary/Makefile | 21 +++++++++++++++------
 sysutils/arc_summary/distinfo | 10 +++++++---
 2 files changed, 22 insertions(+), 9 deletions(-)
Comment 17 Joseph Mingrone freebsd_committer freebsd_triage 2023-12-15 13:33:20 UTC
Thanks for the reports and thanks for confirming things work for you now.  If there are still problems, please re-open or submit a new bug.