Bug 191510

Comment 1 Mark Linimon 2014-07-05 23:49:21 UTC

Over to maintainers.

Comment 2 Steven Hartland 2014-07-06 14:23:47 UTC

This looks like ZFS has backed off from max usage due to app usage on the machine, which is expected behaviour.

(In reply to Steven Hartland from comment #2)
> This looks like ZFS has backed off from max usage due to app usage on the
> machine, which is expected behaviour.

I don't think so because:

1. ARC memory usage never grows above 188 G.
2. I have no memory-hungry processes on this machine.

root@cs0:~# ps axu
USER    PID   %CPU %MEM    VSZ   RSS TT  STAT STARTED            TIME COMMAND
root     11 2292,8  0,0      0   384 ??  RL   14июн14 784228:58,16 [idle]
root      0   60,2  0,0      0  9904 ??  DLs  14июн14   7988:47,05 [kernel]
root     12   45,3  0,0      0   960 ??  WL   14июн14  10349:33,20 [intr]
root      4   10,6  0,0      0   176 ??  DL   14июн14   3197:31,63 [zfskern]
root     13    6,0  0,0      0    48 ??  DL   14июн14   1325:45,03 [geom]
jason 53722    4,7  0,0  52080  6124 ??  S    12:24           3:59,00 sshd: jason@notty (sshd)
root   1256    3,7  0,0  12008  1588 ??  Ss   14июн14     14:07,96 /usr/sbin/syslogd -ccss
www   79586    3,3  0,0  25156  9184 ??  S    25июн14    148:51,93 nginx: worker process (nginx)
www   79575    2,5  0,0  25156  9792 ??  S    25июн14    156:30,95 nginx: worker process (nginx)
www   79571    2,4  0,0  25156  8764 ??  S    25июн14    154:22,46 nginx: worker process (nginx)
root  33499    2,3  0,0      0    16 ??  DLs  13:57           0:11,76 [aiod3]
www   79588    2,3  0,0  25156  9076 ??  S    25июн14    154:41,30 nginx: worker process (nginx)
www   79591    2,2  0,0  25156  9564 ??  S    25июн14    164:38,45 nginx: worker process (nginx)
root  33496    1,8  0,0      0    16 ??  DLs  13:56           0:13,22 [aiod1]
root  97809    1,8  0,0      0    16 ??  DLs  14:02           0:04,57 [aiod4]
root  35718    1,7  0,0  75336 15312 ??  S    чт16        131:47,49 /usr/local/sbin/snmpd -p /var/run/net_snmpd.pid -c /usr/local/etc/snmpd.con
www   79576    1,6  0,0  25156  8796 ??  S    25июн14    154:48,13 nginx: worker process (nginx)
www   79590    1,6  0,0  25156  8788 ??  S    25июн14    152:07,78 nginx: worker process (nginx)
root  97812    1,4  0,0      0    16 ??  DLs  14:02           0:02,15 [aiod5]
www   79580    1,3  0,0  25156  9048 ??  S    25июн14    160:04,70 nginx: worker process (nginx)
www   79582    1,3  0,0  25156  8544 ??  S    25июн14    157:14,82 nginx: worker process (nginx)
root  55608    1,2  0,0      0    16 ??  DLs  14:00           0:06,40 [aiod6]
root  65838    1,1  0,0      0    16 ??  DLs  14:00           0:07,86 [aiod8]
www   79579    1,1  0,0  25156  9692 ??  S    25июн14    156:07,03 nginx: worker process (nginx)
www   79589    1,1  0,0  25156  8532 ??  S    25июн14    146:36,24 nginx: worker process (nginx)
root  97883    1,1  0,0      0    16 ??  DLs  14:05           0:00,27 [aiod2]
root  97815    1,0  0,0      0    16 ??  DLs  14:04           0:02,10 [aiod9]
www   79573    0,9  0,0  25156  9652 ??  S    25июн14    162:47,38 nginx: worker process (nginx)
www   79570    0,7  0,0  25156  8792 ??  S    25июн14    156:29,25 nginx: worker process (nginx)
www   79577    0,5  0,0  25156  9352 ??  S    25июн14    148:34,48 nginx: worker process (nginx)
www   79585    0,5  0,0  25156  9052 ??  S    25июн14    155:20,44 nginx: worker process (nginx)
www   79569    0,4  0,0  25156  9788 ??  S    25июн14    153:58,32 nginx: worker process (nginx)
www   79592    0,4  0,0  25156  8788 ??  S    25июн14    152:43,48 nginx: worker process (nginx)
www   79572    0,3  0,0  25156  9588 ??  S    25июн14    158:04,14 nginx: worker process (nginx)
root  53765    0,3  0,0  20620  5148  0  S+   12:25           0:32,40 top -aSHz
www   79583    0,2  0,0  25156  9316 ??  S    25июн14    154:24,06 nginx: worker process (nginx)
root      1    0,0  0,0   6280   560 ??  ILs  14июн14      0:01,57 /sbin/init --
root      2    0,0  0,0      0    16 ??  DL   14июн14      0:06,31 [mps_scan0]
root      3    0,0  0,0      0    16 ??  DL   14июн14      0:06,86 [mps_scan1]
root      5    0,0  0,0      0    16 ??  DL   14июн14      0:00,00 [xpt_thrd]
root      6    0,0  0,0      0    16 ??  DL   14июн14      0:00,00 [ipmi0: kcs]
root      7    0,0  0,0      0    16 ??  DL   14июн14      0:01,45 [pagedaemon]
root      8    0,0  0,0      0    16 ??  DL   14июн14      0:00,00 [vmdaemon]
root      9    0,0  0,0      0    16 ??  DL   14июн14      0:00,02 [pagezero]
root     10    0,0  0,0      0    16 ??  DL   14июн14      0:00,00 [audit]
root     14    0,0  0,0      0    16 ??  DL   14июн14     40:02,48 [yarrow]
root     15    0,0  0,0      0   128 ??  DL   14июн14      1:54,87 [usb]
root     16    0,0  0,0      0    16 ??  DL   14июн14      0:07,12 [bufdaemon]
root     17    0,0  0,0      0    16 ??  DL   14июн14    133:39,76 [syncer]
root     18    0,0  0,0      0    16 ??  DL   14июн14      0:07,71 [vnlru]
root     19    0,0  0,0      0    16 ??  DL   14июн14      0:00,04 [g_mirror swap0]
root     20    0,0  0,0      0    16 ??  DL   14июн14      0:00,04 [g_mirror swap1]
root    987    0,0  0,0  14184  1612 ??  Is   14июн14      0:00,00 /usr/sbin/moused -p /dev/ums0 -t auto -I /var/run/moused.ums0.pid
root   1027    0,0  0,0  10372  4456 ??  Is   14июн14      0:00,65 /sbin/devd
root   1267    0,0  0,0  14092  1932 ??  Ss   14июн14      0:02,50 /usr/sbin/rpcbind -h 10.0.8.30 -l
root   1298    0,0  0,0  12008  1852 ??  Is   14июн14      0:00,00 /usr/sbin/mountd -h 10.0.8.30 -l -S /etc/exports /etc/zfs/exports
root   1304    0,0  0,0   9868  1628 ??  Is   14июн14      0:00,26 nfsd: master (nfsd)
root   1305    0,0  0,0   9868  1780 ??  S    14июн14     37:23,54 nfsd: server (nfsd)
root   1308    0,0  0,0 274112  1744 ??  Is   14июн14      0:01,57 /usr/sbin/rpc.statd -h 10.0.8.30
root   1311    0,0  0,0  14092  1744 ??  Ss   14июн14      0:02,36 /usr/sbin/rpc.lockd -h 10.0.8.30
root   1385    0,0  0,0  22216  3508 ??  Ss   14июн14      1:17,27 /usr/sbin/ntpd -g -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.dri
root   1388    0,0  0,0  12004  1528 ??  Ss   14июн14     38:53,23 /usr/sbin/powerd
root   1402    0,0  0,0  28144  5016 ??  I    14июн14      0:12,78 /usr/local/sbin/smartd -c /usr/local/etc/smartd.conf -p /var/run/smartd.pid
root   1448    0,0  0,0  28868  4168 ??  Is   14июн14      0:02,46 /usr/sbin/sshd
root   1451    0,0  0,0  20288  4592 ??  Ss   14июн14      0:27,04 sendmail: accepting connections (sendmail)
smmsp  1454    0,0  0,0  20288  4408 ??  Is   14июн14      0:00,66 sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail)
root   1458    0,0  0,0  14096  1760 ??  Is   14июн14      0:06,77 /usr/sbin/cron -s
root  24965    0,0  0,0  52080  4772 ??  Is   12:05           0:00,02 sshd: jason [priv] (sshd)
jason 24967    0,0  0,0  52080  5080 ??  S    12:05           0:01,40 sshd: jason@pts/0 (sshd)
root  35701    0,0  0,0  19152  2292 ??  Is   чт16          0:00,67 /usr/local/bin/rsync --daemon
root  53717    0,0  0,0  52080  4836 ??  Is   12:24           0:00,05 sshd: jason [priv] (sshd)
jason 53723    0,0  0,0  17388  3240 ??  Is   12:24           0:00,01 tcsh -c dd of=/mnt/ztemp/jason/lvhdd.dd.gz bs=128k
jason 53725    0,0  0,0   9872  1916 ??  S    12:24           0:16,87 dd of=/mnt/ztemp/jason/lvhdd.dd.gz bs=128k
root  79568    0,0  0,0  21060  4352 ??  I    25июн14      0:00,02 nginx: master process /usr/local/sbin/nginx
www   79574    0,0  0,0  25156  9308 ??  S    25июн14    159:31,99 nginx: worker process (nginx)
www   79578    0,0  0,0  25156  9688 ??  S    25июн14    156:39,05 nginx: worker process (nginx)
www   79581    0,0  0,0  25156  9048 ??  S    25июн14    152:05,38 nginx: worker process (nginx)
www   79584    0,0  0,0  25156  8792 ??  S    25июн14    149:30,11 nginx: worker process (nginx)
www   79587    0,0  0,0  25156  8740 ??  S    25июн14    157:19,82 nginx: worker process (nginx)
root  97816    0,0  0,0  52080  4772 ??  Is   14:04           0:00,02 sshd: jason [priv] (sshd)
jason 97818    0,0  0,0  52080  5080 ??  S    14:04           0:00,03 sshd: jason@pts/1 (sshd)
root   1520    0,0  0,0  12008  1588 v1  Is+  14июн14      0:00,04 /usr/libexec/getty Pc ttyv1
root   1521    0,0  0,0  12008  1588 v2  Is+  14июн14      0:00,04 /usr/libexec/getty Pc ttyv2
root   1522    0,0  0,0  12008  1588 v3  Is+  14июн14      0:00,04 /usr/libexec/getty Pc ttyv3
jason 24968    0,0  0,0  17388  4148  0  Is   12:05           0:00,02 -tcsh (tcsh)
root  24970    0,0  0,0  46560  3548  0  I    12:05           0:00,02 sudo su -
root  24971    0,0  0,0  43180  2220  0  I    12:06           0:00,00 su -
root  24972    0,0  0,0  17388  3952  0  I    12:06           0:00,02 -su (tcsh)
jason 97819    0,0  0,0  17388  4148  1  Is   14:04           0:00,02 -tcsh (tcsh)
root  97821    0,0  0,0  46560  3548  1  I    14:04           0:00,01 sudo su -
root  97832    0,0  0,0  43180  2220  1  I    14:05           0:00,00 su -
root  97833    0,0  0,0  17388  3952  1  S    14:05           0:00,03 -su (tcsh)
root  97902    0,0  0,0  14144  2092  1  R+   14:06           0:00,00 ps axu
root@cs0:~#

Comment 4 Steven Hartland 2014-07-08 12:33:20 UTC

Are you sure that its not reduced over time due to something high memory usage processes which now aren't running?

To check this reboot and check the values then.

Comment 5 Steven Hartland 2014-07-08 13:02:49 UTC

sysctl kstat.zfs.misc.arcstats would also be useful

(In reply to Steven Hartland from comment #4)
> Are you sure that its not reduced over time due to something high memory
> usage processes which now aren't running?
> 
> To check this reboot and check the values then.

Absolutely, this machine is just nginx fileserver, without any other tasks. I can collect ARC graph if this information will be useful.


(In reply to Steven Hartland from comment #5)
> sysctl kstat.zfs.misc.arcstats would also be useful

kstat.zfs.misc.arcstats.hits: 6357561684
kstat.zfs.misc.arcstats.misses: 1754865888
kstat.zfs.misc.arcstats.demand_data_hits: 2247259288
kstat.zfs.misc.arcstats.demand_data_misses: 8782100
kstat.zfs.misc.arcstats.demand_metadata_hits: 2211824311
kstat.zfs.misc.arcstats.demand_metadata_misses: 8160603
kstat.zfs.misc.arcstats.prefetch_data_hits: 1283671663
kstat.zfs.misc.arcstats.prefetch_data_misses: 1737864288
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 614806422
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 58897
kstat.zfs.misc.arcstats.mru_hits: 1548194108
kstat.zfs.misc.arcstats.mru_ghost_hits: 10388582
kstat.zfs.misc.arcstats.mfu_hits: 3287889203
kstat.zfs.misc.arcstats.mfu_ghost_hits: 102318783
kstat.zfs.misc.arcstats.allocated: 1774678632
kstat.zfs.misc.arcstats.deleted: 1646058550
kstat.zfs.misc.arcstats.stolen: 1146219028
kstat.zfs.misc.arcstats.recycle_miss: 3101763
kstat.zfs.misc.arcstats.mutex_miss: 4427461
kstat.zfs.misc.arcstats.evict_skip: 284890938
kstat.zfs.misc.arcstats.evict_l2_cached: 188456702423040
kstat.zfs.misc.arcstats.evict_l2_eligible: 32054253667840
kstat.zfs.misc.arcstats.evict_l2_ineligible: 9169893205504
kstat.zfs.misc.arcstats.hash_elements: 18175960
kstat.zfs.misc.arcstats.hash_elements_max: 18488668
kstat.zfs.misc.arcstats.hash_collisions: 1127435975
kstat.zfs.misc.arcstats.hash_chains: 3877554
kstat.zfs.misc.arcstats.hash_chain_max: 24
kstat.zfs.misc.arcstats.p: 145378360667
kstat.zfs.misc.arcstats.c: 197771088957
kstat.zfs.misc.arcstats.c_min: 33203088384
kstat.zfs.misc.arcstats.c_max: 265624707072
kstat.zfs.misc.arcstats.size: 200751138272
kstat.zfs.misc.arcstats.hdr_size: 1329000768                                                                                                      
kstat.zfs.misc.arcstats.data_size: 194945960448                                                                                                   
kstat.zfs.misc.arcstats.other_size: 1720016232                                                                                                    
kstat.zfs.misc.arcstats.l2_hits: 558004136                                                                                                        
kstat.zfs.misc.arcstats.l2_misses: 1196861331                                                                                                     
kstat.zfs.misc.arcstats.l2_feeds: 4171054                                                                                                         
kstat.zfs.misc.arcstats.l2_rw_clash: 72326                                                                                                        
kstat.zfs.misc.arcstats.l2_read_bytes: 73002745243136                                                                                             
kstat.zfs.misc.arcstats.l2_write_bytes: 113011830327808                                                                                           
kstat.zfs.misc.arcstats.l2_writes_sent: 3913397                                                                                                   
kstat.zfs.misc.arcstats.l2_writes_done: 3913396                                                                                                   
kstat.zfs.misc.arcstats.l2_writes_error: 0                                                                                                        
kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 117425
kstat.zfs.misc.arcstats.l2_evict_lock_retry: 91025
kstat.zfs.misc.arcstats.l2_evict_reading: 219
kstat.zfs.misc.arcstats.l2_free_on_write: 4702368
kstat.zfs.misc.arcstats.l2_abort_lowmem: 56
kstat.zfs.misc.arcstats.l2_cksum_bad: 18
kstat.zfs.misc.arcstats.l2_io_error: 0
kstat.zfs.misc.arcstats.l2_size: 1838076781056
kstat.zfs.misc.arcstats.l2_asize: 1838076145664
kstat.zfs.misc.arcstats.l2_hdr_size: 3164330456
kstat.zfs.misc.arcstats.l2_compress_successes: 144015
kstat.zfs.misc.arcstats.l2_compress_zeros: 0
kstat.zfs.misc.arcstats.l2_compress_failures: 0
kstat.zfs.misc.arcstats.l2_write_trylock_fail: 387445125
kstat.zfs.misc.arcstats.l2_write_passed_headroom: 134983793
kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 59111238565
kstat.zfs.misc.arcstats.l2_write_in_l2: 850651818119
kstat.zfs.misc.arcstats.l2_write_io_in_progress: 847
kstat.zfs.misc.arcstats.l2_write_not_cacheable: 36478134670
kstat.zfs.misc.arcstats.l2_write_full: 2008530
kstat.zfs.misc.arcstats.l2_write_buffer_iter: 4171054
kstat.zfs.misc.arcstats.l2_write_pios: 3913397
kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 27587228945605632
kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 225730988
kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 19832
kstat.zfs.misc.arcstats.memory_throttle_count: 0
kstat.zfs.misc.arcstats.duplicate_buffers: 0
kstat.zfs.misc.arcstats.duplicate_buffers_size: 0
kstat.zfs.misc.arcstats.duplicate_reads: 0

Comment 7 Steven Hartland 2014-07-08 14:28:05 UTC

Just because the processes are using small amounts of memory doesn't mean others aspects of the kernel aren't spiking up and demanding ram such as mbufs, so just because its only running nginx doesn't mean you haven't seen a memory spike in other areas.


Are you seeing any movement at all over time in:
kstat.zfs.misc.arcstats.c

(In reply to Steven Hartland from comment #7)
> Just because the processes are using small amounts of memory doesn't mean
> others aspects of the kernel aren't spiking up and demanding ram such as
> mbufs, so just because its only running nginx doesn't mean you haven't seen
> a memory spike in other areas.
I'm using static mbuf setting, they're using ~500M RAM. I'm also wondering what part of kernel could use ~50G of RAM for a short period - this amount is always free (not inactive).

> Are you seeing any movement at all over time in:
> kstat.zfs.misc.arcstats.c
I will update machine to 9.3R and build some MRTG graphs for memory usage.

Created attachment 144730 [details]
Memory graph

Green: ARC size
Blue: Free memory

Comment on attachment 144730 [details]
Memory graph

So the ARC target size after the boot is 247G. Then ARC grows to ~185G and immediately after that target size drops down.
BTW, why do you think that I'm experiencing this because of other memory consumers? As you can see, ZFS memory throttle count is zero.

Created attachment 144771 [details]
Memory graph #2

A new graph. Server was rebooted because of power problem.

Comment 12 Steven Hartland 2014-08-23 18:25:17 UTC

Please try the latest patch on:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594

Comment 13 Steven Hartland 2014-08-28 20:07:28 UTC

Addresssed by: http://svnweb.freebsd.org/changeset/base/270759

(In reply to Steven Hartland from comment #12)
> Please try the latest patch on:
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594

Sorry for the long silence - it's too hard to work in absence of electricity and under mortar fire.

So, which exactly patch should I try? Two last are made against -CURRENT and 10, so they won't apply to 9.

Comment 15 Steven Hartland 2014-09-03 11:18:37 UTC

They should apply to 9 quite easily tbh, code in that area hasnt changed much I don't believe.

(In reply to Steven Hartland from comment #15)
> They should apply to 9 quite easily tbh, code in that area hasnt changed
> much I don't believe.

root@jnb:/usr/src# svn info
Path: .
Working Copy Root Path: /usr/src
URL: https://svn0.us-west.freebsd.org/base/releng/9.3
Relative URL: ^/releng/9.3
Repository Root: https://svn0.us-west.freebsd.org/base
Repository UUID: ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Revision: 271006
Node Kind: directory
Schedule: normal
Last Changed Author: gjb
Last Changed Rev: 268512
Last Changed Date: 2014-07-11 00:53:54 +0300 (Fri, 11 Jul 2014)

root@jnb:/usr/src# svn diff
root@jnb:/usr/src# wget 'https://bugs.freebsd.org/bugzilla/attachment.cgi?id=146456'
--2014-09-03 14:21:19--  https://bugs.freebsd.org/bugzilla/attachment.cgi?id=146456
Resolving bugs.freebsd.org (bugs.freebsd.org)... 8.8.178.110, 2001:1900:2254:206a::50:0
Connecting to bugs.freebsd.org (bugs.freebsd.org)|8.8.178.110|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://bz-attachments.freebsd.org/attachment.cgi?id=146456 [following]
--2014-09-03 14:21:21--  http://bz-attachments.freebsd.org/attachment.cgi?id=146456
Resolving bz-attachments.freebsd.org (bz-attachments.freebsd.org)... 8.8.178.110, 2001:1900:2254:206a::50:0
Connecting to bz-attachments.freebsd.org (bz-attachments.freebsd.org)|8.8.178.110|:80... connected.
HTTP request sent, awaiting response... 302 /attachment.cgi?id=146456
Location: https://bz-attachments.FreeBSD.org/attachment.cgi?id=146456 [following]
--2014-09-03 14:21:22--  https://bz-attachments.freebsd.org/attachment.cgi?id=146456
Connecting to bz-attachments.freebsd.org (bz-attachments.freebsd.org)|8.8.178.110|:443... connected.
HTTP request sent, awaiting response... 200 OK
Cookie coming from bz-attachments.freebsd.org attempted to set domain to bugs.freebsd.org
Length: unspecified [text/plain]
Saving to: 'arc-reclaim-stable10.patch'

    [ <=>                                                                                                     ] 7,665       --.-K/s   in 0.001s  

2014-09-03 14:21:23 (7.89 MB/s) - 'arc-reclaim-stable10.patch' saved [7665]

root@jnb:/usr/src# svn patch arc-reclaim-stable10.patch 
C         sys/cddl/compat/opensolaris/kern/opensolaris_kmem.c
>         rejected hunk @@ -126,18 +126,47 @@
U         sys/cddl/compat/opensolaris/sys/kmem.h
C         sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
>         rejected hunk @@ -204,10 +201,24 @@
>         rejected hunk @@ -217,7 +228,36 @@
>         applied hunk @@ -2421,9 +2461,12 @@ with offset -16
>         applied hunk @@ -2443,8 +2486,11 @@ with offset -16
>         applied hunk @@ -2455,15 +2501,25 @@ with offset -16
>         applied hunk @@ -2507,9 +2563,6 @@ with offset -16
>         applied hunk @@ -2516,6 +2569,8 @@ with offset -16
C         sys/vm/vm_pageout.c
>         rejected hunk @@ -115,10 +115,14 @@
>         applied hunk @@ -126,7 +130,7 @@ with offset -4
>         rejected hunk @@ -1637,15 +1641,11 @@
>         rejected hunk @@ -1691,7 +1691,18 @@
Summary of conflicts:
  Text conflicts: 3
root@jnb:/usr/src#

Comment 17 Steven Hartland 2014-09-04 18:25:07 UTC

Created attachment 146813 [details]
arc reclaim refactor (against releng/9.3)

Looks like there are more differences in 9.3 than I thought.

Try the attached patch, its WIP based on the latest review code on https://reviews.freebsd.org/D702

I've only kernel compiled tested I'm afraid as I don't have any 9 boxes.

(In reply to Steven Hartland from comment #17)
> Created attachment 146813 [details]
> arc reclaim refactor (against releng/9.3)
> 
> Looks like there are more differences in 9.3 than I thought.
> 
> Try the attached patch, its WIP based on the latest review code on
> https://reviews.freebsd.org/D702
> 
> I've only kernel compiled tested I'm afraid as I don't have any 9 boxes.

Hmm, this patch breaks world building:

===> cddl/lib/libzpool (all)
cc  -O3 -pipe -fno-strict-aliasing -march=core2 -I/usr/src/cddl/lib/libzpool/../../../sys/cddl/compat/opensolaris -I/usr/src/cddl/lib/libzpool/../../compat/opensolaris/include -I/usr/src/cddl/lib/libzpool/../../compat/opensolaris/lib/libumem -I/usr/src/cddl/lib/libzpool/../../contrib/opensolaris/lib/libzpool/common -I/usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/sys -I/usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs -I/usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/common/zfs -I/usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common -I/usr/src/cddl/lib/libzpool/../../contrib/opensolaris/head -I/usr/src/cddl/lib/libzpool/../../lib/libumem -I/usr/src/cddl/lib/libzpool/../../contrib/opensolaris/lib/libnvpair -DWANTS_MUTEX_OWNED -I/usr/src/cddl/lib/libzpool/../../../lib/libpthread/thread -I/usr/src/cddl/lib/libzpool/../../../lib/libpthread/sys -I/usr/src/cddl/lib/libzpool/../../../lib/libthr/arch/amd64/include -DDEBUG=1 -DNEED_SOLARIS_BOOLEAN -DNDEBUG -std=iso9899:1999 -Qunused-arguments  -fstack-protector -Wno-pointer-sign -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-conversion -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -Wno-parentheses -Wno-unknown-pragmas -c /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c -o arc.o
/usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:243:8: warning: implicit declaration of function
      'sysctl_handle_int' is invalid in C99 [-Wimplicit-function-declaration]
        err = sysctl_handle_int(oidp, &val, 0, req);
              ^
/usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:247:12: error: use of undeclared identifier 'minfree'
        if (val < minfree)
                  ^
/usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:249:12: error: use of undeclared identifier 'cnt'
        if (val > cnt.v_page_count)
                  ^
1 warning and 2 errors generated.
*** [arc.o] Error code 1

Stop in /usr/src/cddl/lib/libzpool.
*** [all] Error code 1

Stop in /usr/src/cddl/lib.
*** [cddl/lib__L] Error code 1

Stop in /usr/src.
*** [libraries] Error code 1

Stop in /usr/src.
*** [_libraries] Error code 1

Stop in /usr/src.
*** [buildworld] Error code 1

Stop in /usr/src.

Comment 19 Steven Hartland 2014-09-05 11:25:22 UTC

Created attachment 146856 [details]
arc reclaim refactor (against releng/9.3)

Fixed world build

Comment 20 Steven Hartland 2014-09-05 11:44:11 UTC

Created attachment 146858 [details]
arc reclaim refactor (against releng/9.3)

Added missing machine/vmparam.h include causing all platforms to take the small memory code paths

(In reply to Steven Hartland from comment #20)
> Created attachment 146858 [details]
> arc reclaim refactor (against releng/9.3)
> 
> Added missing machine/vmparam.h include causing all platforms to take the
> small memory code paths

Looks like things are much better now. I'll post some thats here later.

(In reply to vsjcfm from comment #21)
> (In reply to Steven Hartland from comment #20)
> > Created attachment 146858 [details]
> > arc reclaim refactor (against releng/9.3)
> > 
> > Added missing machine/vmparam.h include causing all platforms to take the
> > small memory code paths
> 
> Looks like things are much better now. I'll post some thats here later.

Yep, works as expected.

root@cs0:~# fgrep " memory " /var/run/dmesg.boot
real memory  = 274877906944 (262144 MB)
avail memory = 265898815488 (253580 MB)
root@cs0:~# zfs-stats -a

------------------------------------------------------------------------
ZFS Subsystem Report                            Thu Sep 11 11:23:57 2014
------------------------------------------------------------------------

System Information:

        Kernel Version:                         903000 (osreldate)
        Hardware Platform:                      amd64
        Processor Architecture:                 amd64

        ZFS Storage pool Version:               5000
        ZFS Filesystem Version:                 5

FreeBSD 9.3-RELEASE #0 r271159M: Fri Sep 5 16:38:52 EEST 2014 root
11:23  up  1:46, 1 user, load averages: 0,18 0,87 1,37

------------------------------------------------------------------------

System Memory:

        0.01%   20.14   MiB Active,     0.14%   355.82  MiB Inact
        94.12%  233.77  GiB Wired,      0.03%   66.97   MiB Cache
        5.71%   14.18   GiB Free,       0.00%   2.51    MiB Gap

        Real Installed:                         256.00  GiB
        Real Available:                 99.98%  255.96  GiB
        Real Managed:                   97.04%  248.38  GiB

        Logical Total:                          256.00  GiB
        Logical Used:                   94.30%  241.41  GiB
        Logical Free:                   5.70%   14.59   GiB                                                                                       
                                                                                                                                                  
Kernel Memory:                                  229.70  GiB                                                                                       
        Data:                           100.00% 229.69  GiB                                                                                       
        Text:                           0.00%   11.19   MiB                                                                                       
                                                                                                                                                  
Kernel Memory Map:                              240.93  GiB                                                                                       
        Size:                           95.17%  229.29  GiB                                                                                       
        Free:                           4.83%   11.65   GiB                                                                                       
                                                                                                                                                  
------------------------------------------------------------------------                                                                          
                                                                                                                                                  
ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                450.07k
        Recycle Misses:                         3.10k
        Mutex Misses:                           1.66k
        Evict Skips:                            2.93k

ARC Size:                               92.20%  228.09  GiB
        Target Size: (Adaptive)         92.20%  228.09  GiB
        Min Size (Hard Limit):          12.50%  30.92   GiB
        Max Size (High Water):          8:1     247.38  GiB

ARC Size Breakdown:
        Recently Used Cache Size:       78.46%  178.96  GiB
        Frequently Used Cache Size:     21.54%  49.13   GiB

ARC Hash Breakdown:
        Elements Max:                           2.71m
        Elements Current:               99.94%  2.71m
        Collisions:                             925.34k
        Chain Max:                              8
        Chains:                                 577.91k

------------------------------------------------------------------------

ARC Efficiency:                                 23.85m
        Cache Hit Ratio:                86.86%  20.71m
        Cache Miss Ratio:               13.14%  3.13m
        Actual Hit Ratio:               83.30%  19.87m

        Data Demand Efficiency:         99.39%  14.51m
        Data Prefetch Efficiency:       1.26%   2.95m

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             4.10%   848.28k
          Most Recently Used:           67.23%  13.93m
          Most Frequently Used:         28.67%  5.94m
          Most Recently Used Ghost:     0.00%   18
          Most Frequently Used Ghost:   0.00%   283

        CACHE HITS BY DATA TYPE:
          Demand Data:                  69.61%  14.42m
          Prefetch Data:                0.18%   37.05k
          Demand Metadata:              26.29%  5.45m
          Prefetch Metadata:            3.92%   811.53k

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  2.84%   88.93k
          Prefetch Data:                92.84%  2.91m
          Demand Metadata:              3.57%   111.97k
          Prefetch Metadata:            0.75%   23.47k

------------------------------------------------------------------------

L2 ARC Summary: (HEALTHY)
        Passed Headroom:                        26.15k
        Tried Lock Failures:                    650.09k
        IO In Progress:                         0
        Low Memory Aborts:                      3
        Free on Write:                          1
        Writes While Full:                      0
        R/W Clashes:                            0
        Bad Checksums:                          0
        IO Errors:                              0
        SPA Mismatch:                           171.98m

L2 ARC Size: (Adaptive)                         70.24   MiB
        Header Size:                    0.02%   11.87   KiB

L2 ARC Breakdown:                               3.13m
        Hit Ratio:                      0.01%   204
        Miss Ratio:                     99.99%  3.13m
        Feeds:                                  6.36k

L2 ARC Buffer:
        Bytes Scanned:                          8.77    TiB
        Buffer Iterations:                      6.36k
        List Iterations:                        406.98k
        NULL List Iterations:                   38

L2 ARC Writes:
        Writes Sent:                    100.00% 978

------------------------------------------------------------------------

File-Level Prefetch: (HEALTHY)

DMU Efficiency:                                 50.42m
        Hit Ratio:                      72.29%  36.45m
        Miss Ratio:                     27.71%  13.97m

        Colinear:                               13.97m
          Hit Ratio:                    0.00%   477
          Miss Ratio:                   100.00% 13.97m

        Stride:                                 32.79m
          Hit Ratio:                    100.00% 32.79m
          Miss Ratio:                   0.00%   35

DMU Misc:
        Reclaim:                                13.97m
          Successes:                    0.07%   9.19k
          Failures:                     99.93%  13.96m

        Streams:                                3.66m
          +Resets:                      0.00%   25
          -Resets:                      100.00% 3.66m
          Bogus:                                0

------------------------------------------------------------------------

VDEV Cache Summary:                             177.69k
        Hit Ratio:                      22.87%  40.64k
        Miss Ratio:                     69.06%  122.72k
        Delegations:                    8.07%   14.34k

------------------------------------------------------------------------

ZFS Tunables (sysctl):
        kern.maxusers                           16717
        vm.kmem_size                            266698121216
        vm.kmem_size_scale                      1
        vm.kmem_size_min                        0
        vm.kmem_size_max                        329853485875
        vfs.zfs.arc_max                         265624379392
        vfs.zfs.arc_min                         33203047424
        vfs.zfs.arc_free_target                 1811218
        vfs.zfs.arc_meta_used                   2187249752
        vfs.zfs.arc_meta_limit                  66406094848
        vfs.zfs.l2arc_write_max                 41943040
        vfs.zfs.l2arc_write_boost               83886080
        vfs.zfs.l2arc_headroom                  4
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_noprefetch                0
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.anon_size                       705024
        vfs.zfs.anon_metadata_lsize             0
        vfs.zfs.anon_data_lsize                 0
        vfs.zfs.mru_size                        192158378496
        vfs.zfs.mru_metadata_lsize              161177088
        vfs.zfs.mru_data_lsize                  191584529408
        vfs.zfs.mru_ghost_size                  52754266112
        vfs.zfs.mru_ghost_metadata_lsize        102661120
        vfs.zfs.mru_ghost_data_lsize            52651604992
        vfs.zfs.mfu_size                        51316666880
        vfs.zfs.mfu_metadata_lsize              18357248
        vfs.zfs.mfu_data_lsize                  51140888064
        vfs.zfs.mfu_ghost_size                  34173508096
        vfs.zfs.mfu_ghost_metadata_lsize        45357056
        vfs.zfs.mfu_ghost_data_lsize            34128151040
        vfs.zfs.l2c_only_size                   1554432
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.nopwrite_enabled                1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.prefetch_disable                0
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.block_cap                256
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.top_maxinflight                 32
        vfs.zfs.resilver_delay                  2
        vfs.zfs.scrub_delay                     4
        vfs.zfs.scan_idle                       50
        vfs.zfs.scan_min_time_ms                1000
        vfs.zfs.free_min_time_ms                1000
        vfs.zfs.resilver_min_time_ms            3000
        vfs.zfs.no_scrub_io                     0
        vfs.zfs.no_scrub_prefetch               0
        vfs.zfs.metaslab.gang_bang              131073
        vfs.zfs.metaslab.debug_load             0
        vfs.zfs.metaslab.debug_unload           0
        vfs.zfs.metaslab.df_alloc_threshold     131072
        vfs.zfs.metaslab.df_free_pct            4
        vfs.zfs.metaslab.min_alloc_size         10485760
        vfs.zfs.metaslab.load_pct               50
        vfs.zfs.metaslab.unload_delay           8
        vfs.zfs.metaslab.preload_limit          3
        vfs.zfs.metaslab.preload_enabled        1
        vfs.zfs.metaslab.weight_factor_enable   0
        vfs.zfs.condense_pct                    200
        vfs.zfs.mg_noalloc_threshold            0
        vfs.zfs.write_to_degraded               0
        vfs.zfs.check_hostid                    1
        vfs.zfs.recover                         0
        vfs.zfs.deadman_synctime_ms             1000000
        vfs.zfs.deadman_checktime_ms            5000
        vfs.zfs.deadman_enabled                 1
        vfs.zfs.spa_asize_inflation             24
        vfs.zfs.txg.timeout                     10
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.cache.size                 41943040
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.trim_on_init               1
        vfs.zfs.vdev.max_active                 1000
        vfs.zfs.vdev.sync_read_min_active       10
        vfs.zfs.vdev.sync_read_max_active       10
        vfs.zfs.vdev.sync_write_min_active      10
        vfs.zfs.vdev.sync_write_max_active      10
        vfs.zfs.vdev.async_read_min_active      1
        vfs.zfs.vdev.async_read_max_active      3
        vfs.zfs.vdev.async_write_min_active     1
        vfs.zfs.vdev.async_write_max_active     10
        vfs.zfs.vdev.scrub_min_active           1
        vfs.zfs.vdev.scrub_max_active           2
        vfs.zfs.vdev.aggregation_limit          131072
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.vdev.bio_delete_disable         0
        vfs.zfs.vdev.trim_max_bytes             2147483648
        vfs.zfs.vdev.trim_max_pending           64
        vfs.zfs.max_auto_ashift                 13
        vfs.zfs.min_auto_ashift                 9
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zio.use_uma                     0
        vfs.zfs.sync_pass_deferred_free         2
        vfs.zfs.sync_pass_dont_compress         5
        vfs.zfs.sync_pass_rewrite               2
        vfs.zfs.snapshot_list_prefetch          0
        vfs.zfs.super_owner                     0
        vfs.zfs.debug                           0
        vfs.zfs.version.ioctl                   3
        vfs.zfs.version.acl                     1
        vfs.zfs.version.spa                     5000
        vfs.zfs.version.zpl                     5
        vfs.zfs.trim.enabled                    0
        vfs.zfs.trim.txg_delay                  32
        vfs.zfs.trim.timeout                    30
        vfs.zfs.trim.max_interval               1

------------------------------------------------------------------------

root@cs0:~# top -aSHz -d 1
root@cs0:~# sysctl vm. vfs.zfs. | grep "target:"
vm.v_free_target: 1726387
vm.v_inactive_target: 2589580
vm.stats.vm.v_free_target: 1726387
vm.stats.vm.v_inactive_target: 2589580
vfs.zfs.arc_free_target: 1811218
root@cs0:~# exit

BTW, why vm.v_free_target: 1726387 isn't equal to vfs.zfs.arc_free_target: 1811218?

Comment 24 Steven Hartland 2014-09-15 08:16:05 UTC

(In reply to vsjcfm from comment #23)
> BTW, why vm.v_free_target: 1726387 isn't equal to vfs.zfs.arc_free_target:
> 1811218?

In the version of the patch you have there is cnt.v_free_reserved + cnt.v_cache_min

I see that patch works without any issues. Can we assume it's readiness? If so, will be an MFS to releng/?

Comment 26 Steven Hartland 2014-09-15 08:32:24 UTC

At the moment no, there's still discussion going on about the value that should be used for zfs_arc_free_target.

(In reply to Steven Hartland from comment #26)
> At the moment no, there's still discussion going on about the value that
> should be used for zfs_arc_free_target.

I'm not a FreeBSD's VM expert, but I think that value of vm.stats.vm.v_free_target is a best choice.

I see that PR is was closed. Can you tell me, please, in which revision problem was fixed?

Comment 29 Steven Hartland 2014-11-03 17:15:04 UTC

The head revision was r272483 and stable/10 was r272875

(In reply to Steven Hartland from comment #29)
> The head revision was r272483 and stable/10 was r272875

But will be a MFC to stable/9? I will upgrade to releng/10.1 as soon as possible, but others may stay on releng/9.

Comment 31 Steven Hartland 2014-11-03 17:19:51 UTC

I don't think these changes will apply cleanly to 9

The changes didn't make it into 10.1 I'm afraid, so you'll either need 10.1 + that commit, stable/10 or 10.2 once that's done.