Bug 129396
| Summary: | [kernel] cannot set vm.kmem_size_max beyond 3GB | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Weldon Godfrey <wgodfrey> |
| Component: | kern | Assignee: | Alan Cox <alc> |
| Status: | Closed FIXED | ||
| Severity: | Affects Only Me | ||
| Priority: | Normal | ||
| Version: | 7.1-PRERELEASE | ||
| Hardware: | Any | ||
| OS: | Any | ||
It looks like this might be the same problem that was fixed in -CURRENT a few months ago - see http://svn.freebsd.org/viewvc/base?view=revision&revision=180308 -- Bruce Ah, sorry, my search for kmem in gnats didn't pull that up. It does sound like the same issue. Weldon -----Original Message----- From: Bruce Cran [mailto:bruce@cran.org.uk]=20 Sent: Wednesday, December 03, 2008 2:07 PM To: Weldon Godfrey Cc: freebsd-gnats-submit@FreeBSD.org Subject: Re: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB It looks like this might be the same problem that was fixed in -CURRENT a few months ago - see http://svn.freebsd.org/viewvc/base?view=3Drevision&revision=3D180308 --=20 Bruce Responsible Changed From-To: freebsd-bugs->alc Potential MFC candidate, if that is not possible, please close this PR In the two files that was mentioned in the fix for -CURRENT, I applied them to my 7.1-PRERELEASE and recompiled/installed the kernel. I was able to set vm.kmem_size_max beyond 4GB. I ran into a kem shortage panic, but this time I paid attention that the size it said it was at the time of the panic was just under my setting for vm.kmem_size (1GB). I noticed that it had been panicing saying near 1GB size when vm.kmem_size_max was shown to be at 3GB or 16GB by sysctl -a. So, I tried setting kmem_size to 10G, that caused the known too big for boot-up panic, so did 2G. I was able to boot set at 1.5GB. It stayed up a longer, I started my rm -r that starts the panic and it did, but this time it said the size was 1.5GB. From this, I am guessing: 1. There may be more than just the changes in sys/kern/kern_malloc.c and=20 sys/vm/vm_kern.h 2. It appears that vm.kmem_size_max isn't being used or there is a false alarm panic when vm.kmem_size is exceeded? Weldon -----Original Message----- From: Bruce Cran [mailto:bruce@cran.org.uk]=20 Sent: Wednesday, December 03, 2008 3:27 PM To: alc@freebsd.org Cc: Weldon Godfrey Subject: Re: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB On Wed, 3 Dec 2008 20:40:05 GMT "Weldon Godfrey" <wgodfrey@ena.com> wrote: > The following reply was made to PR kern/129396; it has been noted by > GNATS. >=20 > From: "Weldon Godfrey" <wgodfrey@ena.com> > To: "Bruce Cran" <bruce@cran.org.uk> > Cc: <freebsd-gnats-submit@FreeBSD.org> > Subject: RE: misc/129396: [kern] cannot set vm.kmem_size_max beyond > 3GB Date: Wed, 3 Dec 2008 14:25:26 -0600 >=20 > Ah, sorry, my search for kmem in gnats didn't pull that up. It does > sound like the same issue. > =20 > Weldon > =20 > -----Original Message----- > From: Bruce Cran [mailto:bruce@cran.org.uk]=3D20 > Sent: Wednesday, December 03, 2008 2:07 PM > To: Weldon Godfrey > Cc: freebsd-gnats-submit@FreeBSD.org > Subject: Re: misc/129396: [kern] cannot set vm.kmem_size_max beyond > 3GB=20 > It looks like this might be the same problem that was fixed in > -CURRENT a few months ago - see > = http://svn.freebsd.org/viewvc/base?view=3D3Drevision&revision=3D3D180308 > =20 > --=3D20 > Bruce Hi alc, Is there any chance the fix to support a kmem_map greater than 4G could be merged back to -STABLE? It would be nice to have in 7.x at least. Or is it one of those changes that can't be made because it would change the ABI? --=20 Bruce brucec@freebsd.org I went ahead and upgraded to 8 (HEAD). I get the panic again (I have kmem_size set to 1GB, kmem_size_max set to 15GB, and it panics saying about 1GB is allocated. Is this because kmem_map isn't growing on its own or is there a false alarm panic caused by a check someplace? -----Original Message----- From: Weldon Godfrey=20 Sent: Friday, December 05, 2008 8:03 AM To: 'Bruce Cran'; 'alc@freebsd.org' Cc: 'freebsd-gnats-submit@FreeBSD.org' Subject: RE: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB In the two files that was mentioned in the fix for -CURRENT, I applied them to my 7.1-PRERELEASE and recompiled/installed the kernel. I was able to set vm.kmem_size_max beyond 4GB. I ran into a kem shortage panic, but this time I paid attention that the size it said it was at the time of the panic was just under my setting for vm.kmem_size (1GB). I noticed that it had been panicing saying near 1GB size when vm.kmem_size_max was shown to be at 3GB or 16GB by sysctl -a. So, I tried setting kmem_size to 10G, that caused the known too big for boot-up panic, so did 2G. I was able to boot set at 1.5GB. It stayed up a longer, I started my rm -r that starts the panic and it did, but this time it said the size was 1.5GB. From this, I am guessing: 1. There may be more than just the changes in sys/kern/kern_malloc.c and=20 sys/vm/vm_kern.h 2. It appears that vm.kmem_size_max isn't being used or there is a false alarm panic when vm.kmem_size is exceeded? Weldon -----Original Message----- From: Bruce Cran [mailto:bruce@cran.org.uk]=20 Sent: Wednesday, December 03, 2008 3:27 PM To: alc@freebsd.org Cc: Weldon Godfrey Subject: Re: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB On Wed, 3 Dec 2008 20:40:05 GMT "Weldon Godfrey" <wgodfrey@ena.com> wrote: > The following reply was made to PR kern/129396; it has been noted by > GNATS. >=20 > From: "Weldon Godfrey" <wgodfrey@ena.com> > To: "Bruce Cran" <bruce@cran.org.uk> > Cc: <freebsd-gnats-submit@FreeBSD.org> > Subject: RE: misc/129396: [kern] cannot set vm.kmem_size_max beyond > 3GB Date: Wed, 3 Dec 2008 14:25:26 -0600 >=20 > Ah, sorry, my search for kmem in gnats didn't pull that up. It does > sound like the same issue. > =20 > Weldon > =20 > -----Original Message----- > From: Bruce Cran [mailto:bruce@cran.org.uk]=3D20 > Sent: Wednesday, December 03, 2008 2:07 PM > To: Weldon Godfrey > Cc: freebsd-gnats-submit@FreeBSD.org > Subject: Re: misc/129396: [kern] cannot set vm.kmem_size_max beyond > 3GB=20 > It looks like this might be the same problem that was fixed in > -CURRENT a few months ago - see > = http://svn.freebsd.org/viewvc/base?view=3D3Drevision&revision=3D3D180308 > =20 > --=3D20 > Bruce Hi alc, Is there any chance the fix to support a kmem_map greater than 4G could be merged back to -STABLE? It would be nice to have in 7.x at least. Or is it one of those changes that can't be made because it would change the ABI? --=20 Bruce brucec@freebsd.org I am an ultra-noobie on this, so please forgive me if this is a stupid
question. Shouldn't this routine in vm/vm_kern.c have some logic to
malloc more kmem if under kmem_size_max before panic?
if (vm_map_findspace(map, vm_map_min(map), size, &addr)) {
vm_map_unlock(map);
if ((flags & M_NOWAIT) =3D=3D 0) {
for (i =3D 0; i < 8; i++) {
EVENTHANDLER_INVOKE(vm_lowmem, 0);
uma_reclaim();
vm_map_lock(map);
if (vm_map_findspace(map,
vm_map_min(map),
size, &addr) =3D=3D 0) {
break;
}
vm_map_unlock(map);
tsleep(&i, 0, "nokva", (hz / 4) * (i +
1));
}
if (i =3D=3D 8) {
panic("kmem_malloc(%ld): kmem_map too
small: %ld total allocated",
(long)size, (long)map->size);
}
} else {
return (0);
}
}
Weldon
-----Original Message-----
From: Weldon Godfrey=20
Sent: Thursday, December 11, 2008 10:42 AM
To: Weldon Godfrey; 'Bruce Cran'; 'alc@freebsd.org'
Cc: 'freebsd-gnats-submit@FreeBSD.org'
Subject: RE: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB
I went ahead and upgraded to 8 (HEAD). I get the panic again (I have
kmem_size set to 1GB, kmem_size_max set to 15GB, and it panics saying
about 1GB is allocated. Is this because kmem_map isn't growing on its
own or is there a false alarm panic caused by a check someplace?
-----Original Message-----
From: Weldon Godfrey=20
Sent: Friday, December 05, 2008 8:03 AM
To: 'Bruce Cran'; 'alc@freebsd.org'
Cc: 'freebsd-gnats-submit@FreeBSD.org'
Subject: RE: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB
In the two files that was mentioned in the fix for -CURRENT, I applied
them to my 7.1-PRERELEASE and recompiled/installed the kernel. I was
able to set vm.kmem_size_max beyond 4GB. I ran into a kem shortage
panic, but this time I paid attention that the size it said it was at
the time of the panic was just under my setting for vm.kmem_size (1GB).
I noticed that it had been panicing saying near 1GB size when
vm.kmem_size_max was shown to be at 3GB or 16GB by sysctl -a.
So, I tried setting kmem_size to 10G, that caused the known too big for
boot-up panic, so did 2G. I was able to boot set at 1.5GB. It stayed
up a longer, I started my rm -r that starts the panic and it did, but
this time it said the size was 1.5GB.
From this, I am guessing:
1. There may be more than just the changes in sys/kern/kern_malloc.c
and=20
sys/vm/vm_kern.h
2. It appears that vm.kmem_size_max isn't being used or there is a
false alarm panic when vm.kmem_size is exceeded?
Weldon
-----Original Message-----
From: Bruce Cran [mailto:bruce@cran.org.uk]=20
Sent: Wednesday, December 03, 2008 3:27 PM
To: alc@freebsd.org
Cc: Weldon Godfrey
Subject: Re: misc/129396: [kern] cannot set vm.kmem_size_max beyond 3GB
On Wed, 3 Dec 2008 20:40:05 GMT
"Weldon Godfrey" <wgodfrey@ena.com> wrote:
> The following reply was made to PR kern/129396; it has been noted by
> GNATS.
>=20
> From: "Weldon Godfrey" <wgodfrey@ena.com>
> To: "Bruce Cran" <bruce@cran.org.uk>
> Cc: <freebsd-gnats-submit@FreeBSD.org>
> Subject: RE: misc/129396: [kern] cannot set vm.kmem_size_max beyond
> 3GB Date: Wed, 3 Dec 2008 14:25:26 -0600
>=20
> Ah, sorry, my search for kmem in gnats didn't pull that up. It does
> sound like the same issue.
> =20
> Weldon
> =20
> -----Original Message-----
> From: Bruce Cran [mailto:bruce@cran.org.uk]=3D20
> Sent: Wednesday, December 03, 2008 2:07 PM
> To: Weldon Godfrey
> Cc: freebsd-gnats-submit@FreeBSD.org
> Subject: Re: misc/129396: [kern] cannot set vm.kmem_size_max beyond
> 3GB=20
> It looks like this might be the same problem that was fixed in
> -CURRENT a few months ago - see
> =
http://svn.freebsd.org/viewvc/base?view=3D3Drevision&revision=3D3D180308
> =20
> --=3D20
> Bruce
Hi alc,
Is there any chance the fix to support a kmem_map greater than 4G could
be merged back to -STABLE? It would be nice to have in 7.x at least.
Or is it one of those changes that can't be made because it would
change the ABI?
--=20
Bruce
brucec@freebsd.org
As I may have mentioned, I went ahead and installed 8-HEAD. It did the same thing in 1.x, panics on kmem_map too small, and saying it is at the size of vm.kmem_size and not near vm.kmem_size_max. I noticed I can set vm.kmem_size 4GB without kernel boot panic ( 2GB and above in 7.x seems to cause panic). I did try 10GB and that did panic at boot. However, setting it to 4GB allowed me to perform the tests fine. So this is the workaround for me, using 8-HEAD and setting vm.kmem_size to 4GB. Weldon State Changed From-To: open->closed A subset of the kernel and kmem map changes for amd64 have been applied to RELENG_7 (7.1-STABLE). The maximum supported kmem map size on amd64 is now a little less than 4GB, specifically, I believe it is 4GB-4KB. (In contrast, HEAD allows the kmem map to exceed 4GB.) When was this fixed in head? My copy of 8-CURRENT amd64 doesn't allow vm.kmem_size to exceed 4GB or it will panic on boot. Also, my copy of 8-CURRENT, vm.kmem_size_max doesn't seem to do anything as the system will panic, saying that kmem is at the size vm.kmem_size. If this was fixed in HEAD since Dec 2008, please let me know and I'll plan to upgrade. Weldon -----Original Message----- From: alc@FreeBSD.org [mailto:alc@FreeBSD.org]=20 Sent: Saturday, February 07, 2009 3:18 AM To: Weldon Godfrey; alc@FreeBSD.org; alc@FreeBSD.org Subject: Re: kern/129396: [kernel] cannot set vm.kmem_size_max beyond 3GB Synopsis: [kernel] cannot set vm.kmem_size_max beyond 3GB State-Changed-From-To: open->closed State-Changed-By: alc State-Changed-When: Sat Feb 7 09:09:38 UTC 2009 State-Changed-Why:=20 A subset of the kernel and kmem map changes for amd64 have been applied to RELENG_7 (7.1-STABLE). The maximum supported kmem map size on amd64 is now a little less than 4GB, specifically, I believe it is 4GB-4KB. (In contrast, HEAD allows the kmem map to exceed 4GB.) http://www.freebsd.org/cgi/query-pr.cgi?pr=3D129396 Weldon Godfrey wrote: > When was this fixed in head? My copy of 8-CURRENT amd64 doesn't allow > vm.kmem_size to exceed 4GB or it will panic on boot. Also, my copy of > 8-CURRENT, vm.kmem_size_max doesn't seem to do anything as the system > will panic, saying that kmem is at the size vm.kmem_size. > > If this was fixed in HEAD since Dec 2008, please let me know and I'll > plan to upgrade. > > Nothing relating to kmem map size has changed in 8-CURRENT since last summer. I am able to set vm.kmem_size in /boot/loader.conf to greater than 4GB and boot. Specifically, I set it 4GB+1MB. After booting, "sysctl vm" reports: ... vm.kmem_size_scale: 3 vm.kmem_size_max: 4509713203 vm.kmem_size_min: 0 vm.kmem_size: 4296015872 ... There is still a hard limit on vm.kmem_size. If you tried setting it to 5GB or more, I'm sure it crashed. The limit is somewhere between 4.5 and 5 GB. Regards, Alan Ok, then that is the hard limit I am reaching. (it is amazing now I am saying plus or minus a GIG). =20 It isn't a biggie to me currently. My stable configuration currently: Zil off (I tried turning it back on and had an episode where the zfs filesystem just hangs, I'll try again later on...i don't feel comfortable with this setting with NFS) Prefetch off ARC limited to 2GB vm.kmem_size=3D4GB vm.kmem_size_max=3D15GB The only question I have is that I have seen under 7.x and HEAD the system panic due to kmem exhaustion and it saying the size of kmem was what vm.kmem_size and that was way under what vm.kmem_size_max is, so it appears it wasn't growning. Could it be that it won't grow because my vm.kmem_size_max is over the 5GB limit even though sysctl shows it set to something like 15GB? Or is there a bug that doesn't allow kmem to grow? Weldon -----Original Message----- From: Alan Cox [mailto:alc@cs.rice.edu]=20 Sent: Tuesday, February 10, 2009 12:45 PM To: Weldon Godfrey Cc: alc@FreeBSD.org; FreeBSD-gnats-submit@FreeBSD.org Subject: Re: kern/129396: [kernel] cannot set vm.kmem_size_max beyond 3GB Weldon Godfrey wrote: > When was this fixed in head? My copy of 8-CURRENT amd64 doesn't allow > vm.kmem_size to exceed 4GB or it will panic on boot. Also, my copy of > 8-CURRENT, vm.kmem_size_max doesn't seem to do anything as the system > will panic, saying that kmem is at the size vm.kmem_size. > > If this was fixed in HEAD since Dec 2008, please let me know and I'll > plan to upgrade. > > =20 Nothing relating to kmem map size has changed in 8-CURRENT since last=20 summer. I am able to set vm.kmem_size in /boot/loader.conf to greater than 4GB=20 and boot. Specifically, I set it 4GB+1MB. After booting, "sysctl vm"=20 reports: ... vm.kmem_size_scale: 3 vm.kmem_size_max: 4509713203 vm.kmem_size_min: 0 vm.kmem_size: 4296015872 ... There is still a hard limit on vm.kmem_size. If you tried setting it to 5GB or more, I'm sure it crashed. The limit is somewhere between 4.5=20 and 5 GB. Regards, Alan Weldon Godfrey wrote: > Ok, then that is the hard limit I am reaching. (it is amazing now I am > saying plus or minus a GIG). > > It isn't a biggie to me currently. My stable configuration currently: > > Zil off (I tried turning it back on and had an episode where the zfs > filesystem just hangs, I'll try again later on...i don't feel > comfortable with this setting with NFS) > Prefetch off > ARC limited to 2GB > vm.kmem_size=4GB > vm.kmem_size_max=15GB > > The only question I have is that I have seen under 7.x and HEAD the > system panic due to kmem exhaustion and it saying the size of kmem was > what vm.kmem_size and that was way under what vm.kmem_size_max is, so it > appears it wasn't growning. Could it be that it won't grow because my > vm.kmem_size_max is over the 5GB limit even though sysctl shows it set > to something like 15GB? Or is there a bug that doesn't allow kmem to > grow? > > The kmem size is static. It doesn't grow during execution. It establishes the upper bound on how much physical memory the kernel is allowed to claim for its heap. vm.kmem_size_max is not what you think it is. During boot, the kernel computes a size for kmem based on the size of the kernel's overall virtual address space and the amount of physical memory that your machine has. Ideally, the latter of these factors is the determining factor. Unfortunately, that was not true until last summer. Before last summer, the 2GB size of the kernel virtual address space was the limiting factor on any machine with more than a couple gigabytes of memory. The effect of vm.kmem_size_max is to establish on upper bound on the size computed by this autosizing function or the manual setting of vm.kmem_size. There is no reason for you to set vm.kmem_size_max any longer. The default is already larger than your manual setting of vm.kmem_size. Regards, Alan Thanks for the learning lesson. BTW. We really appreciate the support and all the work for port ZFS into FreeBSD. Thanks! Weldon -----Original Message----- From: Alan Cox [mailto:alc@cs.rice.edu]=20 Sent: Wednesday, February 11, 2009 11:42 AM To: Weldon Godfrey Cc: alc@FreeBSD.org; FreeBSD-gnats-submit@FreeBSD.org Subject: Re: kern/129396: [kernel] cannot set vm.kmem_size_max beyond 3GB Weldon Godfrey wrote: > Ok, then that is the hard limit I am reaching. (it is amazing now I am > saying plus or minus a GIG). =20 > > It isn't a biggie to me currently. My stable configuration currently: > > Zil off (I tried turning it back on and had an episode where the zfs > filesystem just hangs, I'll try again later on...i don't feel > comfortable with this setting with NFS) > Prefetch off > ARC limited to 2GB > vm.kmem_size=3D4GB > vm.kmem_size_max=3D15GB > > The only question I have is that I have seen under 7.x and HEAD the > system panic due to kmem exhaustion and it saying the size of kmem was > what vm.kmem_size and that was way under what vm.kmem_size_max is, so it > appears it wasn't growning. Could it be that it won't grow because my > vm.kmem_size_max is over the 5GB limit even though sysctl shows it set > to something like 15GB? Or is there a bug that doesn't allow kmem to > grow? > > =20 The kmem size is static. It doesn't grow during execution. It=20 establishes the upper bound on how much physical memory the kernel is=20 allowed to claim for its heap. vm.kmem_size_max is not what you think it is. During boot, the kernel=20 computes a size for kmem based on the size of the kernel's overall=20 virtual address space and the amount of physical memory that your=20 machine has. Ideally, the latter of these factors is the determining=20 factor. Unfortunately, that was not true until last summer. Before=20 last summer, the 2GB size of the kernel virtual address space was the=20 limiting factor on any machine with more than a couple gigabytes of=20 memory. The effect of vm.kmem_size_max is to establish on upper bound=20 on the size computed by this autosizing function or the manual setting=20 of vm.kmem_size. There is no reason for you to set vm.kmem_size_max any longer. The=20 default is already larger than your manual setting of vm.kmem_size. Regards, Alan |
I noticed during my testing, while trying to copy two 2TB files at the same time (I am using ZFS), when the files get to a certain size, the system panics. It didn't dump core or log a reason in syslog, however, one time it managed not to reboot (just hang after panic) and I was able to capture that it was complaining that kmem was too small. I thought that was odd, since I have set the max to 15GB (the system has 16GB of RAM), so I checked sysctl and vm.kmem_max was only at 3GB. This is in my loader.conf: vm.kmem_size_max="16106127360" vm.kmem_size="1073741824" checking right after boot, the vm.kmem_size was set, it was at 1073741824 here is sysctl right after reboot with the above loader.conf: store1# sysctl -a | grep kmem vm.kmem_size_scale: 3 vm.kmem_size_max: 3221225472 vm.kmem_size_min: 0 vm.kmem_size: 1073741824 store1# sysctl -a | grep kern.maxvn kern.maxvnodes: 100000 store1# As a work-a-round, I tried to set these options in the kernel config and recompile the kernel, it won't compile, this is the error: cc -c -O2 -frename-registers -pipe -fno-strict-aliasing -std=c99 -g -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/altq -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -mcmodel=kernel -mno-red-zone-mfpmath=387 -mno-sse -mno-sse2 -mno-mmx -mno-3dnow -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -Werror /usr/src/sys/kern/kern_malloc.c cc1: warnings being treated as errors /usr/src/sys/kern/kern_malloc.c: In function 'kmeminit': /usr/src/sys/kern/kern_malloc.c:598: warning: large integer implicitly truncated to unsigned type *** Error code 1 Fix: I am guessing these tunables need a larger cast? u_int vm_kmem_size; SYSCTL_UINT(_vm, OID_AUTO, kmem_size, CTLFLAG_RD, &vm_kmem_size, 0, "Size of kernel memory"); u_int vm_kmem_size_min; SYSCTL_UINT(_vm, OID_AUTO, kmem_size_min, CTLFLAG_RD, &vm_kmem_size_min, 0, "Minimum size of kernel memory"); u_int vm_kmem_size_max; SYSCTL_UINT(_vm, OID_AUTO, kmem_size_max, CTLFLAG_RD, &vm_kmem_size_max, 0, "Maximum size of kernel memory"); u_int vm_kmem_size_scale; SYSCTL_UINT(_vm, OID_AUTO, kmem_size_scale, CTLFLAG_RD, &vm_kmem_size_scale, 0, "Scale factor for kernel memory size"); How-To-Repeat: try to set vm.kmem_size_max or OPTIONS VM_KMEM_SIZE_MAX beyond 3221225472