Bug 193465 - [mips] malloc failures on mips, ath(4)
Summary: [mips] malloc failures on mips, ath(4)
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: mips Any
: --- Affects Only Me
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-09-08 17:38 UTC by Sean Bruno
Modified: 2014-09-22 05:09 UTC (History)
5 users (show)

See Also:


Attachments
r269962 vmstat -z output on my mips32 router. (6.00 KB, text/plain)
2014-09-08 17:38 UTC, Sean Bruno
no flags Details
HEAD vmstat -z output on my mips32 router. (6.78 KB, text/plain)
2014-09-08 17:57 UTC, Sean Bruno
no flags Details
vmstat -z with applied patch from Xin (6.60 KB, text/plain)
2014-09-13 22:37 UTC, Sean Bruno
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Sean Bruno freebsd_committer 2014-09-08 17:38:16 UTC
Created attachment 147072 [details]
r269962 vmstat -z output on my mips32 router.

It looks like svn r269963 removed a series of #if PAGE_SIZE checks that are tripping up mips boards that have a substantially small amount of RAM(64M in my case).

Reverting the removal of the #if PAGE_SIZE checks in kern_malloc.c is enough to alleviate this error.  I have also included some very strange vmstat -z output where I've captured at 269962 and then at 269963 that shows, at 269963, almost all allocation to be in the 16k buckets where 269962 doesn't have any allocation entry above 4k.

ath(4) is unable to malloc dma'able memory on initialization and causes a hard deadlock waiting in kmem_arena (as shown by ctrl-t on serial console).

wlan: <802.11 Link Layer>
pci0: driver added
found->	vendor=0x168c, dev=0x002a, revid=0x01
	domain=0, bus=0, slot=0, func=0
	class=02-80-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0146, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	intpin=a, irq=255
	MSI-X supports 1 message in map 0x10
pci0:0:0:0: reprobing on driver added
ath0: <Atheros 9280> at device 0.0 on pci0
ath0: Lazy allocation of 0x10000 bytes rid 0x10 type 3 at 0x10000000
ath0: ath_pci_attach: looking up firmware @ 'pcib.0.bus.0.0.0.eeprom_firmware'
ath0: ath_pci_attach: EEPROM firmware @ 0xc6eef000
ath0: malloc of tx buffers failed, size 139264
ath0: malloc of rx buffers failed, size 139264
ath0: [HT] enabling HT modes
ath0: [HT] 1 stream STBC receive enabled
ath0: [HT] 1 stream STBC transmit enabled
ath0: [HT] 2 RX streams; 2 TX streams
ath0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps
ath0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
ath0: 2T2R
ath0: 11ng MCS 20MHz
ath0: MCS 0-7: 6.5Mbps - 65Mbps
ath0: MCS 8-15: 13Mbps - 130Mbps
ath0: 11ng MCS 40MHz:
ath0: MCS 0-7: 13.5Mbps - 135Mbps
ath0: MCS 8-15: 27Mbps - 270Mbps
ath0: 11ng MCS 40MHz SGI:
ath0: MCS 0-7: 15Mbps - 150Mbps
ath0: MCS 8-15: 30Mbps - 300Mbps
ath0: AR9280 mac 128.2 RF5133 phy 13.0
ath0: 2GHz radio: 0x0000; 5GHz radio: 0x00c0
ath0: Use hw queue 1 for WME_AC_BE traffic
ath0: Use hw queue 0 for WME_AC_BK traffic
ath0: Use hw queue 2 for WME_AC_VI traffic
ath0: Use hw queue 3 for WME_AC_VO traffic
ath0: Use hw queue 8 for CAB traffic
ath0: Use hw queue 9 for beacons
ath0: using multicast key search
load: 0.97  cmd: kldload 48 [kmem arena] 53.66r 0.00u 0.01s 0% 1552k
Comment 1 Sean Bruno freebsd_committer 2014-09-08 17:57:48 UTC
Created attachment 147073 [details]
HEAD vmstat -z output on my mips32 router.
Comment 2 Sean Bruno freebsd_committer 2014-09-08 18:05:18 UTC
Specifically, this is what I am doing to work around this issue:

Index: sys/kern/kern_malloc.c
===================================================================
--- sys/kern/kern_malloc.c	(revision 271278)
+++ sys/kern/kern_malloc.c	(working copy)
@@ -151,10 +151,21 @@
 	{1024, "1024", },
 	{2048, "2048", },
 	{4096, "4096", },
+#if PAGE_SIZE > 4096
 	{8192, "8192", },
+#if PAGE_SIZE > 8192
 	{16384, "16384", },
+#if PAGE_SIZE > 16384
 	{32768, "32768", },
+#if PAGE_SIZE > 32768
 	{65536, "65536", },
+#if PAGE_SIZE > 65536
+#error "Unsupported PAGE_SIZE"
+#endif
+#endif
+#endif
+#endif
+#endif
 	{0, NULL},
 };
Comment 3 Sean Bruno freebsd_committer 2014-09-08 18:53:51 UTC
(In reply to Sean Bruno from comment #2)
> Specifically, this is what I am doing to work around this issue:
> 
> Index: sys/kern/kern_malloc.c
> ===================================================================
> --- sys/kern/kern_malloc.c	(revision 271278)
> +++ sys/kern/kern_malloc.c	(working copy)
> @@ -151,10 +151,21 @@
>  	{1024, "1024", },
>  	{2048, "2048", },
>  	{4096, "4096", },
> +#if PAGE_SIZE > 4096
>  	{8192, "8192", },
> +#if PAGE_SIZE > 8192
>  	{16384, "16384", },
> +#if PAGE_SIZE > 16384
>  	{32768, "32768", },
> +#if PAGE_SIZE > 32768
>  	{65536, "65536", },
> +#if PAGE_SIZE > 65536
> +#error "Unsupported PAGE_SIZE"
> +#endif
> +#endif
> +#endif
> +#endif
> +#endif
>  	{0, NULL},
>  };

This is incorrect.  I had to revert r266964 and r266963 in order to get the system to work again.

Just using this change results in a panic:
Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #0 r271278M: Mon Sep  8 11:36:16 PDT 2014
    sbruno@alice:/home/sbruno/bsd/obj/mips/mips.mips/home/sbruno/bsd/fbsd_head/sys/WZR-300HP mips
gcc version 4.2.1 20070831 patched [FreeBSD]
WARNING: WITNESS option enabled, expect reduced performance.
MEMGUARD DEBUGGING ALLOCATOR INITIALIZED:
	MEMGUARD map base: 0xc0800000
	MEMGUARD map size: 104860 KBytes
Preloaded elf kernel "kernel" at 0x80646970.
real memory  = 67108864 (65536K bytes)
Physical memory chunk(s):
0x006e1000 - 0x03ebcfff, 58572800 bytes (14300 pages)
avail memory = 57790464 (55MB)
Trap cause = 2 (TLB miss (load or instr. fetch) - kernel mode)
[ thread pid 0 tid 0 ]
Stopped at      strncmp+0x8:    lb      v1,0(a0)
db> reset
Comment 4 Adrian Chadd freebsd_committer 2014-09-12 19:38:12 UTC
Those two commits (266963/266964) aren't relevant. :P
Comment 5 Sean Bruno freebsd_committer 2014-09-12 19:45:13 UTC
(In reply to Adrian Chadd from comment #4)
> Those two commits (266963/266964) aren't relevant. :P

This should read "I had to revert 269964 and 269963"
Comment 6 Sean Bruno freebsd_committer 2014-09-13 22:36:39 UTC
Xin provided a patch on freebsd-src and I'm moving it here with results.

Patch:

Index: sys/kern/kern_malloc.c
===================================================================
--- sys/kern/kern_malloc.c      (revision 271494)
+++ sys/kern/kern_malloc.c      (working copy)
@@ -717,6 +717,8 @@ kmeminit(void)
         * a given architecture.
         */
        mem_size = vm_cnt.v_page_count;
+       if (mem_size <= 32768) /* delphij XXX 128MB */
+               kmem_zmax = PAGE_SIZE;

        if (vm_kmem_size_scale < 1)
                vm_kmem_size_scale = VM_KMEM_SIZE_SCALE;
Comment 7 Sean Bruno freebsd_committer 2014-09-13 22:37:56 UTC
Created attachment 147287 [details]
vmstat -z with applied patch from Xin

Hrm ... we can boot again, however there's still availability of "stuff" above 4k.

That seems bad.
Comment 8 Adrian Chadd freebsd_committer 2014-09-19 06:17:04 UTC
Hi,

So what's the verdict?

Can we only create those UMA zones entirely if we have substantially more RAM (say, 256MB) ?


-a
Comment 9 Sean Bruno freebsd_committer 2014-09-19 15:26:45 UTC
(In reply to Adrian Chadd from comment #8)
> Hi,
> 
> So what's the verdict?
> 
> Can we only create those UMA zones entirely if we have substantially more
> RAM (say, 256MB) ?
> 
> 
> -a

Look at the vmstat -z that I posted with Xin's patch.

It clearly does *not* allocate above 4k now, but the buckets are there in the output of vmstat.  I don't know if that's a bad thing or not.
Comment 10 Adrian Chadd freebsd_committer 2014-09-19 22:59:38 UTC
Ok, let's commit that for now and worry about the details after?
Comment 11 commit-hook freebsd_committer 2014-09-22 05:08:00 UTC
A commit references this bug:

Author: sbruno
Date: Mon Sep 22 05:07:23 UTC 2014
New revision: 271944
URL: http://svnweb.freebsd.org/changeset/base/271944

Log:
  svn revisions r269964 and r269963 seemed to have impaired small memory
  footprint systems(32M/64M) and didn't leave enough free memory to load modules
  when it was setting up page tables that for sizes that are never used on
  these smallish boards.

  Set kmem_zmax to PAGE_SIZE on these smaller systems (< 128M) to keep this
  from happening. Verified on mips32 h/w.

  PR:             193465
  Submitted by:   delphij
  Reviewed by:    adrian

Changes:
  head/sys/kern/kern_malloc.c
Comment 12 Sean Bruno freebsd_committer 2014-09-22 05:09:10 UTC
Yell at me if I didn't do this right.