Vlan numbering started with number 100 1) After adding 538 th Vlans (vid = 637) were suddenly added additional options pcp=0 cfi=0 .. =627 } { hook="vlan628" vlan=628 vid=628 } { hook="vlan629" vlan=629 vid=629 } { hook="vlan630" vlan=630 vid=630 } { hook="vlan631" vlan=631 vid=631 } { hook="vlan632" vlan=632 vid=632 } { hook="vlan633" vlan=633 vid=633 } { hook="vlan634" vlan=634 vid=634 } { hook="vlan635" vlan=635 vid=635 } { hook="vlan636" vlan=636 vid=636 } { hook="vlan637" vlan=637 vid=637 pcp=0 cfi=0 } { hook="vlan638" vlan=638 vid=638 pcp=0 cfi=0 } { hook="vlan639" vlan=639 vid=639 pcp=0 cfi=0 } { hook="vlan640" vlan=640 vid=640 pcp=0 cfi=0 } { hook="vlan641" vlan=641 vid=641 pcp=0 cfi=0 } { hook="vlan642" vlan=642 vid=642 pcp=0 cfi=0 } { hook="vlan643" vlan=643 vid=643 pcp=0 cfi=0 } { hook="vlan644" vlan=644 vid=644 pcp=0 cfi=0 } { hook="vlan645" vlan=645 vid=645 pcp=0 cfi=0 } { hook="vlan646" vlan=646 vid=646 pcp=0 cfi=0 } { hook="vlan647" vlan=647 vid=647 pcp=0 cfi=0 } { hook="vlan648" vlan=648 vid=648 pcp=0 cfi=0 } { hook="vlan649" vlan=649 vid=649 pcp=0 cfi=0 } ] } 2) After adding 560 th Vlans (vid = 659) Vlan list instead of a text format has become a binary ... # ngctl msg em2:lower gettable Rec'd response 3 from "[7]:": 0000: 3a 02 00 00 76 6c 61 6e 31 30 30 00 00 00 00 00 :...vlan100..... 0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0020: 00 00 00 00 64 00 64 00 00 00 76 6c 61 6e 31 30 ....d.d...vlan10 0030: 31 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 1............... 0040: 00 00 00 00 00 00 00 00 00 00 65 00 65 00 00 00 ..........e.e... 0050: 76 6c 61 6e 31 30 32 00 00 00 00 00 00 00 00 00 vlan102......... 0060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0070: 66 00 66 00 00 00 76 6c 61 6e 31 30 33 00 00 00 f.f...vlan103... 0080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ .. Script to create Vlan #!/bin/sh ETHER_IF=em2 ngctl -f- <<EOF mkpeer ${ETHER_IF}: vlan lower downstream name ${ETHER_IF}:lower vlan-em2 connect ${ETHER_IF}: vlan-em2: upper nomatch EOF for i in `jot 560 100`; do ngctl mkpeer vlan-em2: eiface vlan$i ether ngctl msg vlan-em2: addfilter \{ vlan=$i hook=\"vlan$i\" \} done Tuning: cat /etc/sysctl.conf .. net.graph.recvspace=2048000 net.graph.maxdgram=2048000 kern.ipc.nmbclusters=2000000 ..
Responsible Changed From-To: freebsd-bugs->freebsd-net reclassify.
batch change: For bugs that match the following - Status Is In progress AND - Untouched since 2018-01-01. AND - Affects Base System OR Documentation DO: Reset to open status. Note: I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.
That's a memory allocation issue with the netgraph framework. There are several ones, which become visible, if you add enough nodes. I've to have a look into it anyway.
In FreeBSD 12.1, after adding 300 vlans, the memory runs out to display the `ngctl list`. # ngctl list ngctl: send msg: No buffer space available # netstat -m 4220/1100/5320 mbufs in use (current/cache/total) 4103/504/4607/62482 mbuf clusters in use (current/cache/total/max) 10/496 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/62482 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/18513 9k jumbo clusters in use (current/cache/total/max) 0/0/0/10413 16k jumbo clusters in use (current/cache/total/max) 18522K/2566K/21088K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 sendfile syscalls 0 sendfile syscalls completed without I/O request 0 requests for I/O initiated by sendfile 0 pages read by sendfile as part of a request 0 pages were valid at time of a sendfile request 0 pages were valid and substituted to bogus page 0 pages were requested for read ahead by applications 0 pages were read ahead by sendfile 0 times sendfile encountered an already busy page 0 requests for sfbufs denied 0 requests for sfbufs delayed # sysctl net.graph.recvspace net.graph.maxdgram kern.ipc.nmbclusters net.graph.recvspace: 20480 net.graph.maxdgram: 20480 kern.ipc.nmbclusters: 62482 # ngctl msg em2:lower gettable ... { hook="vlan386" vlan=386 vid=386 } { hook="vlan387" vlan=387 vid=387 } { hook="vlan388" vlan=388 vid=388 } { hook="vlan389" vlan=389 vid=389 } { hook="vlan390" vlan=390 vid=390 } { hook="vlan391" vlan=391 vid=391 } { hook="vlan392" vlan=392 vid=392 } { hook="vlan393" vlan=393 vid=393 } { hook="vlan394" vlan=394 vid=394 } { hook="vlan395" vlan=395 vid=395 } { hook="vlan396" vlan=396 vid=396 } { hook="vlan397" vlan=397 vid=397 } { hook="vlan398" vlan=398 vid=398 } { hook="vlan399" vlan=399 vid=399 } ] }
I also show DHclient errors: ... Feb 20 01:15:12 core dhclient[22934]: ngeth298: not found Feb 20 01:15:12 core dhclient[22934]: exiting. Feb 20 01:15:12 core root[22937]: /etc/rc.d/dhclient: WARNING: failed to start dhclient Feb 20 01:15:14 core dhclient[23001]: ngeth298: not found Feb 20 01:15:14 core dhclient[23001]: exiting. Feb 20 01:15:14 core root[23004]: /etc/rc.d/dhclient: WARNING: failed to start dhclient Feb 20 01:15:14 core dhclient[23053]: ngeth299: not found Feb 20 01:15:14 core dhclient[23053]: exiting. Feb 20 01:15:14 core root[23056]: /etc/rc.d/dhclient: WARNING: failed to start dhclient Feb 20 01:15:16 core dhclient[23120]: ngeth299: not found Feb 20 01:15:16 core dhclient[23120]: exiting. Feb 20 01:15:16 core root[23123]: /etc/rc.d/dhclient: WARNING: failed to start dhclient ... ngeth297: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=28<VLAN_MTU,JUMBO_MTU> ether 00:00:00:00:00:00 inet6 fe80::a00:27ff:fe29:81e6%ngeth297 prefixlen 64 scopeid 0x132 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> ngeth298: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=28<VLAN_MTU,JUMBO_MTU> ether 00:00:00:00:00:00 inet6 fe80::a00:27ff:fe29:81e6%ngeth298 prefixlen 64 scopeid 0x133 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> ngeth299: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=28<VLAN_MTU,JUMBO_MTU> ether 00:00:00:00:00:00 inet6 fe80::a00:27ff:fe29:81e6%ngeth299 prefixlen 64 scopeid 0x134 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
(In reply to Vladislav V. Prodan from comment #4) > # ngctl list > ngctl: send msg: No buffer space available Socket limit exceeded here. You can increase this limit using sysctls: net.graph.recvspace=2048000 net.graph.maxdgram=2048000 But, it also can't exceed kern.ipc.maxsockbuf Quick, example: mk_ngeth.sh: #!/bin/sh for i in `jot 1000` do ngctl mkpeer . eiface test ether done # sh mk_ngeth.sh # ifconfig |grep ngeth|wc -l 1000 # sysctl net.graph.recvspace net.graph.recvspace: 20480 # sysctl net.graph.maxdgram net.graph.maxdgram: 20480 # sysctl kern.ipc.maxsockbuf kern.ipc.maxsockbuf: 2097152 # ngctl ls ngctl: send msg: No buffer space available # sysctl kern.ipc.maxsockbuf=32000000 kern.ipc.maxsockbuf: 2097152 -> 32000000 # sysctl net.graph.recvspace=24000000 net.graph.recvspace: 30000 -> 24000000 # ngctl ls | wc -l 1002
I do have a netgraph system with more than 20k nodes. So providing larger buffers will not suffice. It's a quick workaround, thank you for this. But the real solution needs to transfer chunks of the data. I've to sleep about this idea.
May somebody with sufficent rights change the subject line of this ticket from "through nethraph" to "through netgraph"? ... I missed it while searching.
(In reply to lutz from comment #8) This is an excuse to create another bug report, but already to bugzilla, so that they add the ability to search by similar words
(In reply to Aleksandr Fedorov from comment #6) Yes, thank you. On FreeBsd 12.1, with kern.ipc.maxsockbuf = 83886080 and adding 564 vlan, we get the binary format: # ngctl msg em2:lower gettable |more Rec'd response 3 from "[b]:": 0000: 34 02 00 00 76 6c 61 6e 31 30 30 00 00 00 00 00 4...vlan100..... 0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0020: 00 00 00 00 64 00 64 00 00 00 76 6c 61 6e 31 30 ....d.d...vlan10 0030: 31 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 1............... 0040: 00 00 00 00 00 00 00 00 00 00 65 00 65 00 00 00 ..........e.e... 0050: 76 6c 61 6e 31 30 32 00 00 00 00 00 00 00 00 00 vlan102......... 0060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0070: 66 00 66 00 00 00 76 6c 61 6e 31 30 33 00 00 00 f.f...vlan103... ... Further, after a sequence of commands leads to a kernel panic. # delete all freshly created vlans ngctl list | awk ' $2 ~ /ngeth/ {print "ngctl shutdown "$2":";}' | sh # add already 700 vlan cat test_creating_vlan.sh: #!/bin/sh # man ng_vlan ETHER_IF=em2 ngctl -f- <<EOF mkpeer ${ETHER_IF}: vlan lower downstream name ${ETHER_IF}:lower vlan-${ETHER_IF} connect ${ETHER_IF}: vlan-${ETHER_IF}: upper nomatch EOF for i in `jot 700 100`; do ngctl mkpeer vlan-${ETHER_IF}: eiface vlan$i ether ngctl msg vlan-${ETHER_IF}: addfilter \{ vlan=$i hook=\"vlan$i\" \} done ### Again we delete these 700 vlans ngctl list | awk ' $2 ~ /ngeth/ {print "ngctl shutdown "$2":";}' | sh ### and kernel panic
I opened a new ticket for the kernel panic issue - PR 244247.
Can we get any reproducer script(s) and network configurations (rc.conf et al) that demonstrate the issue included as attachments please
(In reply to Kubilay Kocak from comment #12) Look at PR 244247. I almost gave the settings and scripts there.
(In reply to Vladislav V. Prodan from comment #13) If there is no data, I will provide. If necessary, I will give access to VPS over IPv6.
Please have a look at D23840 and discuss it in Phabricator.
(In reply to lutz from comment #15) Link to the patch https://reviews.freebsd.org/D23840
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=45d75e3ac3fb5bf8230ca28dc09b48c6e5ed7a4f commit 45d75e3ac3fb5bf8230ca28dc09b48c6e5ed7a4f Author: Lutz Donnerhacke <donner@FreeBSD.org> AuthorDate: 2021-02-07 21:07:34 +0000 Commit: Lutz Donnerhacke <donner@FreeBSD.org> CommitDate: 2021-02-08 13:31:58 +0000 netgraph/ng_base: Allow larger BINARY2ASCII conversions Allocate the necessary memory for the conversion dynamically starting with a value which is sufficient for almost all normal cases. PR: 187835 Reviewed by: kp Differential Revision: https://reviews.freebsd.org/D23840 sys/netgraph/ng_base.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
A commit in branch stable/11 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=0d8ba15a0e9dbb5928d0fd89011417b707f7a366 commit 0d8ba15a0e9dbb5928d0fd89011417b707f7a366 Author: Lutz Donnerhacke <donner@FreeBSD.org> AuthorDate: 2021-02-07 21:07:34 +0000 Commit: Lutz Donnerhacke <donner@FreeBSD.org> CommitDate: 2021-05-09 13:20:48 +0000 netgraph/ng_base: Allow larger BINARY2ASCII conversions Allocate the necessary memory for the conversion dynamically starting with a value which is sufficient for almost all normal cases. PR: 187835 Reviewed by: kp Differential Revision: https://reviews.freebsd.org/D23840 (cherry picked from commit 45d75e3ac3fb5bf8230ca28dc09b48c6e5ed7a4f) sys/netgraph/ng_base.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
A commit in branch stable/12 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=e1caa707fee06d00a0e6ba0445d734e748721d77 commit e1caa707fee06d00a0e6ba0445d734e748721d77 Author: Lutz Donnerhacke <donner@FreeBSD.org> AuthorDate: 2021-02-07 21:07:34 +0000 Commit: Lutz Donnerhacke <donner@FreeBSD.org> CommitDate: 2021-05-09 13:23:19 +0000 netgraph/ng_base: Allow larger BINARY2ASCII conversions Allocate the necessary memory for the conversion dynamically starting with a value which is sufficient for almost all normal cases. PR: 187835 Reviewed by: kp Differential Revision: https://reviews.freebsd.org/D23840 (cherry picked from commit 45d75e3ac3fb5bf8230ca28dc09b48c6e5ed7a4f) sys/netgraph/ng_base.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=2b3074de33533f2703707c9b1c5e50a98621fdd1 commit 2b3074de33533f2703707c9b1c5e50a98621fdd1 Author: Lutz Donnerhacke <donner@FreeBSD.org> AuthorDate: 2021-02-07 21:07:34 +0000 Commit: Lutz Donnerhacke <donner@FreeBSD.org> CommitDate: 2021-05-09 13:25:05 +0000 netgraph/ng_base: Allow larger BINARY2ASCII conversions Allocate the necessary memory for the conversion dynamically starting with a value which is sufficient for almost all normal cases. PR: 187835 Reviewed by: kp Differential Revision: https://reviews.freebsd.org/D23840 (cherry picked from commit 45d75e3ac3fb5bf8230ca28dc09b48c6e5ed7a4f) sys/netgraph/ng_base.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)