Bug 187835 - ngctl(8) strange behavior when adding more than 530 vlan through netgraph
Summary: ngctl(8) strange behavior when adding more than 530 vlan through netgraph
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 12.1-STABLE
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-net mailing list
URL:
Keywords: needs-qa
Depends on:
Blocks:
 
Reported: 2014-03-21 23:20 UTC by Vladislav V. Prodan
Modified: 2020-02-26 12:49 UTC (History)
4 users (show)

See Also:
koobs: mfc-stable12?
koobs: mfc-stable11?


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Vladislav V. Prodan 2014-03-21 23:20:00 UTC
Vlan numbering started with number 100

1)
After adding 538 th Vlans (vid = 637) were suddenly added additional options pcp=0 cfi=0

..
=627 } { hook="vlan628" vlan=628 vid=628 } { hook="vlan629" vlan=629 vid=629 } { hook="vlan630" vlan=630 vid=630 } { hook="vlan631" vlan=631 vid=631 } { hook="vlan632" vlan=632 vid=632 } { hook="vlan633" vlan=633 vid=633 } { hook="vlan634" vlan=634 vid=634 } { hook="vlan635" vlan=635 vid=635 } { hook="vlan636" vlan=636 vid=636 } { hook="vlan637" vlan=637 vid=637 pcp=0 cfi=0 } { hook="vlan638" vlan=638 vid=638 pcp=0 cfi=0 } { hook="vlan639" vlan=639 vid=639 pcp=0 cfi=0 } { hook="vlan640" vlan=640 vid=640 pcp=0 cfi=0 } { hook="vlan641" vlan=641 vid=641 pcp=0 cfi=0 } { hook="vlan642" vlan=642 vid=642 pcp=0 cfi=0 } { hook="vlan643" vlan=643 vid=643 pcp=0 cfi=0 } { hook="vlan644" vlan=644 vid=644 pcp=0 cfi=0 } { hook="vlan645" vlan=645 vid=645 pcp=0 cfi=0 } { hook="vlan646" vlan=646 vid=646 pcp=0 cfi=0 } { hook="vlan647" vlan=647 vid=647 pcp=0 cfi=0 } { hook="vlan648" vlan=648 vid=648 pcp=0 cfi=0 } { hook="vlan649" vlan=649 vid=649 pcp=0 cfi=0 } ] }

2)
After adding 560 th Vlans (vid = 659) Vlan list instead of a text format has become a binary ...

# ngctl msg em2:lower gettable
Rec'd response 3 from "[7]:":
0000:  3a 02 00 00 76 6c 61 6e 31 30 30 00 00 00 00 00  :...vlan100.....
0010:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0020:  00 00 00 00 64 00 64 00 00 00 76 6c 61 6e 31 30  ....d.d...vlan10
0030:  31 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  1...............
0040:  00 00 00 00 00 00 00 00 00 00 65 00 65 00 00 00  ..........e.e...
0050:  76 6c 61 6e 31 30 32 00 00 00 00 00 00 00 00 00  vlan102.........
0060:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0070:  66 00 66 00 00 00 76 6c 61 6e 31 30 33 00 00 00  f.f...vlan103...
0080:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
..

Script to create Vlan

#!/bin/sh

ETHER_IF=em2

ngctl -f- <<EOF
     mkpeer ${ETHER_IF}: vlan lower downstream
     name ${ETHER_IF}:lower vlan-em2
     connect ${ETHER_IF}: vlan-em2: upper nomatch
EOF

for i in `jot 560 100`; do
    ngctl mkpeer vlan-em2: eiface vlan$i ether
    ngctl msg vlan-em2: addfilter \{ vlan=$i hook=\"vlan$i\" \}
done

Tuning:

cat /etc/sysctl.conf
..
net.graph.recvspace=2048000
net.graph.maxdgram=2048000
kern.ipc.nmbclusters=2000000
..
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2014-03-22 00:25:29 UTC
Responsible Changed
From-To: freebsd-bugs->freebsd-net

reclassify.
Comment 2 Eitan Adler freebsd_committer freebsd_triage 2018-05-28 19:47:21 UTC
batch change:

For bugs that match the following
-  Status Is In progress 
AND
- Untouched since 2018-01-01.
AND
- Affects Base System OR Documentation

DO:

Reset to open status.


Note:
I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.
Comment 3 lutz 2020-02-18 11:56:43 UTC
That's a memory allocation issue with the netgraph framework. There are several ones, which become visible, if you add enough nodes.

I've to have a look into it anyway.
Comment 4 Vladislav V. Prodan 2020-02-19 21:16:46 UTC
In FreeBSD 12.1, after adding 300 vlans, the memory runs out to display the `ngctl list`.

# ngctl list
ngctl: send msg: No buffer space available

# netstat -m
4220/1100/5320 mbufs in use (current/cache/total)
4103/504/4607/62482 mbuf clusters in use (current/cache/total/max)
10/496 mbuf+clusters out of packet secondary zone in use (current/cache)
0/0/0/62482 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/18513 9k jumbo clusters in use (current/cache/total/max)
0/0/0/10413 16k jumbo clusters in use (current/cache/total/max)
18522K/2566K/21088K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 sendfile syscalls
0 sendfile syscalls completed without I/O request
0 requests for I/O initiated by sendfile
0 pages read by sendfile as part of a request
0 pages were valid at time of a sendfile request
0 pages were valid and substituted to bogus page
0 pages were requested for read ahead by applications
0 pages were read ahead by sendfile
0 times sendfile encountered an already busy page
0 requests for sfbufs denied
0 requests for sfbufs delayed

# sysctl net.graph.recvspace net.graph.maxdgram kern.ipc.nmbclusters
net.graph.recvspace: 20480
net.graph.maxdgram: 20480
kern.ipc.nmbclusters: 62482

# ngctl msg em2:lower gettable
...
 { hook="vlan386" vlan=386 vid=386 } { hook="vlan387" vlan=387 vid=387 } { hook="vlan388" vlan=388 vid=388 } { hook="vlan389" vlan=389 vid=389 } { hook="vlan390" vlan=390 vid=390 } { hook="vlan391" vlan=391 vid=391 } { hook="vlan392" vlan=392 vid=392 } { hook="vlan393" vlan=393 vid=393 } { hook="vlan394" vlan=394 vid=394 } { hook="vlan395" vlan=395 vid=395 } { hook="vlan396" vlan=396 vid=396 } { hook="vlan397" vlan=397 vid=397 } { hook="vlan398" vlan=398 vid=398 } { hook="vlan399" vlan=399 vid=399 } ] }
Comment 5 Vladislav V. Prodan 2020-02-19 21:37:34 UTC
I also show DHclient errors:

...
Feb 20 01:15:12 core dhclient[22934]: ngeth298: not found
Feb 20 01:15:12 core dhclient[22934]: exiting.
Feb 20 01:15:12 core root[22937]: /etc/rc.d/dhclient: WARNING: failed to start dhclient
Feb 20 01:15:14 core dhclient[23001]: ngeth298: not found
Feb 20 01:15:14 core dhclient[23001]: exiting.
Feb 20 01:15:14 core root[23004]: /etc/rc.d/dhclient: WARNING: failed to start dhclient
Feb 20 01:15:14 core dhclient[23053]: ngeth299: not found
Feb 20 01:15:14 core dhclient[23053]: exiting.
Feb 20 01:15:14 core root[23056]: /etc/rc.d/dhclient: WARNING: failed to start dhclient
Feb 20 01:15:16 core dhclient[23120]: ngeth299: not found
Feb 20 01:15:16 core dhclient[23120]: exiting.
Feb 20 01:15:16 core root[23123]: /etc/rc.d/dhclient: WARNING: failed to start dhclient

...
ngeth297: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=28<VLAN_MTU,JUMBO_MTU>
        ether 00:00:00:00:00:00
        inet6 fe80::a00:27ff:fe29:81e6%ngeth297 prefixlen 64 scopeid 0x132
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ngeth298: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=28<VLAN_MTU,JUMBO_MTU>
        ether 00:00:00:00:00:00
        inet6 fe80::a00:27ff:fe29:81e6%ngeth298 prefixlen 64 scopeid 0x133
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ngeth299: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=28<VLAN_MTU,JUMBO_MTU>
        ether 00:00:00:00:00:00
        inet6 fe80::a00:27ff:fe29:81e6%ngeth299 prefixlen 64 scopeid 0x134
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Comment 6 Aleksandr Fedorov 2020-02-20 11:29:59 UTC
(In reply to Vladislav V. Prodan from comment #4)

> # ngctl list
> ngctl: send msg: No buffer space available

Socket limit exceeded here.

You can increase this limit using sysctls:
net.graph.recvspace=2048000
net.graph.maxdgram=2048000

But, it also can't exceed kern.ipc.maxsockbuf

Quick, example:

mk_ngeth.sh:
#!/bin/sh

for i in `jot 1000`
do
        ngctl mkpeer . eiface test ether
done

# sh mk_ngeth.sh
# ifconfig |grep ngeth|wc -l
    1000
# sysctl net.graph.recvspace
net.graph.recvspace: 20480
# sysctl net.graph.maxdgram
net.graph.maxdgram: 20480
# sysctl kern.ipc.maxsockbuf
kern.ipc.maxsockbuf: 2097152

# ngctl ls
ngctl: send msg: No buffer space available

# sysctl kern.ipc.maxsockbuf=32000000
kern.ipc.maxsockbuf: 2097152 -> 32000000
# sysctl net.graph.recvspace=24000000
net.graph.recvspace: 30000 -> 24000000

# ngctl ls | wc -l
    1002
Comment 7 lutz 2020-02-20 15:16:48 UTC
I do have a netgraph system with more than 20k nodes. So providing larger buffers will not suffice. It's a quick workaround, thank you for this.

But the real solution needs to transfer chunks of the data. I've to sleep about this idea.
Comment 8 lutz 2020-02-20 15:18:41 UTC
May somebody with sufficent rights change the subject line of this ticket from "through nethraph" to "through netgraph"? ... I missed it while searching.
Comment 9 Vladislav V. Prodan 2020-02-20 15:35:28 UTC
(In reply to lutz from comment #8)

This is an excuse to create another bug report, but already to bugzilla, so that they add the ability to search by similar words
Comment 10 Vladislav V. Prodan 2020-02-20 16:30:00 UTC
(In reply to Aleksandr Fedorov from comment #6)

Yes, thank you.

On FreeBsd 12.1, with kern.ipc.maxsockbuf = 83886080 and adding 564 vlan, we get the binary format:

# ngctl msg em2:lower gettable |more
Rec'd response 3 from "[b]:":
0000:  34 02 00 00 76 6c 61 6e 31 30 30 00 00 00 00 00  4...vlan100.....
0010:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0020:  00 00 00 00 64 00 64 00 00 00 76 6c 61 6e 31 30  ....d.d...vlan10
0030:  31 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  1...............
0040:  00 00 00 00 00 00 00 00 00 00 65 00 65 00 00 00  ..........e.e...
0050:  76 6c 61 6e 31 30 32 00 00 00 00 00 00 00 00 00  vlan102.........
0060:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0070:  66 00 66 00 00 00 76 6c 61 6e 31 30 33 00 00 00  f.f...vlan103...
...

Further, after a sequence of commands leads to a kernel panic.

# delete all freshly created vlans
ngctl list | awk ' $2 ~ /ngeth/ {print "ngctl shutdown "$2":";}' | sh

# add already 700 vlan
cat test_creating_vlan.sh:

#!/bin/sh

# man ng_vlan

ETHER_IF=em2

ngctl -f- <<EOF
     mkpeer ${ETHER_IF}: vlan lower downstream
     name ${ETHER_IF}:lower vlan-${ETHER_IF}
     connect ${ETHER_IF}: vlan-${ETHER_IF}: upper nomatch
EOF

for i in `jot 700 100`; do
    ngctl mkpeer vlan-${ETHER_IF}: eiface vlan$i ether
    ngctl msg vlan-${ETHER_IF}: addfilter \{ vlan=$i hook=\"vlan$i\" \}
done

### Again we delete these 700 vlans
ngctl list | awk ' $2 ~ /ngeth/ {print "ngctl shutdown "$2":";}' | sh

### and
kernel panic
Comment 11 Vladislav V. Prodan 2020-02-20 17:08:07 UTC
I opened a new ticket for the kernel panic issue - PR 244247.
Comment 12 Kubilay Kocak freebsd_committer freebsd_triage 2020-02-21 04:24:31 UTC
Can we get any reproducer script(s) and network configurations (rc.conf et al) that demonstrate the issue included as attachments please
Comment 13 Vladislav V. Prodan 2020-02-21 04:35:00 UTC
(In reply to Kubilay Kocak from comment #12)

Look at PR 244247. I almost gave the settings and scripts there.
Comment 14 Vladislav V. Prodan 2020-02-21 04:46:49 UTC
(In reply to Vladislav V. Prodan from comment #13)

If there is no data, I will provide.
If necessary, I will give access to VPS over IPv6.
Comment 15 lutz 2020-02-26 12:49:02 UTC
Please have a look at D23840 and discuss it in Phabricator.
Comment 16 lutz 2020-02-26 12:49:42 UTC
(In reply to lutz from comment #15)
Link to the patch https://reviews.freebsd.org/D23840