Bug 275106 - [regression] ng_ksocket(4) produces kernel panic
Summary: [regression] ng_ksocket(4) produces kernel panic
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Some People
Assignee: Gleb Smirnoff
URL:
Keywords: crash, regression
: 281268 (view as bug list)
Depends on:
Blocks:
 
Reported: 2023-11-15 15:38 UTC by Eugene Grosbein
Modified: 2024-09-04 16:35 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Eugene Grosbein freebsd_committer freebsd_triage 2023-11-15 15:38:29 UTC
Let's start ngctl without arguments (interactive mode) and run the following commands to create ng_ksocket(4) instance listening tcp/9 port and connected to ng_hole(4) to implement TCP "discard" service for demonstration:

mkpeer hole dummy dummy
name .:dummy hole
mkpeer hole: ksocket hohook inet/stream/tcp
name hole:hohook kso
# setsockopt(SOL_SOCKET, SO_REUSEPORT, 1)
msg kso: setopt { level=0xffff name=0x0200 value = [1] }
msg kso: bind inet/0.0.0.0:9
msg kso: listen 1
msg kso: accept

For FreeBSD releases before 12.0, it works: one can do "telnet localhost 9" to make connection to the port. Also ngctl command "shutdown kso:" works just fine thereafter.

For FreeBSD versions after 12.0-RELEASE, "shutdown kso:" works if we run it before "msg kso: listen 1". After that stage, "shutdown kso:" produces instant panic:

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x18
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80b26298
stack pointer           = 0x28:0xfffffe00085ff9c0
frame pointer           = 0x28:0xfffffe00085ffa00
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 730 (ngctl)
rdi: 0000000000000000 rsi: fffffe00099651e0 rdx: 0000000000000000
rcx: 00000000000003aa  r8: 0000000000000001  r9: 0000000000010000
rax: fffff80004064b40 rbx: 0000000000000018 rbp: fffffe00085ffa00
r10: 0000000000000001 r11: 0000000000010000 r12: 00000000000003aa
r13: 0000000000000000 r14: fffff8000352c400 r15: ffffffff823256a5
trap number             = 12
panic: page fault
cpuid = 0
time = 1700061288
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00085ff780
vpanic() at vpanic+0x150/frame 0xfffffe00085ff7d0
panic() at panic+0x43/frame 0xfffffe00085ff830
trap_fatal() at trap_fatal+0x40a/frame 0xfffffe00085ff890
trap_pfault() at trap_pfault+0xae/frame 0xfffffe00085ff8f0
calltrap() at calltrap+0x8/frame 0xfffffe00085ff8f0
--- trap 0xc, rip = 0xffffffff80b26298, rsp = 0xfffffe00085ff9c0, rbp = 0xfffffe00085ffa00 ---
__mtx_lock_flags() at __mtx_lock_flags+0x48/frame 0xfffffe00085ffa00
ng_ksocket_shutdown() at ng_ksocket_shutdown+0x39/frame 0xfffffe00085ffa20
ng_rmnode() at ng_rmnode+0x188/frame 0xfffffe00085ffa50
ng_apply_item() at ng_apply_item+0x4fb/frame 0xfffffe00085ffae0
ng_snd_item() at ng_snd_item+0x2cc/frame 0xfffffe00085ffb20
ngc_send() at ngc_send+0x1b3/frame 0xfffffe00085ffbc0
sosend_generic() at sosend_generic+0x5fe/frame 0xfffffe00085ffc70
sousrsend() at sousrsend+0x79/frame 0xfffffe00085ffcd0
kern_sendit() at kern_sendit+0x1c0/frame 0xfffffe00085ffd60
sendit() at sendit+0xb7/frame 0xfffffe00085ffdb0
sys_sendto() at sys_sendto+0x4d/frame 0xfffffe00085ffe00
amd64_syscall() at amd64_syscall+0x139/frame 0xfffffe00085fff30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00085fff30
--- syscall (133, FreeBSD ELF64, sendto), rip = 0x556e6b62ffa, rsp = 0x556e4385f98, rbp = 0x556e4385fe0 ---
KDB: enter: panic
[ thread pid 730 tid 100070 ]
Stopped at      kdb_enter+0x32: movq    $0,0xde93a3(%rip)
db>

I got this panic using somewhat old 14.0-CURRENT system running in my bhyve.

This bug makes it impossible running net/mpd5 as PPP over TCP server as mpd5 uses ng_ksocket.
Comment 1 commit-hook freebsd_committer freebsd_triage 2023-11-17 17:25:37 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=43f7e21668105cc5a3c66eae5ecef0203c2df62f

commit 43f7e21668105cc5a3c66eae5ecef0203c2df62f
Author:     Gleb Smirnoff <glebius@FreeBSD.org>
AuthorDate: 2023-11-17 17:24:30 +0000
Commit:     Gleb Smirnoff <glebius@FreeBSD.org>
CommitDate: 2023-11-17 17:24:30 +0000

    ng_ksocket: fix accept(2)

    - Provide listen upcall and set it on NGM_KSOCKET_LISTEN
    - Mask EWOULDBLOCK on NGM_KSOCKET_ACCEPT

    Reviewed by:            afedorov
    Differential Revision:  https://reviews.freebsd.org/D42637
    PR:                     272319
    PR:                     275106
    Fixes:                  779f106aa169256b7010a1d8f963ff656b881e92

 sys/netgraph/ng_ksocket.c | 41 +++++++++++++++++++++++++++++++++++------
 1 file changed, 35 insertions(+), 6 deletions(-)
Comment 2 commit-hook freebsd_committer freebsd_triage 2023-11-17 17:25:38 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=efad7cbfdc06e92bcc589a6c0cae2f3bea0d5cb9

commit efad7cbfdc06e92bcc589a6c0cae2f3bea0d5cb9
Author:     Gleb Smirnoff <glebius@FreeBSD.org>
AuthorDate: 2023-11-17 17:23:58 +0000
Commit:     Gleb Smirnoff <glebius@FreeBSD.org>
CommitDate: 2023-11-17 17:23:58 +0000

    ng_ksocket: fix upcall clearing on node shutdown

    Note: imho, the proper solution would be to guarantee that upcalls
    won't ever be called after soclose(), but this isn't the case, yet.
    This change at least makes the node work the way it always worked.

    Reviewed by:            afedorov
    Differential Revision:  https://reviews.freebsd.org/D42636
    PR:                     272319
    PR:                     275106
    Fixes:                  779f106aa169256b7010a1d8f963ff656b881e92

 sys/netgraph/ng_ksocket.c | 21 ++++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)
Comment 3 commit-hook freebsd_committer freebsd_triage 2023-11-30 17:02:40 UTC
A commit in branch stable/14 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=09f4b840bd7cb6427af2a28a10bd839da6dd76d5

commit 09f4b840bd7cb6427af2a28a10bd839da6dd76d5
Author:     Gleb Smirnoff <glebius@FreeBSD.org>
AuthorDate: 2023-11-17 17:23:58 +0000
Commit:     Gleb Smirnoff <glebius@FreeBSD.org>
CommitDate: 2023-11-30 17:01:39 +0000

    ng_ksocket: fix upcall clearing on node shutdown

    Note: imho, the proper solution would be to guarantee that upcalls
    won't ever be called after soclose(), but this isn't the case, yet.
    This change at least makes the node work the way it always worked.

    Reviewed by:            afedorov
    Differential Revision:  https://reviews.freebsd.org/D42636
    PR:                     272319
    PR:                     275106
    Fixes:                  779f106aa169256b7010a1d8f963ff656b881e92

    (cherry picked from commit efad7cbfdc06e92bcc589a6c0cae2f3bea0d5cb9)

 sys/netgraph/ng_ksocket.c | 21 ++++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)
Comment 4 commit-hook freebsd_committer freebsd_triage 2023-11-30 17:02:41 UTC
A commit in branch stable/14 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=ae3c8991cf0db9beff762f90b55e8995326eb894

commit ae3c8991cf0db9beff762f90b55e8995326eb894
Author:     Gleb Smirnoff <glebius@FreeBSD.org>
AuthorDate: 2023-11-17 17:24:30 +0000
Commit:     Gleb Smirnoff <glebius@FreeBSD.org>
CommitDate: 2023-11-30 17:01:40 +0000

    ng_ksocket: fix accept(2)

    - Provide listen upcall and set it on NGM_KSOCKET_LISTEN
    - Mask EWOULDBLOCK on NGM_KSOCKET_ACCEPT

    Reviewed by:            afedorov
    Differential Revision:  https://reviews.freebsd.org/D42637
    PR:                     272319
    PR:                     275106
    Fixes:                  779f106aa169256b7010a1d8f963ff656b881e92

    (cherry picked from commit 43f7e21668105cc5a3c66eae5ecef0203c2df62f)

 sys/netgraph/ng_ksocket.c | 41 +++++++++++++++++++++++++++++++++++------
 1 file changed, 35 insertions(+), 6 deletions(-)
Comment 5 Gleb Smirnoff freebsd_committer freebsd_triage 2023-11-30 17:05:04 UTC
Fixes to ng_ksocket merged to stable/14. With INVARIANTS stable/14 would still panic. However, problem now being catched by INVARIANTS was there always, so can be ignored.

The problem reported by INVARIANTS fixed in the main branch. The change is too intrusive to be merged to a stable branch.
Comment 6 Eugene Grosbein freebsd_committer freebsd_triage 2023-12-01 07:34:37 UTC
(In reply to Gleb Smirnoff from comment #5)

Thank you. Should it work if I merge both fixes to stable/13?
Comment 7 Gleb Smirnoff freebsd_committer freebsd_triage 2023-12-01 16:44:59 UTC
I don't know for sure, but very likely. Try it.
Comment 8 Eugene Grosbein freebsd_committer freebsd_triage 2024-09-04 16:03:20 UTC
*** Bug 281268 has been marked as a duplicate of this bug. ***
Comment 9 commit-hook freebsd_committer freebsd_triage 2024-09-04 16:09:20 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=881886673a3423926c8b5f0955c4b966317f8ae8

commit 881886673a3423926c8b5f0955c4b966317f8ae8
Author:     Gleb Smirnoff <glebius@FreeBSD.org>
AuthorDate: 2023-11-17 17:24:30 +0000
Commit:     Eugene Grosbein <eugen@FreeBSD.org>
CommitDate: 2024-09-04 16:07:52 +0000

    MFC: ng_ksocket: fix accept(2)

    - Provide listen upcall and set it on NGM_KSOCKET_LISTEN
    - Mask EWOULDBLOCK on NGM_KSOCKET_ACCEPT

    Reviewed by:            afedorov
    Differential Revision:  https://reviews.freebsd.org/D42637
    PR:                     272319
    PR:                     275106
    Fixes:                  779f106aa169256b7010a1d8f963ff656b881e92

    (cherry picked from commit 43f7e21668105cc5a3c66eae5ecef0203c2df62f)

 sys/netgraph/ng_ksocket.c | 41 +++++++++++++++++++++++++++++++++++------
 1 file changed, 35 insertions(+), 6 deletions(-)
Comment 10 commit-hook freebsd_committer freebsd_triage 2024-09-04 16:21:24 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=5d0ad5fffbb3b5a71405a47b9b74d7920f9428fd

commit 5d0ad5fffbb3b5a71405a47b9b74d7920f9428fd
Author:     Gleb Smirnoff <glebius@FreeBSD.org>
AuthorDate: 2023-11-17 17:23:58 +0000
Commit:     Eugene Grosbein <eugen@FreeBSD.org>
CommitDate: 2024-09-04 16:20:18 +0000

    MFC: ng_ksocket: fix upcall clearing on node shutdown

    Note: imho, the proper solution would be to guarantee that upcalls
    won't ever be called after soclose(), but this isn't the case, yet.
    This change at least makes the node work the way it always worked.

    Reviewed by:            afedorov
    Differential Revision:  https://reviews.freebsd.org/D42636
    PR:                     272319
    PR:                     275106
    Fixes:                  779f106aa169256b7010a1d8f963ff656b881e92

    (cherry picked from commit efad7cbfdc06e92bcc589a6c0cae2f3bea0d5cb9)

 sys/netgraph/ng_ksocket.c | 21 ++++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)
Comment 11 commit-hook freebsd_committer freebsd_triage 2024-09-04 16:35:28 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=1ec3289e481ed004f2820abf9a7d56460459d0f4

commit 1ec3289e481ed004f2820abf9a7d56460459d0f4
Author:     Eugene Grosbein <eugen@FreeBSD.org>
AuthorDate: 2024-09-04 16:32:32 +0000
Commit:     Eugene Grosbein <eugen@FreeBSD.org>
CommitDate: 2024-09-04 16:32:32 +0000

    ng_ksocket(4): unbreak after previous commit

    Fix locking.
    This is direct commit to stable/13 due to different code base.

    PR:             272319, 275106
    Fixes:          5d0ad5fffbb3b5a71405a47b9b74d7920f9428fd

 sys/netgraph/ng_ksocket.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)