Comment 1 Michael Tuexen 2019-01-09 17:24:56 UTC

I understand the system settings you are using. Does the system panic just after booting or do you need to perform any action to trigger the panic?

Can't help much, but I was using HTCP for some time on 11-STABLE and 12-STABLE. And I never faced such issues.

I wasn't actually enabling:
net.inet.tcp.cc.htcp.adaptive_backoff=1
net.inet.tcp.cc.htcp.rtt_scaling=1

But never faced any core dumps. And I'm also using my own custom kernel, that is much closer to MINIMAL.

(In reply to Michael Tuexen from comment #1)

[...]
Dec 27 11:24:21 hq kernel: mfid0 on mfi0
Dec 27 11:24:21 hq kernel: mfid0: 3430848MB (7026376704 sectors) RAID volume (no label) is optimal
Dec 27 11:24:21 hq kernel: Trying to mount root from ufs:/dev/mfid0p3 [rw]...
Dec 27 11:24:21 hq kernel: cd0 at ahcich0 bus 0 scbus0 target 0 lun 0
Dec 27 11:24:21 hq kernel: cd0: <ASUS DRW-24F1MT 1.00> Removable CD-ROM SCSI device
Dec 27 11:24:21 hq kernel: cd0: Serial Number S1AI6Y
Dec 27 11:24:21 hq kernel: cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
Dec 27 11:24:21 hq kernel: cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
Dec 27 11:24:21 hq kernel: WARNING: / was not properly dismounted
Dec 27 11:24:21 hq kernel: [2] uhub1: 2 ports with 2 removable, self powered
Dec 27 11:24:21 hq kernel: [2] uhub2: 2 ports with 2 removable, self powered
Dec 27 11:24:21 hq kernel: [2] uhub0: 21 ports with 21 removable, self powered
Dec 27 11:24:21 hq kernel: [2] ugen0.2: <ATEN UC-10KM V1.3.124> at usbus0
Dec 27 11:24:21 hq kernel: [2] ukbd0 on uhub0
Dec 27 11:24:21 hq kernel: [2] ukbd0: <HID Keyboard> on usbus0
Dec 27 11:24:21 hq kernel: [2] kbd2 at ukbd0
Dec 27 11:24:21 hq kernel: [2] ums0 on uhub0
Dec 27 11:24:21 hq kernel: [2] ums0: <HID Mouse> on usbus0
Dec 27 11:24:21 hq kernel: [2] ums0: 5 buttons and [XYZ] coordinates ID=0
Dec 27 11:24:21 hq kernel: [2] ugen2.2: <vendor 0x8087 product 0x8000> at usbus2
Dec 27 11:24:21 hq kernel: [2] uhub3 on uhub1
Dec 27 11:24:21 hq kernel: [2] uhub3: <vendor 0x8087 product 0x8000, class 9/0, rev 2.00/0.05, addr 2> on usbus2
Dec 27 11:24:21 hq kernel: [2] ugen1.2: <vendor 0x8087 product 0x8008> at usbus1
Dec 27 11:24:21 hq kernel: [2] uhub4 on uhub2
Dec 27 11:24:21 hq kernel: [2] uhub4: <vendor 0x8087 product 0x8008, class 9/0, rev 2.00/0.05, addr 2> on usbus1
Dec 27 11:24:21 hq kernel: [3] uhub3: 6 ports with 6 removable, self powered
Dec 27 11:24:21 hq kernel: [3] uhub4: 6 ports with 6 removable, self powered
Dec 27 11:24:21 hq kernel: [3] ugen0.3: <American Power Conversion Smart-UPS X 1500 FWUPS 09.1  ID20> at usbus0
Dec 27 11:24:21 hq kernel: [3] uhid0 on uhub0
Dec 27 11:24:21 hq kernel: [3] uhid0: <American Power Conversion Smart-UPS X 1500 FWUPS 09.1  ID20, class 0/0, rev 2.00/1.06, addr 2> on usbus0
Dec 27 11:24:21 hq kernel: [4] random: unblocking device.
Dec 27 11:24:21 hq kernel: [4] lo0: link state changed to UP
Dec 27 11:24:21 hq kernel: [4] re0: link state changed to DOWN
Dec 27 11:24:21 hq kernel: [7] re1: link state changed to DOWN
Dec 27 11:24:21 hq kernel: [8] re0: link state changed to UP
Dec 27 11:24:21 hq kernel: [8] re0.102: link state changed to UP
Dec 27 11:24:21 hq kernel: [8] re0.100: link state changed to UP
Dec 27 11:24:21 hq kernel: [8] re0.222: link state changed to UP
Dec 27 11:24:21 hq kernel: [8] re0.20: link state changed to UP
Dec 27 11:24:21 hq kernel: [8] re0.10: link state changed to UP
Dec 27 11:24:21 hq kernel: [8] re0.11: link state changed to UP
Dec 27 11:24:21 hq kernel: [9] pflog0: promiscuous mode enabled
Dec 27 11:24:50 hq root: /etc/rc: WARNING: failed to start syslogd
Dec 27 11:24:57 hq savecore: reboot after panic: page fault
Dec 27 11:24:57 hq pflogd[51555]: [priv]: log file /var/log/pflog moved to /var/log/pflog.bad.b0d25165
Dec 27 11:24:57 hq savecore: writing core to /var/crash/vmcore.0
Dec 27 11:25:15 hq kernel: .
Dec 27 11:25:15 hq ntpd[10702]: ntpd 4.2.8p11-a (1): Starting
Dec 27 11:25:16 hq ntpd[15675]: leapsecond file ('/var/db/ntpd.leap-seconds.list'): good hash signature
Dec 27 11:25:16 hq ntpd[15675]: leapsecond file ('/var/db/ntpd.leap-seconds.list'): loaded, expire=2019-06-28T00:00:00Z last=2017-01-01T00:00:00Z ofs=37
Dec 27 11:25:17 hq kernel: OK
Dec 27 11:25:18 hq squid[36210]: Squid Parent: will start 1 kids
Dec 27 11:25:18 hq squid[36210]: Squid Parent: (squid-1) process 41959 started
Dec 27 11:25:36 hq kernel:  blanktime.
Dec 27 11:28:42 hq syslogd: kernel boot file is /boot/kernel.old/kernel
Dec 27 11:28:42 hq kernel: [121]
Dec 27 11:28:42 hq syslogd: last message repeated 1 times
Dec 27 11:28:42 hq kernel: [121] Fatal trap 12: page fault while in kernel mode
Dec 27 11:28:42 hq kernel: [121] cpuid = 0; apic id = 00
Dec 27 11:28:42 hq kernel: [121] fault virtual address  = 0x30
Dec 27 11:28:42 hq kernel: [121] fault code             = supervisor read data, page not present
Dec 27 11:28:42 hq kernel: [121] instruction pointer    = 0x20:0xffffffff804eab36
Dec 27 11:28:42 hq kernel: [121] stack pointer          = 0x28:0xfffffe083ed8d920
Dec 27 11:28:42 hq kernel: [121] frame pointer          = 0x28:0xfffffe083ed8d9c0
Dec 27 11:28:42 hq kernel: [121] code segment           = base rx0, limit 0xfffff, type 0x1b
Dec 27 11:28:42 hq kernel: [121]                        = DPL 0, pres 1, long 1, def32 0, gran 1
Dec 27 11:28:42 hq kernel: [121] processor eflags       = interrupt enabled, resume, IOPL = 0
Dec 27 11:28:42 hq kernel: [121] current process                = 12 (swi4: clock (0))
Dec 27 11:28:42 hq kernel: [121] trap number            = 12
Dec 27 11:28:42 hq kernel: [121] panic: page fault
Dec 27 11:28:42 hq kernel: [121] cpuid = 0
Dec 27 11:28:42 hq kernel: [121] KDB: stack backtrace:
Dec 27 11:28:42 hq kernel: [121] #0 0xffffffff805175c7 at kdb_backtrace+0x67
Dec 27 11:28:42 hq kernel: [121] #1 0xffffffff804d3277 at vpanic+0x177
Dec 27 11:28:42 hq kernel: [121] #2 0xffffffff804d30f3 at panic+0x43
Dec 27 11:28:42 hq kernel: [121] #3 0xffffffff807e4c1f at trap_fatal+0x35f
Dec 27 11:28:42 hq kernel: [121] #4 0xffffffff807e4c79 at trap_pfault+0x49
Dec 27 11:28:42 hq kernel: [121] #5 0xffffffff807e4473 at trap+0x2d3
Dec 27 11:28:42 hq kernel: [121] #6 0xffffffff807c661c at calltrap+0x8
Dec 27 11:28:42 hq kernel: [121] #7 0xffffffff804eb029 at softclock+0x79
Dec 27 11:28:42 hq kernel: [121] #8 0xffffffff804a2189 at intr_event_execute_handlers+0xe9
Dec 27 11:28:42 hq kernel: [121] #9 0xffffffff804a2466 at ithread_loop+0xe6
Dec 27 11:28:42 hq kernel: [121] #10 0xffffffff8049f642 at fork_exit+0x82
Dec 27 11:28:42 hq kernel: [121] #11 0xffffffff807c75ce at fork_trampoline+0xe
Dec 27 11:28:42 hq kernel: [121] Uptime: 2m1s
Dec 27 11:28:42 hq kernel: [121] Dumping 1221 out of 32728 MB:..2%..11%..21%..31%..41%..52%..61%..71%..82%..91%

Comment 4 Michael Tuexen 2019-01-11 08:41:21 UTC

I just tested the combination of TCP fast open client/server with HTCP on a FreeBSD head system and it works fine. Will test stable/11 the next day or so...

Comment 5 Michael Tuexen 2019-01-11 21:44:33 UTC

Just tested on stable/11 and releng/11.2 with a web server having TCP fast open enabled and a client requesting TCP fast open. Everything works as expected from my point (TCP fast open is actually used) and the system does not panic. So right now, I can't reproduce it. However, I'm not using Squid...

(In reply to Michael Tuexen from comment #5)

the problem without running squid

again,
compile and load kernel with options TCP_RFC7413

root@hq:~ # sysctl -w net.inet.tcp.fastopen.enabled=1
net.inet.tcp.fastopen.enabled: 0 -> 1

root@hq:~ # kldload cc_htcp
root@hq:~ # kldstat
Id Refs Address            Size     Name
 1    3 0xffffffff80200000 dd2e30   kernel
 2    1 0xffffffff81211000 8e4      cc_htcp.ko

root@hq:~ # sysctl -w net.inet.tcp.cc.algorithm=htcp
net.inet.tcp.cc.algorithm: newreno -> htcp
root@hq:~ # sysctl -w net.inet.tcp.cc.htcp.adaptive_backoff=1
net.inet.tcp.cc.htcp.adaptive_backoff: 0 -> 1
root@hq:~ # sysctl -w net.inet.tcp.cc.htcp.rtt_scaling=1
net.inet.tcp.cc.htcp.rtt_scaling: 0 -> 1
root@hq:~ #
root@hq:~ # sync
root@hq:~ # service

tried to start squid but kernel panic

hq dumped core - see /var/crash/vmcore.2

Mon Jan 14 17:06:50 MSK 2019

FreeBSD hq 11.2-STABLE FreeBSD 11.2-STABLE #2: Mon Jan 14 16:48:06 MSK 2019     root@hq:/usr/src/sys/amd64/compile/HQ
amd64

panic: page fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
[308]
[308]
[308] Fatal trap 12: page fault while in kernel mode
[308] cpuid = 0; apic id = 00
[308] fault virtual address     = 0x30
[308] fault code                = supervisor read data, page not present
[308] instruction pointer       = 0x20:0xffffffff804eab36
[308] stack pointer             = 0x28:0xfffffe083ed92920
[308] frame pointer             = 0x28:0xfffffe083ed929c0
[308] code segment              = base rx0, limit 0xfffff, type 0x1b
[308]                   = DPL 0, pres 1, long 1, def32 0, gran 1
[308] processor eflags  = interrupt enabled, resume, IOPL = 0
[308] current process           = 12 (swi4: clock (0))
[308] trap number               = 12
[308] panic: page fault
[308] cpuid = 0
[308] KDB: stack backtrace:
[308] #0 0xffffffff805175c7 at kdb_backtrace+0x67
[308] #1 0xffffffff804d3277 at vpanic+0x177
[308] #2 0xffffffff804d30f3 at panic+0x43
[308] #3 0xffffffff807e4c1f at trap_fatal+0x35f
[308] #4 0xffffffff807e4c79 at trap_pfault+0x49
[308] #5 0xffffffff807e4473 at trap+0x2d3
[308] #6 0xffffffff807c661c at calltrap+0x8
[308] #7 0xffffffff804eb029 at softclock+0x79
[308] #8 0xffffffff804a2189 at intr_event_execute_handlers+0xe9
[308] #9 0xffffffff804a2466 at ithread_loop+0xe6
[308] #10 0xffffffff8049f642 at fork_exit+0x82
[308] #11 0xffffffff807c75ce at fork_trampoline+0xe
[308] Uptime: 5m8s
[308] Dumping 1197 out of 32728 MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/cc_htcp.ko...Reading symbols from /usr/lib/debug//boot/kernel/cc_htcp.ko.debug...done
.
done.
Loaded symbols for /boot/kernel/cc_htcp.ko
#0  doadump (textdump=<value optimized out>) at pcpu.h:230
230     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) #0  doadump (textdump=<value optimized out>) at pcpu.h:230
#1  0xffffffff804d2e8a in kern_reboot (howto=260)
    at ../../../kern/kern_shutdown.c:383
#2  0xffffffff804d32b1 in vpanic (fmt=<value optimized out>,
    ap=<value optimized out>) at ../../../kern/kern_shutdown.c:776
#3  0xffffffff804d30f3 in panic (fmt=<value optimized out>)
    at ../../../kern/kern_shutdown.c:707
#4  0xffffffff807e4c1f in trap_fatal (frame=0xfffffe083ed92860,
    eva=<value optimized out>) at ../../../amd64/amd64/trap.c:877
#5  0xffffffff807e4c79 in trap_pfault (frame=0xfffffe083ed92860,
    usermode=<value optimized out>) at cpufunc.h:832
#6  0xffffffff807e4473 in trap (frame=0xfffffe083ed92860)
    at ../../../amd64/amd64/trap.c:415
#7  0xffffffff807c661c in calltrap () at ../../../amd64/amd64/exception.S:231
#8  0xffffffff804eab36 in softclock_call_cc (c=<value optimized out>,
    cc=0xffffffff80d79e80, direct=0) at ../../../kern/kern_timeout.c:749
#9  0xffffffff804eb029 in softclock (arg=0xffffffff80d79e80)
    at ../../../kern/kern_timeout.c:867
#10 0xffffffff804a2189 in intr_event_execute_handlers (
    p=<value optimized out>, ie=0xfffff80008291600)
    at ../../../kern/kern_intr.c:1336
#11 0xffffffff804a2466 in ithread_loop (arg=0xfffff800082aa880)
    at ../../../kern/kern_intr.c:1349
#12 0xffffffff8049f642 in fork_exit (
    callout=0xffffffff804a2380 <ithread_loop>, arg=<value optimized out>,
    frame=<value optimized out>) at ../../../kern/kern_fork.c:1072
#13 0xffffffff807c75ce in fork_trampoline ()
    at ../../../amd64/amd64/exception.S:972
#14 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
(kgdb)

This goes nowhere. We can't reproduce it and you are not using GENERIC kernel. Can you try any of those:

1. Reproduce it on GENERIC kernel.
2. Try posting your kernel config here.
3. Further minify your test case (by removing pf for example if you are trying to get transparent proxying through pf).

Comment 8 Michael Tuexen 2019-01-14 18:23:51 UTC

(In reply to Sergey Samoyloff from comment #6)
Here is the config I'm using when trying to reproduce it:

tuexen@stable11:~ % uname -a
FreeBSD stable11 11.2-RELEASE-p8 FreeBSD 11.2-RELEASE-p8 #0 r342950: Fri Jan 11 23:37:10 CET 2019     tuexen@stable11:/usr/home/tuexen/releng/11.2/sys/amd64/compile/TCP  amd64
tuexen@stable11:~ % sysctl net.inet.tcp.cc.algorithm
net.inet.tcp.cc.algorithm: htcp
tuexen@stable11:~ % sysctl net.inet.tcp.cc.htcp
net.inet.tcp.cc.htcp.rtt_scaling: 1
net.inet.tcp.cc.htcp.adaptive_backoff: 1
tuexen@stable11:~ % sysctl net.inet.tcp.fastopen.enabled
net.inet.tcp.fastopen.enabled: 1
tuexen@stable11:~ % cat stable/11/sys/amd64/conf/TCP
include         GENERIC
ident           TCP     

makeoptions     WITH_EXTRA_TCP_STACKS=1
options		TCP_RFC7413
options         IPFIREWALL
options         IPFIREWALL_VERBOSE
options         IPFIREWALL_VERBOSE_LIMIT=5
options         IPFIREWALL_DEFAULT_TO_ACCEPT
tuexen@stable11:~ % cat /etc/make.conf 
MODULES_OVERRIDE=tcp cc opensolaris dtrace

I have a HTTP server supporting TCP fast open running and an sshd not using TCP fast open. I can log into the machine without problems, and access the HTTP server actually using TCP fast open.

So the problem is not just using a combination of the CC module and using TCP fast open.

Just to double check: When you compiled you custom kernel with TCP fast open enabled, you also recompiled the kernel modules you are using, right?

(In reply to Volodymyr Kostyrko from comment #7)
GENERIC kernel without TCP_RFC7413,
kernel config, sysctl.conf and make.conf - https://yadi.sk/d/49tTdL-kZJXFSQ

(In reply to Michael Tuexen from comment #8)
>When you compiled you custom kernel with TCP fast open enabled, you also 
>recompiled the kernel modules you are using, right?
yes, all kernel modules

Comment 10 Michael Tuexen 2021-06-24 18:47:37 UTC

Since I was not able the reproduce the issue, closing this now. If the problem persists, please reopen.