Bug 203735 - Transparent interception of ipv6 with squid and pf causes panic
Summary: Transparent interception of ipv6 with squid and pf causes panic
Status: Closed Overcome By Events
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 10.2-STABLE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-pf (Nobody)
URL:
Keywords: crash, needs-patch, needs-qa
Depends on:
Blocks:
 
Reported: 2015-10-13 08:28 UTC by kraduk
Modified: 2021-06-30 01:46 UTC (History)
7 users (show)

See Also:
koobs: mfc-stable11?
koobs: mfc-stable10?


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description kraduk 2015-10-13 08:28:48 UTC

    
Comment 1 kraduk 2015-10-13 08:40:19 UTC
I am getting regular kernel panics when I do transparent web interception with squid and pf. I am unsure of whether this is an issue with squid or the pf kernel module

Here is the kernel backtrace

(kgdb) bt
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
#1  0xffffffff805f4852 in kern_reboot (howto=260) at /build/stable/usr/src/sys/kern/kern_shutdown.c:451
#2  0xffffffff805f4c35 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /build/stable/usr/src/sys/kern/kern_shutdown.c:758
#3  0xffffffff805f4ac3 in panic (fmt=0x0) at /build/stable/usr/src/sys/kern/kern_shutdown.c:687
#4  0xffffffff808c68bb in trap_fatal (frame=<value optimized out>, eva=<value optimized out>) at /build/stable/usr/src/sys/amd64/amd64/trap.c:851
#5  0xffffffff808c6bbd in trap_pfault (frame=0xfffffe011bc6c2e0, usermode=<value optimized out>) at /build/stable/usr/src/sys/amd64/amd64/trap.c:674
#6  0xffffffff808c625a in trap (frame=0xfffffe011bc6c2e0) at /build/stable/usr/src/sys/amd64/amd64/trap.c:440
#7  0xffffffff808ac522 in calltrap () at /build/stable/usr/src/sys/amd64/amd64/exception.S:236
#8  0xffffffff807f2d19 in sa6_recoverscope (sin6=0xfffff800289c60c0) at /build/stable/usr/src/sys/netinet6/scope6.c:408
#9  0xffffffff807d428f in in6_mapped_peeraddr (so=<value optimized out>, nam=0xfffffe011bc6c550) at /build/stable/usr/src/sys/netinet6/in6_pcb.c:455
#10 0xffffffff805b02c8 in export_fd_to_sb (data=0xfffff80006e692b8, type=2, fd=75, fflags=7, refcnt=1, offset=0, rightsp=<value optimized out>, efbuf=0xfffff8002a834000)
    at /build/stable/usr/src/sys/kern/kern_descrip.c:3723
#11 0xffffffff805afb00 in kern_proc_filedesc_out (p=<value optimized out>, sb=<value optimized out>, maxlen=<value optimized out>) at /build/stable/usr/src/sys/kern/kern_descrip.c:3566
#12 0xffffffff8059ca3d in note_procstat_files (arg=0xfffff80006b50000, sb=0xfffff80091702580, sizep=0xfffffe011bc6c7c8) at /build/stable/usr/src/sys/kern/imgact_elf.c:1848
#13 0xffffffff8059a624 in elf64_coredump (td=0xfffff80006cf1000, vp=0xfffff800383f1760, limit=9223372036854775807, flags=<value optimized out>)
    at /build/stable/usr/src/sys/kern/imgact_elf.c:1573
#14 0xffffffff805f824c in sigexit (td=0xfffff80006cf1000, sig=6) at /build/stable/usr/src/sys/kern/kern_sig.c:3332
#15 0xffffffff805f88a6 in postsig (sig=<value optimized out>) at /build/stable/usr/src/sys/kern/kern_sig.c:2877
#16 0xffffffff80640787 in ast (framep=<value optimized out>) at /build/stable/usr/src/sys/kern/subr_trap.c:281
#17 0xffffffff808ac870 in Xfast_syscall () at /build/stable/usr/src/sys/amd64/amd64/exception.S:421
#18 0x000000080264872a in ?? ()


I updated the kernel to the latest a few days ago but it still happens. Squid is also the latest version in ports

FreeBSD XXX 10.2-STABLE FreeBSD 10.2-STABLE #7: Wed Oct  7 09:17:12 BST 2015     root@r2:/build/stable/usr/obj/build/stable/usr/src/sys/me  amd64


squid -v
Squid Cache: Version 3.5.9
Service Name: squid
configure options:  '--with-default-user=squid' '--bindir=/usr/local/sbin' '--sbindir=/usr/local/sbin' '--datadir=/usr/local/etc/squid' '--libexecdir=/usr/local/libexec/squid' '--localstatedir=/var' '--sysconfdir=/usr/local/etc/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid/squid.pid' '--with-swapdir=/var/squid/cache' '--without-gnutls' '--enable-auth' '--enable-build-info' '--enable-loadable-modules' '--enable-removal-policies=lru heap' '--disable-epoll' '--disable-linux-netfilter' '--disable-linux-tproxy' '--disable-translation' '--disable-arch-native' '--disable-eui' '--enable-cache-digests' '--disable-delay-pools' '--disable-ecap' '--disable-esi' '--disable-follow-x-forwarded-for' '--enable-htcp' '--enable-icap-client' '--enable-icmp' '--enable-ident-lookups' '--enable-ipv6' '--enable-kqueue' '--with-large-files' '--disable-http-violations' '--without-nettle' '--disable-snmp' '--enable-ssl' '--enable-ssl-crtd' '--disable-stacktraces' '--disable-ipf-transparent' '--disable-ipfw-transparent' '--enable-pf-transparent' '--with-nat-devpf' '--enable-forw-via-db' '--enable-wccp' '--enable-wccpv2' '--with-heimdal-krb5=/usr' 'CFLAGS=-I/usr/include -pipe  -I/usr/include -g -fstack-protector -fno-strict-aliasing' 'LDFLAGS=-L/usr/lib  -pthread  -L/usr/lib -fstack-protector' 'LIBS=-lkrb5 -lgssapi -lgssapi_krb5 ' 'KRB5CONFIG=/usr/bin/krb5-config' '--enable-auth-basic=DB SMB_LM MSNT-multi-domain NCSA PAM POP3 RADIUS fake getpwnam' '--enable-auth-digest=file' '--enable-external-acl-helpers=file_userip time_quota unix_group' '--enable-auth-negotiate=kerberos wrapper' '--enable-auth-ntlm=fake smb_lm' '--enable-storeio=ufs aufs diskd' '--enable-disk-io=AIO Blocking IpcIo Mmapped DiskThreads DiskDaemon' '--enable-log-daemon-helpers=file' '--enable-url-rewrite-helpers=fake' '--enable-storeid-rewrite-helpers=file' '--with-openssl=/usr' '--disable-optimizations' '--enable-debug-cbdata' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/info/' '--build=amd64-portbld-freebsd10.2' 'build_alias=amd64-portbld-freebsd10.2' 'CC=/usr/local/libexec/ccache/world/cc' 'CPPFLAGS=' 'CXX=/usr/local/libexec/ccache/world/c++' 'CXXFLAGS=-pipe -I/usr/include -g -fstack-protector -fno-strict-aliasing ' 'CPP=cpp' --enable-ltdl-convenience


pf ipv6 config is

# pfctl -sa | grep inet6
rdr pass on private inet6 proto tcp from ! <free> to ! (private:network) port = http -> 2001:XXX::65 port 3127
rdr pass on private inet6 proto tcp from ! <ssl_free> to ! (private:network) port = https -> 2001:XXX::65 port 3129
block drop in on tun0 inet6 all
block drop in on ipv6he inet6 all
pass out on ipv6he inet6 all flags S/SA keep state (if-bound)
pass in on ipv6he inet6 from 2001:XXX::/126 to 2001:XXX::/126 flags S/SA keep state (if-bound)
pass in inet6 from 2001:YYY::/64 to any flags S/SA keep state (if-bound)
pass in inet6 from 2001:YYY::/64 to any flags S/SA keep state (if-bound)

# ls -l /dev/pf
crwxrwx---  1 root  squid  0x51 Oct 12 17:34 /dev/pf


these are my listen lines for squid

http_port [2001:xxx::65]:3127 intercept
http_port [2001:xxx::65]:3128 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB  cert=/jails/tproxy/opt/qlproxy/etc/myca.pem
https_port [2001:xxx::65]:3129 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB  cert=/jails/tproxy/opt/qlproxy/etc/myca.pem
Comment 2 Pavel Timofeev 2016-04-20 18:30:18 UTC
(In reply to kraduk from comment #1)
please, try squid from ports tree. We included patch for similar crash from squid-3.5.15_2 version
Comment 3 Pavel Timofeev 2016-04-21 12:02:41 UTC
(In reply to timp87 from comment #2)
I meant '... starting from squid-3.5.15_2 version'
Comment 4 Andrey V. Elsukov freebsd_committer freebsd_triage 2016-08-24 06:11:54 UTC
(In reply to kraduk from comment #1)
> I am getting regular kernel panics when I do transparent web interception
> with squid and pf. I am unsure of whether this is an issue with squid or the
> pf kernel module

It is obvious, the problem is in kernel. User app should not trigger kernel panic.
Comment 5 Andrey V. Elsukov freebsd_committer freebsd_triage 2016-08-24 06:32:55 UTC
It was long time ago, but can you show your kernel config? What is the difference from GENERIC?
Comment 6 Kubilay Kocak freebsd_committer freebsd_triage 2016-08-24 13:32:28 UTC
Open until assigned
Comment 7 Kristof Provost freebsd_committer freebsd_triage 2017-03-20 04:41:03 UTC
The good news is this no longer panics, but it still doesn't work.

This turns out to be somewhat tricky. 
The underlying problem is one of address scope.

It can be fixed on the receive side with a patch like this:

diff --git a/sys/netpfil/pf/pf.c b/sys/netpfil/pf/pf.c
index 81290f91b40..d68f81ddf15 100644
--- a/sys/netpfil/pf/pf.c
+++ b/sys/netpfil/pf/pf.c
@@ -6538,8 +6538,12 @@ done:
            pd.proto == IPPROTO_UDP) && s != NULL && s->nat_rule.ptr != NULL &&
            (s->nat_rule.ptr->action == PF_RDR ||
            s->nat_rule.ptr->action == PF_BINAT) &&
           IN6_IS_ADDR_LOOPBACK(&pd.dst->v6))
-               m->m_flags |= M_SKIP_FIREWALL;
+               m->m_flags |= M_SKIP_FIREWALL | M_FASTFWD_OURS;

This tells ip6_input() to skip the scope checks, which seems appropriate.
It still fails on the reply packet though, so this doesn't actually fix the whole use case.
Comment 8 Dan Langille freebsd_committer freebsd_triage 2020-12-28 15:20:17 UTC
I stumbled across this PR while having IPv6 issues - those problems were configuration issues since resolved.

I am using FreeBSD 12.2 and doing this:

PUBLIC="xn0"

FRESHPORTS_WWW_JAIL="127.163.0.80"
FRESHPORTS_WWW_JAIL_IPV6="fd00::80"


nat on $PUBLIC from 127.163.0.0/24 to any -> 10.0.17.21

rdr pass on $PUBLIC inet  proto tcp from any to ($PUBLIC) port = http  -> $FRESHPORTS_WWW_JAIL
rdr pass on $PUBLIC inet  proto tcp from any to ($PUBLIC) port = https -> $FRESHPORTS_WWW_JAIL

rdr pass on $PUBLIC inet6 proto tcp from any to ($PUBLIC) port = http  -> $FRESHPORTS_WWW_JAIL_IPV6
rdr pass on $PUBLIC inet6 proto tcp from any to ($PUBLIC) port = https -> $FRESHPORTS_WWW_JAIL_IPV6

pass all
Comment 9 Mark Linimon freebsd_committer freebsd_triage 2021-06-30 01:46:15 UTC
^Triage: overcome by events.