Summary: | panic during startup of squid inside jail | ||
---|---|---|---|
Product: | Base System | Reporter: | Thomas von Dein <freebsd> |
Component: | kern | Assignee: | Mark Johnston <markj> |
Status: | Closed FIXED | ||
Severity: | Affects Only Me | CC: | markj |
Priority: | --- | ||
Version: | 12.1-STABLE | ||
Hardware: | Any | ||
OS: | Any |
Description
Thomas von Dein
2020-04-24 11:41:40 UTC
After disabling the following kernel options: # Statically Link in accept filters options ACCEPT_FILTER_DATA options ACCEPT_FILTER_DNS options ACCEPT_FILTER_HTTP The system works as expected, squid starts up and the server keeps running. (In reply to Thomas von Dein from comment #0) From frame 11 could you run: (kgdb) p *so (kgdb) p *so->so_listen (kgdb) p *so->so_listen->sol_accept_filter ? Based on the fault address, the accf_callback points to unmapped memory, but that's somewhat surprising since you're apparently compiling the accept filters directly into the kernel. Are there any other accept filters that are being dynamically loaded? (In reply to Mark Johnston from comment #2) Hello Mark, > From frame 11 could you run: > > (kgdb) p *so > (kgdb) p *so->so_listen > (kgdb) p *so->so_listen->sol_accept_filter Yes: (kgdb) f 11 #11 0xffffffff80cf1ebc in soisconnected (so=0xfffff8108ede7368) at /usr/src/sys/kern/uipc_socket.c:3775 3775 ret = head->sol_accept_filter->accf_callback(so, (kgdb) p *so $1 = {so_lock = {lock_object = {lo_name = 0xffffffff81386dc0 "socket", lo_flags = 21168128, lo_data = 0, lo_witness = 0x0}, mtx_lock = 18446735281986889184}, so_count = 0, so_rdsel = {si_tdlist = {tqh_first = 0x0, tqh_last = 0x0}, si_note = {kl_list = {slh_first = 0x0}, kl_lock = 0xffffffff80ceb1c0 <so_rdknl_lock>, kl_unlock = 0xffffffff80ceb240 <so_rdknl_unlock>, kl_assert_locked = 0xffffffff80ceb2a0 <so_rdknl_assert_locked>, kl_assert_unlocked = 0xffffffff80ceb2b0 <so_rdknl_assert_unlocked>, kl_lockarg = 0xfffff8108ede7368, kl_autodestroy = 0}, si_mtx = 0x0}, so_wrsel = {si_tdlist = {tqh_first = 0x0, tqh_last = 0x0}, si_note = {kl_list = { slh_first = 0x0}, kl_lock = 0xffffffff80ceb2c0 <so_wrknl_lock>, kl_unlock = 0xffffffff80ceb340 <so_wrknl_unlock>, kl_assert_locked = 0xffffffff80ceb3a0 <so_wrknl_assert_locked>, kl_assert_unlocked = 0xffffffff80ceb3b0 <so_wrknl_assert_unlocked>, kl_lockarg = 0xfffff8108ede7368, kl_autodestroy = 0}, si_mtx = 0x0}, so_type = 1, so_options = 4, so_linger = 0, so_state = 259, so_pcb = 0xfffff801f21e7988, so_vnet = 0xfffff81080019b40, so_proto = 0xffffffff81b581b0 <inetsw+192>, so_timeo = 0, so_error = 0, so_sigio = 0x0, so_cred = 0xfffff8012cd3ee00, so_label = 0x0, so_gencnt = 23888, so_emuldata = 0x0, so_dtor = 0x0, osd = {osd_nslots = 0, osd_slots = 0x0, osd_next = {le_next = 0x0, le_prev = 0x0}}, so_fibnum = 0, so_user_cookie = 0, so_ts_clock = 0, so_max_pacing_rate = 0, {{so_rcv = {sb_mtx = { lock_object = {lo_name = 0xffffffff813e98c0 "so_rcv", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 18446735281986889184}, sb_sx = {lock_object = {lo_name = 0xffffffff81435c38 "so_rcv_sx", lo_flags = 36896768, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, sb_sel = 0xfffff8108ede7390, sb_state = 0, sb_mb = 0x0, sb_mbtail = 0x0, sb_lastrecord = 0x0, sb_sndptr = 0x0, sb_fnrdy = 0x0, sb_sndptroff = 0, sb_acc = 0, sb_ccc = 0, sb_hiwat = 1049740, sb_mbcnt = 0, sb_mcnt = 0, sb_ccnt = 0, sb_mbmax = 8397920, sb_ctl = 0, sb_lowat = 1, sb_timeo = 0, sb_flags = 2080, sb_upcall = 0xffffffff826e3000, sb_upcallarg = 0x0, sb_aiojobq = {tqh_first = 0x0, tqh_last = 0xfffff8108ede7580}, sb_aiotask = {ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0, ta_func = 0xffffffff80cc7800 <soaio_rcv>, ta_context = 0xfffff8108ede7368}}, so_snd = {sb_mtx = {lock_object = {lo_name = 0xffffffff813fbdb7 "so_snd", lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0}, sb_sx = {lock_object = {lo_name = 0xffffffff8145a998 "so_snd_sx", lo_flags = 36896768, lo_data = 0, lo_witness = 0x0}, sx_lock = 1}, sb_sel = 0xfffff8108ede73e0, sb_state = 0, sb_mb = 0x0, sb_mbtail = 0x0, sb_lastrecord = 0x0, sb_sndptr = 0x0, sb_fnrdy = 0x0, sb_sndptroff = 0, sb_acc = 0, sb_ccc = 0, sb_hiwat = 1049740, sb_mbcnt = 0, sb_mcnt = 0, sb_ccnt = 0, sb_mbmax = 8397920, sb_ctl = 0, sb_lowat = 2048, sb_timeo = 0, sb_flags = 2048, sb_upcall = 0x0, sb_upcallarg = 0x0, sb_aiojobq = { tqh_first = 0x0, tqh_last = 0xfffff8108ede7670}, sb_aiotask = {ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0, ta_func = 0xffffffff80cc8080 <soaio_snd>, ta_context = 0xfffff8108ede7368}}, so_list = { tqe_next = 0x0, tqe_prev = 0xfffff8011cb60828}, so_listen = 0xfffff8011cb606d0, so_qstate = SQ_INCOMP, so_peerlabel = 0x0, so_oobmark = 0}, {sol_incomp = {tqh_first = 0xffffffff813e98c0, tqh_last = 0x1030000}, sol_comp = {tqh_first = 0x0, tqh_last = 0xfffff801047e65e0}, sol_qlen = 2168675384, sol_incqlen = 4294967295, sol_qlimit = 36896768, sol_accept_filter = 0x0, sol_accept_filter_arg = 0x1, sol_accept_filter_str = 0xfffff8108ede7390 "", sol_upcall = 0x0, sol_upcallarg = 0x0, sol_sbrcv_lowat = 0, sol_sbsnd_lowat = 0, sol_sbrcv_hiwat = 0, sol_sbsnd_hiwat = 0, sol_sbrcv_flags = 0, sol_sbsnd_flags = 0, sol_sbrcv_timeo = 0, sol_sbsnd_timeo = 0}}} (kgdb) p *so->so_listen $2 = {so_lock = {lock_object = {lo_name = 0xffffffff81386dc0 "socket", lo_flags = 21168128, lo_data = 0, lo_witness = 0x0}, mtx_lock = 18446735281986889184}, so_count = 2, so_rdsel = {si_tdlist = {tqh_first = 0x0, tqh_last = 0x0}, si_note = {kl_list = {slh_first = 0xfffff810a58bd780}, kl_lock = 0xffffffff80ceb1c0 <so_rdknl_lock>, kl_unlock = 0xffffffff80ceb240 <so_rdknl_unlock>, kl_assert_locked = 0xffffffff80ceb2a0 <so_rdknl_assert_locked>, kl_assert_unlocked = 0xffffffff80ceb2b0 <so_rdknl_assert_unlocked>, kl_lockarg = 0xfffff8011cb606d0, kl_autodestroy = 0}, si_mtx = 0x0}, so_wrsel = { si_tdlist = {tqh_first = 0x0, tqh_last = 0x0}, si_note = {kl_list = {slh_first = 0x0}, kl_lock = 0xffffffff80ceb2c0 <so_wrknl_lock>, kl_unlock = 0xffffffff80ceb340 <so_wrknl_unlock>, kl_assert_locked = 0xffffffff80ceb3a0 <so_wrknl_assert_locked>, kl_assert_unlocked = 0xffffffff80ceb3b0 <so_wrknl_assert_unlocked>, kl_lockarg = 0xfffff8011cb606d0, kl_autodestroy = 0}, si_mtx = 0x0}, so_type = 1, so_options = 4102, so_linger = 0, so_state = 256, so_pcb = 0xfffff810c0ce07a0, so_vnet = 0xfffff81080019b40, so_proto = 0xffffffff81b581b0 <inetsw+192>, so_timeo = 0, so_error = 0, so_sigio = 0x0, so_cred = 0xfffff8012cd3ee00, so_label = 0x0, so_gencnt = 23864, so_emuldata = 0x0, so_dtor = 0x0, osd = {osd_nslots = 0, osd_slots = 0x0, osd_next = {le_next = 0x0, le_prev = 0x0}}, so_fibnum = 0, so_user_cookie = 0, so_ts_clock = 0, so_max_pacing_rate = 0, {{so_rcv = {sb_mtx = {lock_object = {lo_name = 0xfffff8108ede7368 "\300m8\201\377\377\377\377", lo_flags = 2396944032, lo_data = 4294965264, lo_witness = 0x0}, mtx_lock = 18446735282393188408}, sb_sx = {lock_object = {lo_name = 0x100000000 <error: Cannot access memory at address 0x100000000>, lo_flags = 1024, lo_data = 0, lo_witness = 0xfffff80106739540}, sx_lock = 0}, sb_sel = 0x0, sb_state = 0, sb_mb = 0x0, sb_mbtail = 0x80000000001, sb_lastrecord = 0x10000000100000, sb_sndptr = 0x8000800, sb_fnrdy = 0x0, sb_sndptroff = 0, sb_acc = 0, sb_ccc = 0, sb_hiwat = 0, sb_mbcnt = 0, sb_mcnt = 0, sb_ccnt = 0, sb_mbmax = 0, sb_ctl = 0, sb_lowat = 1, sb_timeo = 0, sb_flags = 2048, sb_upcall = 0x0, sb_upcallarg = 0x0, sb_aiojobq = {tqh_first = 0x0, tqh_last = 0xfffff8011cb608e8}, sb_aiotask = { ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0, ta_func = 0xffffffff80cc7800 <soaio_rcv>, ta_context = 0xfffff8011cb606d0}}, so_snd = {sb_mtx = {lock_object = {lo_name = 0xffffffff813fbdb7 "so_snd", lo_flags = 16908288, lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, sb_sx = {lock_object = {lo_name = 0xffffffff8145a998 "so_snd_sx", lo_flags = 36831232, lo_data = 0, lo_witness = 0x0}, sx_lock = 6}, sb_sel = 0xfffff8011cb60748, sb_state = 0, sb_mb = 0x0, sb_mbtail = 0x0, sb_lastrecord = 0x0, sb_sndptr = 0x0, sb_fnrdy = 0x0, sb_sndptroff = 0, sb_acc = 0, sb_ccc = 0, sb_hiwat = 0, sb_mbcnt = 0, sb_mcnt = 0, sb_ccnt = 0, sb_mbmax = 0, sb_ctl = 0, sb_lowat = 2048, sb_timeo = 0, sb_flags = 2048, sb_upcall = 0x0, sb_upcallarg = 0x0, sb_aiojobq = {tqh_first = 0x0, tqh_last = 0xfffff8011cb609d8}, sb_aiotask = {ta_link = { stqe_next = 0x0}, ta_pending = 0, ta_priority = 0, ta_func = 0xffffffff80cc8080 <soaio_snd>, ta_context = 0xfffff8011cb606d0}}, so_list = {tqe_next = 0x0, tqe_prev = 0x0}, so_listen = 0x0, so_qstate = SQ_NONE, so_peerlabel = 0x0, so_oobmark = 0}, {sol_incomp = {tqh_first = 0xfffff8108ede7368, tqh_last = 0xfffff8108ede76a0}, sol_comp = {tqh_first = 0x0, tqh_last = 0xfffff8011cb60838}, sol_qlen = 0, sol_incqlen = 1, sol_qlimit = 1024, sol_accept_filter = 0xfffff80106739540, sol_accept_filter_arg = 0x0, sol_accept_filter_str = 0x0, sol_upcall = 0x0, sol_upcallarg = 0x0, sol_sbrcv_lowat = 1, sol_sbsnd_lowat = 2048, sol_sbrcv_hiwat = 1048576, sol_sbsnd_hiwat = 1048576, sol_sbrcv_flags = 2048, sol_sbsnd_flags = 2048, sol_sbrcv_timeo = 0, sol_sbsnd_timeo = 0}}} (kgdb) p *so->so_listen->sol_accept_filter $3 = {accf_name = "httpready\000\000\000\000\000\000", accf_callback = 0xffffffff826e3000, accf_create = 0x0, accf_destroy = 0x0, accf_next = {sle_next = 0x0}} > Are there any other accept filters that are being dynamically loaded? I don't really know, it's a standard squid from ports. Tom (In reply to Thomas von Dein from comment #3) Thanks, sorry for the delay. Could you also show output from (kgdb) p accf_http_mod (kgdb) p sohashttpget ? Sorry for the delay. Here's the output: (kgdb) p accf_http_mod $4 = {name = 0xffffffff81451a75 "accf_http", evhand = 0xffffffff80cde880 <accept_filt_generic_mod_event>, priv = 0xffffffff81b54168 <accf_http_filter>} (kgdb) p sohashttpget $5 = {int (struct socket *, void *, int)} 0xffffffff80dd87c0 <sohashttpget> Tom (In reply to Thomas von Dein from comment #5) Thanks, I think I can reproduce the problem now. Did you also have accf_http_load="YES" set in /boot/loader.conf? That is, were you enabling the httpready filter in both loader.conf and the kernel configuration (with options ACCEPT_FILTER_HTTP)? Indeed, the option is enabled in /boot/loader.conf. A commit references this bug: Author: markj Date: Tue May 19 18:35:09 UTC 2020 New revision: 361263 URL: https://svnweb.freebsd.org/changeset/base/361263 Log: Define a module version for accept filter modules. Otherwise accept filters compiled into the kernel do not preempt preloaded accept filter modules. Then, the preloaded file registers its accept filter module before the kernel, and the kernel's attempt fails since duplicate accept filter list entries are not permitted. This causes the preloaded file's module to be released, since module_register_init() does a lookup by name, so the preloaded file is unloaded, and the accept filter's callback points to random memory since preload_delete_name() unmaps the file on x86 as of r336505. Add a new ACCEPT_FILTER_DEFINE macro which wraps the accept filter and module definitions, and ensures that a module version is defined. PR: 245870 Reported by: Thomas von Dein <freebsd@daemon.de> MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Changes: head/sys/netinet/accf_data.c head/sys/netinet/accf_dns.c head/sys/netinet/accf_http.c head/sys/sys/socketvar.h (In reply to Thomas von Dein from comment #7) With the recent commit, this type of "double" configuration (compiling the filter into the kernel and preloading accf_http.ko at boot time), will no longer cause a kernel panic. I'll merge it to stable/12 soon. A commit references this bug: Author: markj Date: Tue Jun 2 00:57:49 UTC 2020 New revision: 361717 URL: https://svnweb.freebsd.org/changeset/base/361717 Log: MFC r361263, r361338: Define a module version for accept filter modules. PR: 245870 Changes: _U stable/12/ stable/12/sys/netinet/accf_data.c stable/12/sys/netinet/accf_dns.c stable/12/sys/netinet/accf_http.c stable/12/sys/sys/socketvar.h Thanks for the report and for the help with debugging. |