Bug 229014 - epoch_cb_task enqueued before being added to a queue
Summary: epoch_cb_task enqueued before being added to a queue
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: arm64 Any
: --- Affects Some People
Assignee: Matt Macy
URL:
Keywords: patch
Depends on:
Blocks:
 
Reported: 2018-06-14 14:52 UTC by Andrew Turner
Modified: 2018-06-23 07:15 UTC (History)
1 user (show)

See Also:


Attachments
Patch to use boottime until after SI_SUB_SMP (1.18 KB, application/mbox)
2018-06-14 14:52 UTC, Andrew Turner
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Andrew Turner freebsd_committer 2018-06-14 14:52:52 UTC
Created attachment 194250 [details]
Patch to use boottime until after SI_SUB_SMP

On one of the ThunderX servers in the netperf cluster the epoch code calls GROUPTASK_ENQUEUE before it has a queue to use.

This seems to be because hardclock_cnt is called before taskqgroup_adjust has been run to set the queue.

For now I've updated epoch_call to use the boottime case until after SI_SUB_SMP has run.

The stack trace is:

nfs_diskless: no interface
Release APs...done
Trying to mount root from nfs:192.168.5.1:/tank/users/andrew/cavium [rw]...
gtask: 0xffff00004174c030 ta_flags=2 ta_priority=0 ta_func=0xffff0000003c532c ta_context=0
panic: queue == NULL
cpuid = 19
time = 11
KDB: stack backtrace:
db_trace_self() at db_trace_self_wrapper+0x28
         pc = 0xffff00000067720c  lr = 0xffff0000000d9d74
         sp = 0xffff000040238310  fp = 0xffff000040238520

db_trace_self_wrapper() at vpanic+0x1a8
         pc = 0xffff0000000d9d74  lr = 0xffff000000385a44
         sp = 0xffff000040238530  fp = 0xffff0000402385e0

vpanic() at panic+0x44
         pc = 0xffff000000385a44  lr = 0xffff000000385af4
         sp = 0xffff0000402385f0  fp = 0xffff000040238670

panic() at grouptaskqueue_enqueue+0x124
         pc = 0xffff000000385af4  lr = 0xffff0000003c6b50
         sp = 0xffff000040238680  fp = 0xffff0000402386a0

grouptaskqueue_enqueue() at hardclock_cnt+0x33c
         pc = 0xffff0000003c6b50  lr = 0xffff0000003215c8
         sp = 0xffff0000402386b0  fp = 0xffff000040238710

hardclock_cnt() at handleevents+0xb8
         pc = 0xffff0000003215c8  lr = 0xffff0000006be478
         sp = 0xffff000040238720  fp = 0xffff000040238760

handleevents() at timercb+0x244
         pc = 0xffff0000006be478  lr = 0xffff0000006bec30
         sp = 0xffff000040238770  fp = 0xffff0000402387d0

timercb() at arm_tmr_intr+0x58
         pc = 0xffff0000006bec30  lr = 0xffff000000664b44
         sp = 0xffff0000402387e0  fp = 0xffff0000402387e0

arm_tmr_intr() at intr_event_handle+0xa8
         pc = 0xffff000000664b44  lr = 0xffff000000349e8c
         sp = 0xffff0000402387f0  fp = 0xffff000040238840

intr_event_handle() at intr_isrc_dispatch+0x5c
         pc = 0xffff000000349e8c  lr = 0xffff0000006c0580
         sp = 0xffff000040238850  fp = 0xffff000040238860

intr_isrc_dispatch() at arm_gic_v3_intr+0x138
         pc = 0xffff0000006c0580  lr = 0xffff00000067c828
         sp = 0xffff000040238870  fp = 0xffff0000402388c0

arm_gic_v3_intr() at intr_irq_handler+0x68
         pc = 0xffff00000067c828  lr = 0xffff0000006c03e8
         sp = 0xffff0000402388d0  fp = 0xffff0000402388f0

intr_irq_handler() at handle_el1h_irq+0x70
         pc = 0xffff0000006c03e8  lr = 0xffff000000679130
         sp = 0xffff000040238900  fp = 0xffff000040238a10

handle_el1h_irq() at vm_page_alloc_domain_after+0x2b4
         pc = 0xffff000000679130  lr = 0xffff00000063eb40
         sp = 0xffff000040238a20  fp = 0xffff000040238b30

vm_page_alloc_domain_after() at uma_small_alloc+0x70
         pc = 0xffff00000063eb40  lr = 0xffff00000069316c
         sp = 0xffff000040238b40  fp = 0xffff000040238b60

uma_small_alloc() at keg_alloc_slab+0xe8
         pc = 0xffff00000069316c  lr = 0xffff000000625838
         sp = 0xffff000040238b70  fp = 0xffff000040238bc0

keg_alloc_slab() at keg_fetch_slab+0x110
         pc = 0xffff000000625838  lr = 0xffff000000627e74
         sp = 0xffff000040238bd0  fp = 0xffff000040238c30

keg_fetch_slab() at zone_fetch_slab+0x60
         pc = 0xffff000000627e74  lr = 0xffff0000006276b0
         sp = 0xffff000040238c40  fp = 0xffff000040238c60

zone_fetch_slab() at zone_import+0x64
         pc = 0xffff0000006276b0  lr = 0xffff000000627770
         sp = 0xffff000040238c70  fp = 0xffff000040238ce0

zone_import() at uma_zalloc_arg+0x378
         pc = 0xffff000000627770  lr = 0xffff000000623cf0
         sp = 0xffff000040238cf0  fp = 0xffff000040238d60

uma_zalloc_arg() at nfscl_reqstart+0xe8
         pc = 0xffff000000623cf0  lr = 0xffff000000272f90
         sp = 0xffff000040238d70  fp = 0xffff000040238df0

nfscl_reqstart() at nfsrpc_getdirpath+0x60
         pc = 0xffff000000272f90  lr = 0xffff00000028d224
         sp = 0xffff000040238e00  fp = 0xffff000040238fb0

nfsrpc_getdirpath() at mountnfs+0x81c
         pc = 0xffff00000028d224  lr = 0xffff00000029cf9c
         sp = 0xffff000040238fc0  fp = 0xffff000040239160

mountnfs() at nfs_mount+0x15dc
         pc = 0xffff00000029cf9c  lr = 0xffff00000029b63c
         sp = 0xffff000040239170  fp = 0xffff000040239480

nfs_mount() at vfs_domount+0x668
         pc = 0xffff00000029b63c  lr = 0xffff000000444530
         sp = 0xffff000040239490  fp = 0xffff0000402396b0

vfs_domount() at vfs_donmount+0x710
         pc = 0xffff000000444530  lr = 0xffff00000044391c
         sp = 0xffff0000402396c0  fp = 0xffff000040239740

vfs_donmount() at kernel_mount+0x58
         pc = 0xffff00000044391c  lr = 0xffff000000446bb0
         sp = 0xffff000040239750  fp = 0xffff0000402397a0

kernel_mount() at parse_mount+0x418
         pc = 0xffff000000446bb0  lr = 0xffff000000448d20
         sp = 0xffff0000402397b0  fp = 0xffff0000402398f0

parse_mount() at vfs_mountroot+0x7d0
         pc = 0xffff000000448d20  lr = 0xffff000000447818
         sp = 0xffff000040239900  fp = 0xffff000040239ab0

vfs_mountroot() at start_init+0x28
         pc = 0xffff000000447818  lr = 0xffff00000031f9ac
         sp = 0xffff000040239ac0  fp = 0xffff000040239b50

start_init() at fork_exit+0x7c
         pc = 0xffff00000031f9ac  lr = 0xffff000000346f80
         sp = 0xffff000040239b60  fp = 0xffff000040239b90

fork_exit() at fork_trampoline+0x10
         pc = 0xffff000000346f80  lr = 0xffff000000691e2c
         sp = 0xffff000040239ba0  fp = 0x0000000000000000

KDB: enter: panic
[ thread pid 1 tid 100002 ]
Stopped at      0
db>
Comment 1 Matthew Macy 2018-06-14 21:45:47 UTC
Can I get a backtrace of what's calling it during that window?
Thanks.
Comment 2 commit-hook freebsd_committer 2018-06-23 07:15:08 UTC
A commit references this bug:

Author: mmacy
Date: Sat Jun 23 07:14:09 UTC 2018
New revision: 335578
URL: https://svnweb.freebsd.org/changeset/base/335578

Log:
  epoch(9): Don't trigger taskq enqueue before the grouptaskqs are setup

  If EARLY_AP_STARTUP is not defined it is possible for an epoch to be
  allocated prior to it being possible to call epoch_call without
  issue.

  Based on patch by andrew@

  PR:		229014
  Reported by:	andrew

Changes:
  head/sys/kern/subr_epoch.c