268934 – [ena] Counters are alloced after they are available for reading which can cause a kernel crash

Bug 268934 - [ena] Counters are alloced after they are available for reading which can cause a kernel crash

Summary: [ena] Counters are alloced after they are available for reading which can cau...

Status:	Closed FIXED

Alias:	None

Product:	Base System
Classification:	Unclassified
Component:	kern (show other bugs)
Version:	13.1-RELEASE
Hardware:	amd64 Any

Importance:	--- Affects Only Me
Assignee:	Marcin Wojtas

URL:
Keywords:

Depends on:
Blocks:

Reported:	2023-01-13 16:16 UTC by ghuckriede
Modified:	2024-06-14 13:47 UTC (History)
CC List:	3 users (show)

See Also:

Attachments
Potential Fix (1.04 KB, patch) 2023-01-13 16:16 UTC, ghuckriede	no flags	Details \| Diff
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description ghuckriede 2023-01-13 16:16:47 UTC

Created attachment 239448 [details]
Potential Fix

The kernel panics because the stats are allocated in ena_attach() [@ena.c:3686].
However ena_get_counters() is registered by ena_setup_ifnet()[@ena.c:2402] in
ena_attach() [@ena.c:3663].  Once ether_ifattach() [@ena.c:2434] is done, the
interface is available and can counters can be read before they are allocated.

N.B. Line numbers refer to the following version of the file.
https://cgit.freebsd.org/src/tree/sys/dev/ena/ena.c @ blob c091091fed206a949b11eb751a4d990d66fa181f 

A potential fix that creates the counters before calling ether_attach() has been attached.
N.B. The EC2 instance is not setup to build, so the provided patch is not tested.


Steps to Reproduce: 
root@freebsd:~ # cat ./dump.sh
#!/bin/sh
while true
do
netstat -I ena0
done
root@freebsd:~ # cat ./reset.sh
#!/bin/sh
while true
do
devctl disable ena0
devctl enable ena0
done
root@freebsd:~ # ./dump.sh &
root@freebsd:~ # ./reset.sh &
<SNIP>
ena0: link is UP
ena0: Link is down
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0*  1500 <Link#1>      02:55:a9:7c:3f:bb        0     0     0        0     0     0
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Jan 13 15:50:46 freebsd dhclient[1577]: ena0: not found
Jan 13 15:50:46 ena0: detached
freebsd dhclient[1577]: exiting.
ena0: <ENA adapter>Name    Mtu Netw mem 0x80008000ork       Addres-0x8000bfff irq 37 at device 5.0 on pci0
s              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: ena_com_validate_version() [TID:100093]: ENA device version: 0.10
Name    Mtu Netwena0: ena_com_validate_version() [TID:100093]: ENA controller version: 0.0.1 implementation version 1
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Netwena0: LLQ is not supported. Fallback to host mode policy.
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: detached
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: <ENA adapter> mem 0x80008000-0x8000bfff irq 37 at device 5.0 on pci0
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: ena_com_validate_version() [TID:100093]: ENA device version: 0.10
Name    Mtu Netwena0: ena_com_validate_version() [TID:100093]: ENA controller version: 0.0.1 implementation version 1
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: LLQ is not supported. Fallback to host mode policy.
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Iena0: detached
pkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: <ENA adapter>Name    Mtu Netw mem 0x80008000-0x8000bfff irq 37 at device 5.0 on pci0
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: ena_com_validate_version() [TID:100093]: ENA device version: 0.10
Name    Mtu Netwena0: ena_com_validate_version() [TID:100093]: ENA controller version: 0.0.1 implementation version 1
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: LLQ is not supported. Fallback to host mode policy.
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: detached
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: <ENA adapter> mem 0x80008000-0x8000bfff irq 37 at device 5.0Name    Mtu Netw on pci0
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: ena_com_validate_version() [TID:100093]: ENA device version: 0.10
Name    Mtu Netwena0: ena_com_validate_version() [TID:100093]: ENA controller version: 0.0.1 implementation version 1
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: LLQ is not supported. Fallback to host mode policy.
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: detached
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: <ENA adapter> mem 0x80008000-0x8000bfff irq 37 at device 5.0 on pci0
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: ena_com_validate_version() [TID:100093]: ENA device version: 0.10
ena0: ena_com_validate_version() [TID:100093]: ENA controller version: 0.0.1 implementation version 1
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: LLQ is not supported. Fallback to host mode policy.
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0*  1500 <Link#1>      02:55:a9:7c:3f:bb        0     0     0        0     0     0
Name    Mtu Network       Address              Ipkts Ierrs Idropena0: Link is down
    Opkts Oerrs  Coll
ena0: link is UP
ena0*  1500 <Link#1>      02:55:a9:7c:3f:bb        0     0     0        0     0     0
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0*  1500 <Link#1>      02:55:a9:7c:3f:bb        0     0     0        0     0     0
Jan 13 15:50:48 freebsd dhclient[1678]: ena0: not found
Jan 13 15:50:48 freebsd dhclient[1678]: exiting.
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: detached
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: <ENA adapter> mem 0x80008000-0x8000bfff irq 37 at device 5.0 on pci0
Name    Mtu Netwena0: ena_com_validate_version() [TID:100093]: ENA device version: 0.10
ena0: ena_com_validate_version() [TID:100093]: ENA controller version: 0.0.1 implementation version 1
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: LLQ is not supported. Fallback to host mode policy.
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Netwena0: detached
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: <ENA adapter> mem 0x80008000-0x8000bfff irq 37 at device 5.0Name    Mtu Netw on pci0
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: ena_com_validate_version() [TID:100093]: ENA device version: 0.10
Name    Mtu Netwena0: ena_com_validate_version() [TID:100093]: ENA controller version: 0.0.1 implementation version 1
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: LLQ is not supported. Fallback to host mode policy.
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs ena0: Link is down
 Coll
ena0: Link is down
ena0*  1500 <Linena0: Link is down
k#1>      02:55:a9:7c:3f:bb        0     0     0        0     0     0
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
<SNIP>


Actual Results:
Fatal data abort:
  x0:                0
  x1:                0
  x2:               d8
  x3: ffff0000da508284
  x4: ffff0000da5081a0
  x5: ffff00009ad620d8
  x6:                0
  x7:                0
  x8:                0
  x9:                0
 x10:                0
 x11:                1
 x12: ffff000000e5a250
 x13:                3
 x14:                3
 x15:                0
 x16: ffff000001280d28
 x17: ffff00000050c088
 x18: ffff0000da508250
 x19: ffff0000da508308
 x20: ffffa0001404a000
 x21:                0
 x22:                0
 x23:               d8
 x24: ffffa00001518390
 x25:               18
 x26:               98
 x27: ffff000000e6c000
 x28: ffff00009ad62000
 x29: ffff0000da508250
  sp: ffff0000da508250
  lr: ffff0000005ed6d0
 elr: ffff00000050c0d8
spsr:         80400045
 far:                0
 esr:         96000007
panic: vm_fault failed: ffff00000050c0d8
cpuid = 0
time = 1673625050
KDB: stack backtrace:
#0 0xffff00000051646c at kdb_backtrace+0x60
#1 0xffff0000004c24c0 at vpanic+0x174
#2 0xffff0000004c2348 at panic+0x44
#3 0xffff0000007f48c0 at data_abort+0x204
#4 0xffff0000007d5010 at handle_el1h_sync+0x10
#5 0xffff0000005ed6cc at if_data_copy+0x7c
#6 0xffff0000005ed6cc at if_data_copy+0x7c
#7 0xffff000000625384 at sysctl_iflist+0xe8
#8 0xffff0000006251e0 at sysctl_rtsock+0x26c
#9 0xffff0000004d4634 at sysctl_root_handler_locked+0x118
#10 0xffff0000004d3aa4 at sysctl_root+0x218
#11 0xffff0000004d4094 at userland_sysctl+0x18c
#12 0xffff0000004d3ec8 at sys___sysctl+0x68
#13 0xffff0000007f3e90 at do_el0_sync+0x560
#14 0xffff0000007d50fc at handle_el0_sync+0x38
Uptime: 25m32s
N.B. The kenel dump was not created on the target after reboot, and therefore not included.


Build Date & Hardware:
Target is an AWS EC2 instance with an EC2 serial console connection
root@freebsd:~ # uname -a
FreeBSD freebsd 13.1-RELEASE-p2 FreeBSD 13.1-RELEASE-p2 GENERIC arm64

Comment 1 David Arinzon 2023-01-18 19:48:34 UTC

Thanks for identifying this issue! We'll fix it in our next driver release.

Comment 2 Mark Linimon freebsd_committer

2024-06-14 04:11:45 UTC

^Triage: assign to committer who commented.

To Ed or David: did this fix ever get committed?

Comment 3 ghuckriede 2024-06-14 13:06:33 UTC

It made it into main.

https://cgit.freebsd.org/src/commit/sys/dev/ena/ena.c?id=b9e80b5280b75f2c641d680245df44b8ff26a7b0