Bug 210044 - DHCP fails on ixv interface in Amazon EC2
Summary: DHCP fails on ixv interface in Amazon EC2
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-virtualization mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-06-05 08:26 UTC by dfr
Modified: 2016-11-09 11:14 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description dfr 2016-06-05 08:26:58 UTC
To reproduce, start an EC2 instance using a recent FreeBSD 11-CURRENT - I used ami-0709186d which is based on r297692. Select c3.2xlarge as the instance type. The instance starts but fails to get an IP address from DHCP:

Booting [/boot/kernel/kernel]...               
/-\|GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #0 r297692: Fri Apr  8 03:07:13 UTC 2016
    root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
WARNING: WITNESS option enabled, expect reduced performance.
VT(vga): text 80x25
XEN: Hypervisor version 4.2 detected.
CPU: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz (2793.32-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306e4  Family=0x6  Model=0x3e  Stepping=4
  Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0xffba2203<SSE3,PCLMULQDQ,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features=0x200<ERMS>
  XSAVE Features=0x1<XSAVEOPT>
Hypervisor: Origin = "XenVMMXenVMM"
real memory  = 16106127360 (15360 MB)
avail memory = 15552671744 (14832 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <Xen HVM>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 hardware threads
ioapic0: Changing APIC ID to 1
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-47 on motherboard
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff80f09eb0, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <Xen> on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 62500000 Hz quality 950
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc100-0xc10f at device 1.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 1.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xf0000000-0xf1ffffff,0xf3008000-0xf3008fff at device 2.0 on pci0
vgapci0: Boot video device
ixv0: <Intel(R) PRO/10GbE Virtual Function Network Driver, Version - 1.4.6-k> mem 0xf3000000-0xf3003fff,0xf3004000-0xf3007fff at device 3.0 on pci0
ixv0: Using MSIX interrupts with 2 vectors
ixv0: Ethernet address: 12:32:fe:54:c2:3d
ixv0: netmap queues/slots: TX 1/1024, RX 1/1024
xenpci0: <Xen Platform Device> port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff irq 47 at device 31.0 on pci0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
xenpv0: <Xen PV bus> on motherboard
granttable0: <Xen Grant-table Device> on xenpv0
xen_et0: <Xen PV Clock> on xenpv0
Event timer "XENTIMER" frequency 1000000000 Hz quality 950
Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
xenstore0: <XenStore> on xenpv0
evtchn0: <Xen event channel user-space device> on xenpv0
privcmd0: <Xen privileged interface user-space device> on xenpv0
debug0: <Xen debug handler> on xenpv0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
fdc0: No FDOUT register!
ppc0: cannot reserve I/O port range
Timecounters tick every 1.000 msec
xenballoon0: <Xen Balloon Device> on xenstore0
xctrl0: <Xen Control Device> on xenstore0
xs_dev0: <Xenstore user-space device> on xenstore0
xenbusb_front0: <Xen Frontend Devices> on xenstore0
xenbusb_back0: <Xen Backend Devices> on xenstore0
xbd0: 10240MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0
xbd0: attaching as ada0
xbd1: 81909MB <Virtual Block Device> at device/vbd/51728 on xenbusb_front0
xbd1: features: flush, write_barrier
xbd1: synchronize cache commands enabled.
xbd2: 81909MB <Virtual Block Device> at device/vbd/51744 on xenbusb_front0
xbd2: features: flush, write_barrier
xbd2: synchronize cache commands enabled.
SMP: AP CPU #6 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #7 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
GEOM: ada0: the secondary GPT header is not in the last LBA.
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
Growing root partition to fill device
ada0 recovered
ada0p2 resized
super-block backups (for fsck_ffs -b #) at:
 3146176, 3932672, 4719168, 5505664, 6292160, 7078656, 7865152, 8651648,
 9438144, 10224640, 11011136, 11797632, 12584128, 13370624, 14157120, 14943616,
 15730112, 16516608, 17303104, 18089600, 18876096, 19662592, 20449088
Setting hostuuid: ec2cf456-7bc0-642d-8f8e-b06924c23adf.
Setting hostid: 0x5cd99da4.
No suitable dump device was found.
Starting file system checks:
/dev/gpt/rootfs: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/gpt/rootfs: clean, 2217586 free (7290 frags, 276287 blocks, 0.3% fragmentation)
Mounting local file systems:.
Setting up harvesting:[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
Feeding entropy:random: unblocking device.
.
Starting dhclient.
ixv0: link state changed to UP
DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 3
DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 4
DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 4
DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 8
DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 11
DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 12
DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 13
DHCPDISCOVER on ixv0 to 255.255.255.255 port 67 interval 4
No DHCPOFFERS received.
No working leases in persistent database - sleeping.

Starting Network: lo0 ixv0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo 
ixv0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=405bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWTSO>
	ether 12:32:fe:54:c2:3d
	inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255 
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: active
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32
Starting devd.
Generating host.conf.
Waiting 30s for the default route interface: .............................
Fetching EC2 user-data failed.
pkg: Error fetching http://pkg.FreeBSD.org/FreeBSD:11:amd64/latest/Latest/pkg.txz: No address record
A pre-built version of pkg could not be found for your system.
Consider changing PACKAGESITE or installing it from ports: 'ports-mgmt/pkg'.
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/latest, please wait...
pkg: Error fetching http://pkg.FreeBSD.org/FreeBSD:11:amd64/latest/Latest/pkg.txz: No address record
A pre-built version of pkg could not be found for your system.
Consider changing PACKAGESITE or installing it from ports: 'ports-mgmt/pkg'.
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/latest, please wait...
Creating and/or trimming log files.
Starting syslogd.
freebsd-update: Firstboot freebsd-update disabled on 11.0-CURRENT
fetch: http://169.254.169.254/latest/meta-data/block-device-mapping/: Network is unreachable
No ephemeral disks are available, so no swap space is being created.
Clearing /tmp (X related).
Updating motd:.
Mounting late file systems:.
Creating user ec2-user
lock order reversal:
 1st 0xfffff8000a958418 ufs (ufs) @ /usr/src/sys/kern/vfs_syscalls.c:3389
 2nd 0xfffffe03a5842100 bufwait (bufwait) @ /usr/src/sys/ufs/ffs/ffs_vnops.c:263
 3rd 0xfffff80042655b78 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2498
stack backtrace:
#0 0xffffffff80a91a90 at witness_debugger+0x70
#1 0xffffffff80a91984 at witness_checkorder+0xe54
#2 0xffffffff80a0d1d6 at __lockmgr_args+0x4d6
#3 0xffffffff80ce3376 at ffs_lock+0xa6
#4 0xffffffff80fe8d90 at VOP_LOCK1_APV+0x100
#5 0xffffffff80aff4da at _vn_lock+0x9a
#6 0xffffffff80aefb13 at vget+0x63
#7 0xffffffff80ae24dc at vfs_hash_get+0xcc
#8 0xffffffff80cdea00 at ffs_vgetf+0x40
#9 0xffffffff80cd62f1 at softdep_sync_buf+0xb51
#10 0xffffffff80ce3f66 at ffs_syncvnode+0x256
#11 0xffffffff80ce3210 at ffs_fsync+0x20
#12 0xffffffff80fe7e57 at VOP_FSYNC_APV+0xf7
#13 0xffffffff80afc180 at sys_fsync+0x170
#14 0xffffffff80e92f6b at amd64_syscall+0x2db
#15 0xffffffff80e7218b at Xfast_syscall+0xfb
lock order reversal:
 1st 0xfffffe03a5842100 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3513
 2nd 0xfffff8000ab1b200 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:281
stack backtrace:
#0 0xffffffff80a91a90 at witness_debugger+0x70
#1 0xffffffff80a91984 at witness_checkorder+0xe54
#2 0xffffffff80a3cc42 at _sx_xlock+0x72
#3 0xffffffff80ce89d7 at ufsdirhash_remove+0x37
#4 0xffffffff80cebb87 at ufs_dirremove+0x127
#5 0xffffffff80cf309e at ufs_rename+0x135e
#6 0xffffffff80fe8280 at VOP_RENAME_APV+0x100
#7 0xffffffff80afc6d8 at kern_renameat+0x4a8
#8 0xffffffff80e92f6b at amd64_syscall+0x2db
#9 0xffffffff80e7218b at Xfast_syscall+0xfb
Fetching SSH public key for ec2-user
ftp: Can't connect to `169.254.169.254:80': Network is unreachable
ftp: Can't connect to `169.254.169.254:http'
Fetching SSH public key failed!
Generating RSA host key.
2048 SHA256:Ewe4xOV55Q6QT0RpJcafi3Xn+OOAjxpqRHfLBYVnHPs root@ (RSA)
Generating ECDSA host key.
256 SHA256:LBowH/9P1OQeHw2UpIe738RJU8Vq+BnD4J5in5B5Lbc root@ (ECDSA)
Generating ED25519 host key.
256 SHA256:2tYcx1kj7n0TLhymu+8AOXb6hS4AXEvllJG49UlcHWQ root@ (ED25519)
Performing sanity check on sshd configuration.
Could not load host key: /etc/ssh/ssh_host_dsa_key

Starting sshd.
Could not load host key: /etc/ssh/ssh_host_dsa_key

ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: 256 SHA256:LBowH/9P1OQeHw2UpIe738RJU8Vq+BnD4J5in5B5Lbc root@ (ECDSA)
ec2: 256 SHA256:2tYcx1kj7n0TLhymu+8AOXb6hS4AXEvllJG49UlcHWQ root@ (ED25519)
ec2: 2048 SHA256:Ewe4xOV55Q6QT0RpJcafi3Xn+OOAjxpqRHfLBYVnHPs root@ (RSA)
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################
Starting sendmail_submit.
Starting sendmail_msp_queue.
Configuring vt: blanktime.
Starting cron.
Starting background file system checks in 60 seconds.
mount: /dev/gpt/rootfs: Device busy

Sun Jun  5 08:08:52 UTC 2016


FreeBSD/amd64 (Amnesiac) (ttyu0)



login: Stopping cron.
Waiting for PIDS: 781.
Stopping sshd.
Waiting for PIDS: 721.
Stopping devd.
Waiting for PIDS: 369.
Writing entropy file:.
Writing early boot entropy file:.
.
Terminated
Jun  5 08:13:57  syslogd: exiting on signal 15

Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...1 0 done
All buffers synced.
lock order reversal:
 1st 0xfffff8000a91f9a0 ufs (ufs) @ /usr/src/sys/kern/vfs_mount.c:1222
 2nd 0xfffff8000a88b240 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2498
stack backtrace:
#0 0xffffffff80a91a90 at witness_debugger+0x70
#1 0xffffffff80a91984 at witness_checkorder+0xe54
#2 0xffffffff80a0d1d6 at __lockmgr_args+0x4d6
#3 0xffffffff80ade95c at vop_stdlock+0x3c
#4 0xffffffff80fe8d90 at VOP_LOCK1_APV+0x100
#5 0xffffffff80aff4da at _vn_lock+0x9a
#6 0xffffffff80aefb13 at vget+0x63
#7 0xffffffff8090857d at devfs_allocv+0xcd
#8 0xffffffff80908083 at devfs_root+0x43
#9 0xffffffff80ae815f at dounmount+0x44f
#10 0xffffffff80af1854 at vfs_unmountall+0xc4
#11 0xffffffff80ad2a37 at bufshutdown+0x3b7
#12 0xffffffff80a34c19 at kern_reboot+0x189
#13 0xffffffff80a34a33 at sys_reboot+0x3e3
#14 0xffffffff80e92f6b at amd64_syscall+0x2db
#15 0xffffffff80e7218b at Xfast_syscall+0xfb
Uptime: 7m17s
acpi0: Powering system off
Close
Comment 1 Colin Percival freebsd_committer 2016-06-13 21:13:43 UTC
If this is the bug I think it is, it's an interrupt routing issue and Avere Systems has sent me and jhb a patch which should fix it.  We should have it into the tree soon.
Comment 2 Colin Percival freebsd_committer 2016-06-24 23:46:16 UTC
This should be fixed by r302184.  Please test again when 11.0-BETA1 is released.
Comment 3 dfr 2016-11-09 11:14:07 UTC
Verified fixed with 11.0-RELEASE