Using XenServer, detaching or changing configuration of xn network interface will lead to kernel panic in 9-STABLE and 10-CURRENT Fix: Workaround is to shutdown the VM How-To-Repeat: Detach the interface when the VM is running
Responsible Changed From-To: freebsd-bugs->freebsd-xen Over to maintainer(s).
Hello, I've been able to reproduce this bug with Xen unstable, here is a fix for it. --- From 6206137f80cbe5812294b1733a86f28e5cdc01bd Mon Sep 17 00:00:00 2001 From: Roger Pau Monne <roger.pau@citrix.com> Date: Thu, 28 Feb 2013 17:43:34 +0100 Subject: [PATCH] xen-netfront: fix detach of network interfaces Remove all the media and the interface when detaching it. Prevents the following panic when detaching an interface (xl network-detach freebsd 1) xn1: detached Fatal trap 12: page fault while in kernel mode cpuid = 3; apic id = 06 fault virtual address = 0xffffff80028ff2a0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff809653af stack pointer = 0x28:0xffffff8117cb4940 frame pointer = 0x28:0xffffff8117cb4980 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 699 (devd) [ thread pid 699 tid 100107 ] Stopped at ifmedia_ioctl+0x3f: movq 0x8(%r12),%rcx db> trace Tracing pid 706 tid 100059 td 0xfffffe0006b69480 ifmedia_ioctl() at ifmedia_ioctl+0x3f/frame 0xffffff80f774c980 ifioctl() at ifioctl+0xeb7/frame 0xffffff80f774ca40 kern_ioctl() at kern_ioctl+0x1ce/frame 0xffffff80f774ca90 sys_ioctl() at sys_ioctl+0x11f/frame 0xffffff80f774cae0 amd64_syscall() at amd64_syscall+0x265/frame 0xffffff80f774cbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff80f774cbf0 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x44c60a, rsp = 0x7fffffffd678, rbp = 0x7fffffffd750 --- PR: 176471 --- sys/dev/xen/netfront/netfront.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/sys/dev/xen/netfront/netfront.c b/sys/dev/xen/netfront/netfront.c index 88641e3..167fd12 100644 --- a/sys/dev/xen/netfront/netfront.c +++ b/sys/dev/xen/netfront/netfront.c @@ -2172,6 +2172,9 @@ static void netif_free(struct netfront_info *info) { netif_disconnect_backend(info); + ifmedia_removeall(&info->sc_media); + ether_ifdetach(info->xn_ifp); + if_free(info->xn_ifp); #if 0 close_netdev(info); #endif -- 1.7.7.5 (Apple Git-26)
Hi (again), I've just realized the previous patch also crashed when trying to detach an interface that's up and running, the following patch allows to detach the interface while running without crashing the FreeBSD kernel. Also, what do you mean by "changing configuration of xn network"? --- From 9c0097ed4775c68769049c61e474ddb62cc11d1f Mon Sep 17 00:00:00 2001 From: Roger Pau Monne <roger.pau@citrix.com> Date: Thu, 28 Feb 2013 17:43:34 +0100 Subject: [PATCH] xen-netfront: fix detach of network interfaces Remove all the media and the interface when detaching it. Prevents the following panic when detaching an interface (xl network-detach freebsd 1) xn1: detached Fatal trap 12: page fault while in kernel mode cpuid = 3; apic id = 06 fault virtual address = 0xffffff80028ff2a0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff809653af stack pointer = 0x28:0xffffff8117cb4940 frame pointer = 0x28:0xffffff8117cb4980 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 699 (devd) [ thread pid 699 tid 100107 ] Stopped at ifmedia_ioctl+0x3f: movq 0x8(%r12),%rcx db> trace Tracing pid 706 tid 100059 td 0xfffffe0006b69480 ifmedia_ioctl() at ifmedia_ioctl+0x3f/frame 0xffffff80f774c980 ifioctl() at ifioctl+0xeb7/frame 0xffffff80f774ca40 kern_ioctl() at kern_ioctl+0x1ce/frame 0xffffff80f774ca90 sys_ioctl() at sys_ioctl+0x11f/frame 0xffffff80f774cae0 amd64_syscall() at amd64_syscall+0x265/frame 0xffffff80f774cbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff80f774cbf0 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x44c60a, rsp = 0x7fffffffd678, rbp = 0x7fffffffd750 --- PR: 176471 --- sys/dev/xen/netfront/netfront.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/sys/dev/xen/netfront/netfront.c b/sys/dev/xen/netfront/netfront.c index 88641e3..3a8b8ea 100644 --- a/sys/dev/xen/netfront/netfront.c +++ b/sys/dev/xen/netfront/netfront.c @@ -2171,7 +2171,14 @@ netfront_detach(device_t dev) static void netif_free(struct netfront_info *info) { + XN_LOCK(info); + xn_stop(info); + XN_UNLOCK(info); + callout_drain(&info->xn_stat_ch); netif_disconnect_backend(info); + ifmedia_removeall(&info->sc_media); + ether_ifdetach(info->xn_ifp); + if_free(info->xn_ifp); #if 0 close_netdev(info); #endif -- 1.7.7.5 (Apple Git-26)
On Feb 28, 2013, at 19:58 , Roger Pau Monn=E9 <roger.pau@citrix.com> = wrote: > Hi (again), >=20 > I've just realized the previous patch also crashed when trying to=20 > detach an interface that's up and running, the following patch allows=20= > to detach the interface while running without crashing the FreeBSD=20 > kernel. >=20 > Also, what do you mean by "changing configuration of xn network"? Using the management console, I think it just does detach / attach.=20 The second patch works in 10-CURRENT, will try 9-STABLE.=20 Pete >=20 > --- > =46rom 9c0097ed4775c68769049c61e474ddb62cc11d1f Mon Sep 17 00:00:00 = 2001 > From: Roger Pau Monne <roger.pau@citrix.com> > Date: Thu, 28 Feb 2013 17:43:34 +0100 > Subject: [PATCH] xen-netfront: fix detach of network interfaces >=20 > Remove all the media and the interface when detaching it. Prevents the > following panic when detaching an interface (xl network-detach freebsd > 1) >=20 > xn1: detached >=20 > Fatal trap 12: page fault while in kernel mode > cpuid =3D 3; apic id =3D 06 > fault virtual address =3D 0xffffff80028ff2a0 > fault code =3D supervisor read data, page not present > instruction pointer =3D 0x20:0xffffffff809653af > stack pointer =3D 0x28:0xffffff8117cb4940 > frame pointer =3D 0x28:0xffffff8117cb4980 > code segment =3D base rx0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 699 (devd) > [ thread pid 699 tid 100107 ] > Stopped at ifmedia_ioctl+0x3f: movq 0x8(%r12),%rcx > db> trace > Tracing pid 706 tid 100059 td 0xfffffe0006b69480 > ifmedia_ioctl() at ifmedia_ioctl+0x3f/frame 0xffffff80f774c980 > ifioctl() at ifioctl+0xeb7/frame 0xffffff80f774ca40 > kern_ioctl() at kern_ioctl+0x1ce/frame 0xffffff80f774ca90 > sys_ioctl() at sys_ioctl+0x11f/frame 0xffffff80f774cae0 > amd64_syscall() at amd64_syscall+0x265/frame 0xffffff80f774cbf0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff80f774cbf0 > --- syscall (54, FreeBSD ELF64, sys_ioctl), rip =3D 0x44c60a, rsp =3D = 0x7fffffffd678, rbp =3D 0x7fffffffd750 --- >=20 > PR: 176471 > --- > sys/dev/xen/netfront/netfront.c | 7 +++++++ > 1 files changed, 7 insertions(+), 0 deletions(-) >=20 > diff --git a/sys/dev/xen/netfront/netfront.c = b/sys/dev/xen/netfront/netfront.c > index 88641e3..3a8b8ea 100644 > --- a/sys/dev/xen/netfront/netfront.c > +++ b/sys/dev/xen/netfront/netfront.c > @@ -2171,7 +2171,14 @@ netfront_detach(device_t dev) > static void > netif_free(struct netfront_info *info) > { > + XN_LOCK(info); > + xn_stop(info); > + XN_UNLOCK(info); > + callout_drain(&info->xn_stat_ch); > netif_disconnect_backend(info); > + ifmedia_removeall(&info->sc_media); > + ether_ifdetach(info->xn_ifp); > + if_free(info->xn_ifp); > #if 0 > close_netdev(info); > #endif > --=20 > 1.7.7.5 (Apple Git-26)
I had to reformat the patch (the one in this PR didn't come through well) but I've tested this on FreeBSD 9.0 and our Xen environment and I can successfully add/remove network interfaces and change their properties without crashes. I still have to attempt this on the FreeBSD 9.1 VMs but I don't expect different results. Thank you very much Roger!
This patch is working well for me as well, using it on 9.1-stable. Can we get it committed? Thanks.
I've been running this in production since March 1st with no problems. I also would like to see this committed ASAP.
Author: gibbs Date: Wed May 22 17:13:03 2013 New Revision: 250913 URL: http://svnweb.freebsd.org/changeset/base/250913 Log: Correct panic on detach of Xen PV network interfaces. dev/xen/netfront: In netif_free(), properly stop the interface and drain any pending timers prior to disconnecting from the backend device. Remove all media and detach our interface object from the system prior to deleting it. PR: kern/176471 Submitted by: Roger Pau Monne <roger.pau@citrix.com> Reviewed by: gibbs MFC after: 1 week Modified: head/sys/dev/xen/netfront/netfront.c Modified: head/sys/dev/xen/netfront/netfront.c ============================================================================== --- head/sys/dev/xen/netfront/netfront.c Wed May 22 16:33:28 2013 (r250912) +++ head/sys/dev/xen/netfront/netfront.c Wed May 22 17:13:03 2013 (r250913) @@ -2171,10 +2171,14 @@ netfront_detach(device_t dev) static void netif_free(struct netfront_info *info) { + XN_LOCK(info); + xn_stop(info); + XN_UNLOCK(info); + callout_drain(&info->xn_stat_ch); netif_disconnect_backend(info); -#if 0 - close_netdev(info); -#endif + ifmedia_removeall(&info->sc_media); + ether_ifdetach(info->xn_ifp); + if_free(info->xn_ifp); } static void _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
Author: gibbs Date: Tue Jun 18 23:14:21 2013 New Revision: 251973 URL: http://svnweb.freebsd.org/changeset/base/251973 Log: MFC Xen fixes: svn rev 250081 Author:gibbs Date:2013-04-29T23:08:13.552620Z ============== xenstore/xenstore.c: Prevent access to invalid memory region when listing an empty directory in the XenStore. Reported by: Bei Guan svn rev 250913 Author:gibbs Date:2013-05-22T17:13:03.545339Z ============== Correct panic on detach of Xen PV network interfaces. dev/xen/netfront: In netif_free(), properly stop the interface and drain any pending timers prior to disconnecting from the backend device. Remove all media and detach our interface object from the system prior to deleting it. PR: kern/176471 Submitted by: Roger Pau Monne <roger.pau@citrix.com> Reviewed by: gibbs svn rev 250917 Author:gibbs Date:2013-05-22T19:22:44.923351Z ============== Fix loss of the emulated keyboard on Xen PV HVM domains. xen/xenbus/xenbusb.c: In xenbusb_probe_children(), do not modify the XenBus state of devices for which we have no PV driver support. An emulated device we do support may share this backend. Hide the node from XenBus instead. This prevents closing the vkbd device, which Qemu's emulated keyboard device is using as the source for keyboard events. Tested with qemu-xen-traditional, qemu-xen and qemu stubdomains, all working as expected. Submitted by: Roger Pau Monne <roger.pau@citrix.com> Reviewed by: gibbs MFC after: 1 week svn rev 251175 Author:gibbs Date:2013-05-31T04:43:19.944993Z ============== Apply the ad* => ada* IDE device name transition to the Xen block front driver. Submitted by: Bei Guan <gbtju85@gmail.com> Reviewed by: gibbs MFC after: 1 week svn rev 251176 Author:gibbs Date:2013-05-31T04:45:59.563195Z ============== Make netif_free() safe to call on a partially initialized softc. Sponsored by: Spectra Logic Corporation MFC after: 1 week svn rev 251729 Author:gibbs Date:2013-06-14T03:31:11.033200Z ============== sys/dev/xen/netfront/netfront.c: In netif_free(), call ifmedia_removeall() after ether_ifdetach() so that bpf listeners are detached, any link state processing is completed, and there is no chance for external reference to media information. Suggested by: yongari Modified: stable/9/sys/dev/xen/blkfront/blkfront.c stable/9/sys/dev/xen/netfront/netfront.c stable/9/sys/xen/xenbus/xenbusb.c stable/9/sys/xen/xenstore/xenstore.c Directory Properties: stable/9/sys/ (props changed) stable/9/sys/dev/ (props changed) Modified: stable/9/sys/dev/xen/blkfront/blkfront.c ============================================================================== --- stable/9/sys/dev/xen/blkfront/blkfront.c Tue Jun 18 23:11:36 2013 (r251972) +++ stable/9/sys/dev/xen/blkfront/blkfront.c Tue Jun 18 23:14:21 2013 (r251973) @@ -148,16 +148,16 @@ blkfront_vdevice_to_unit(uint32_t vdevic int base; const char *name; } info[] = { - {3, 6, 0, "ad"}, /* ide0 */ - {22, 6, 2, "ad"}, /* ide1 */ - {33, 6, 4, "ad"}, /* ide2 */ - {34, 6, 6, "ad"}, /* ide3 */ - {56, 6, 8, "ad"}, /* ide4 */ - {57, 6, 10, "ad"}, /* ide5 */ - {88, 6, 12, "ad"}, /* ide6 */ - {89, 6, 14, "ad"}, /* ide7 */ - {90, 6, 16, "ad"}, /* ide8 */ - {91, 6, 18, "ad"}, /* ide9 */ + {3, 6, 0, "ada"}, /* ide0 */ + {22, 6, 2, "ada"}, /* ide1 */ + {33, 6, 4, "ada"}, /* ide2 */ + {34, 6, 6, "ada"}, /* ide3 */ + {56, 6, 8, "ada"}, /* ide4 */ + {57, 6, 10, "ada"}, /* ide5 */ + {88, 6, 12, "ada"}, /* ide6 */ + {89, 6, 14, "ada"}, /* ide7 */ + {90, 6, 16, "ada"}, /* ide8 */ + {91, 6, 18, "ada"}, /* ide9 */ {8, 4, 0, "da"}, /* scsi disk0 */ {65, 4, 16, "da"}, /* scsi disk1 */ Modified: stable/9/sys/dev/xen/netfront/netfront.c ============================================================================== --- stable/9/sys/dev/xen/netfront/netfront.c Tue Jun 18 23:11:36 2013 (r251972) +++ stable/9/sys/dev/xen/netfront/netfront.c Tue Jun 18 23:14:21 2013 (r251973) @@ -2172,10 +2172,17 @@ netfront_detach(device_t dev) static void netif_free(struct netfront_info *info) { + XN_LOCK(info); + xn_stop(info); + XN_UNLOCK(info); + callout_drain(&info->xn_stat_ch); netif_disconnect_backend(info); -#if 0 - close_netdev(info); -#endif + if (info->xn_ifp != NULL) { + ether_ifdetach(info->xn_ifp); + if_free(info->xn_ifp); + info->xn_ifp = NULL; + } + ifmedia_removeall(&info->sc_media); } static void Modified: stable/9/sys/xen/xenbus/xenbusb.c ============================================================================== --- stable/9/sys/xen/xenbus/xenbusb.c Tue Jun 18 23:11:36 2013 (r251972) +++ stable/9/sys/xen/xenbus/xenbusb.c Tue Jun 18 23:14:21 2013 (r251973) @@ -404,6 +404,31 @@ xenbusb_device_sysctl_init(device_t dev) } /** + * \brief Decrement the number of XenBus child devices in the + * connecting state by one and release the xbs_attch_ch + * interrupt configuration hook if the connecting count + * drops to zero. + * + * \param xbs XenBus Bus device softc of the owner of the bus to enumerate. + */ +static void +xenbusb_release_confighook(struct xenbusb_softc *xbs) +{ + mtx_lock(&xbs->xbs_lock); + KASSERT(xbs->xbs_connecting_children > 0, + ("Connecting device count error\n")); + xbs->xbs_connecting_children--; + if (xbs->xbs_connecting_children == 0 + && (xbs->xbs_flags & XBS_ATTACH_CH_ACTIVE) != 0) { + xbs->xbs_flags &= ~XBS_ATTACH_CH_ACTIVE; + mtx_unlock(&xbs->xbs_lock); + config_intrhook_disestablish(&xbs->xbs_attach_ch); + } else { + mtx_unlock(&xbs->xbs_lock); + } +} + +/** * \brief Verify the existance of attached device instances and perform * probe/attach processing for newly arrived devices. * @@ -417,7 +442,7 @@ xenbusb_probe_children(device_t dev) { device_t *kids; struct xenbus_device_ivars *ivars; - int i, count; + int i, count, error; if (device_get_children(dev, &kids, &count) == 0) { for (i = 0; i < count; i++) { @@ -430,7 +455,30 @@ xenbusb_probe_children(device_t dev) continue; } - if (device_probe_and_attach(kids[i])) { + error = device_probe_and_attach(kids[i]); + if (error == ENXIO) { + struct xenbusb_softc *xbs; + + /* + * We don't have a PV driver for this device. + * However, an emulated device we do support + * may share this backend. Hide the node from + * XenBus until the next rescan, but leave it's + * state unchanged so we don't inadvertently + * prevent attachment of any emulated device. + */ + xenbusb_delete_child(dev, kids[i]); + + /* + * Since the XenStore state of this device + * still indicates a pending attach, manually + * release it's hold on the boot process. + */ + xbs = device_get_softc(dev); + xenbusb_release_confighook(xbs); + + continue; + } else if (error) { /* * Transition device to the closed state * so the world knows that attachment will @@ -579,31 +627,6 @@ xenbusb_nop_confighook_cb(void *arg __un { } -/** - * \brief Decrement the number of XenBus child devices in the - * connecting state by one and release the xbs_attch_ch - * interrupt configuration hook if the connecting count - * drops to zero. - * - * \param xbs XenBus Bus device softc of the owner of the bus to enumerate. - */ -static void -xenbusb_release_confighook(struct xenbusb_softc *xbs) -{ - mtx_lock(&xbs->xbs_lock); - KASSERT(xbs->xbs_connecting_children > 0, - ("Connecting device count error\n")); - xbs->xbs_connecting_children--; - if (xbs->xbs_connecting_children == 0 - && (xbs->xbs_flags & XBS_ATTACH_CH_ACTIVE) != 0) { - xbs->xbs_flags &= ~XBS_ATTACH_CH_ACTIVE; - mtx_unlock(&xbs->xbs_lock); - config_intrhook_disestablish(&xbs->xbs_attach_ch); - } else { - mtx_unlock(&xbs->xbs_lock); - } -} - /*--------------------------- Public Functions -------------------------------*/ /*--------- API comments for these methods can be found in xenbusb.h ---------*/ void Modified: stable/9/sys/xen/xenstore/xenstore.c ============================================================================== --- stable/9/sys/xen/xenstore/xenstore.c Tue Jun 18 23:11:36 2013 (r251972) +++ stable/9/sys/xen/xenstore/xenstore.c Tue Jun 18 23:14:21 2013 (r251973) @@ -307,7 +307,8 @@ split(char *strings, u_int len, u_int *n const char **ret; /* Protect against unterminated buffers. */ - strings[len - 1] = '\0'; + if (len > 0) + strings[len - 1] = '\0'; /* Count the strings. */ *num = extract_strings(strings, /*dest*/NULL, len); _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
Fixed and MCFed to stable-9.