Created attachment 174824 [details] dmesg I tried to install 11.0-RC3 on Hyper-V 2012r2 (Generation 1 VM). Installation failed on disk partitioning step (Auto-UFS): ------------------------Abort----------------- |Partitioning error | | | |An installation step has been aborted. Would| |you like to restart the installation or exit| |the installer? | <Restart> <Exit> | --------------------------------------------- dmesg available in attachment. 10.3 installs without issues to the same VM. such behavior is disaster for our environment.
The same problem reported on support forum https://forums.freebsd.org/threads/57682/
Can you post the hyper-v's VM configuration, i.e. get a snapshot of the VM configuration.
Created attachment 174898 [details] Install FreeBSD 11 through RC3
I tried to install FreeBSD 11 through RC3 iso on my 2012 r2, and there is no issue. In my VM settings, HDD locates on "IDE Controller 0" and LUN 0. DVD locates on "IDE Controller 1" and LUN 0. See my attachment. You can see my 2012 R2 version, and VM settings. But if I exchange the "IDE Controller" for HDD and DVD, then the installation will fail. That is a known issue, and patch is under review (https://reviews.freebsd.org/D7693). Could you please tell me what is your VM settings for HDD and DVD?
Created attachment 174900 [details] 11.0-rc3 on diff IDE- error on partitioning step
Created attachment 174901 [details] 11.0-rc3 on same IDE- boot failed
I did tests in the following scenario: - 2012r2 Datacenter in failover cluster configuration - VM stored on SMB3 share (2012r2 storage server) - all up-to-date patches installed on 2012r2 machines - Cluster validation configuration reports no errors I tested 2 configurations: FreeBSD 11.0-rc3 VM vhdx (200Gb size) and dvd on the different IDEs: installation failed on disk partitioning step (for fixed sized and dynamically sized disk) FreeBSD 11.0-rc3 VM vhdx (200Gb size) and dvd on same IDEs: boot from cd failed I've uploaded relevant print screens. Are you able to install 11.0-RC3 on large virtual disk? (e.g. 200 Gb)
(In reply to Alexander from comment #7) Is the vhdx saved on the SMB3 too? Thanks, sephe
Yes. vxdx and all VM files located smb3. This VM works with FreeBSD 10.3-RELEASE, but 11.0-RC3 failed to install. BR, Alexander
I got the same disk partitioning error while trying to install 11.0-RC3 on 2012r2 hyper-v on local server. 11.0-RC3 fails to install on small (<32Gb) vhdx and 200Gb vhdx on internal (non smb3) server disk.
Created attachment 174948 [details] 11.0-RC3 Critical install error
I have tried to install 11.0-RC3 (ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/11.0/FreeBSD-11.0-RC3-amd64-dvd1.iso) on my 2012r2 (Generation 1VM). I used both SMB3 shared storage and local disk. Both of them work fine. The vhdx I used is dynamic size. I have tried 30G vhdx and 200G vhdx. I guess that must be caused by some environment or version issue. See my 2012r2 systeminfo. You can compare it with your local 2012r2 details by the output of cmd "systeminfo".
Created attachment 174979 [details] Workable 2012R2 systeminfo
Do you mean you install FreeBSD11 RC3 successfully on VHDX > 32G (exclusive 200G)? In other words, is the failure related to VHDX size? (In reply to Arkadiy from comment #10)
I was unable to install 11.0-RC3. Installation failed on disk partitioning step on newly created vhdx (tried 30Gb and 200Gb). Moreover, I tried to install 11.0-RC3, 11.0-RC2 on my laptop with Hyper-V. I’ve got the same error. If I install 10.3 on my Hyper-V server or laptop with Hyper-V, it work fine. I’ve downloaded all .iso files from ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/...
Created attachment 174994 [details] Systeminfo of my Hyper-V Server
Disk partitioning fails because any virtual disks are enumerated and then detached (and periph destroyed). Escaping to the shell and rescanning the bus(ses) with camcontrol rescan re-attaches the disks which allows partitioning and installation to occur, but rebooting the installation results in the virtual disks being detached, followed by a subsequent mountroot panic. Can repro this on multiple 2012 R2 systems, clean install, Hyper-V role, up-to-date with all available Automatic Updates installed, new VMs using default configuration.
Created attachment 174998 [details] dmesg screenshot of 11.0-RC3 installation boot
(In reply to Chris K from comment #17) Just now I installed 11-RC3 VM (ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/11.0/FreeBSD-11.0-RC3-amd64-dvd1.iso) without any issue and the VM worked fine for me. I usd the default VM configure. I tried 20GB and 40GB local vhdx. It's strange I could't reproduce the issue.
I cannot reproduce the issue either. Could you please try my VHDX on your local environment? Both of them are fresh install of FreeBSD 11 RC3 on windows 2012r2 and windows 10. https://honzhancustomer.blob.core.windows.net/freebsd11rc3issues/f11r_2012r2.vhdx https://honzhancustomer.blob.core.windows.net/freebsd11rc3issues/FreeBSD11RC3.vhdx The login account: root passwd: User@123
I figured out that version 11 RC1 and earlier I can install without problem. Version 11 RC2 and RC3 has a problems with installation on my laptop (Windows 8.1 Ent with last updates, Hyper-V, CPU Core i7 2,9G, RAM 6Gb). I do not know why it happening.
This problem is related to MS update KB3172614 or KB3179574. Uninstalling them make VPS or installation working again. But I don't know exact reason why.
Thank you Tomas for this solution. I uninstalled both updates KB3172614 and KB3179574, after that I installed FreeBSD 11 RC3 on Hyper-V VM on my laptop without problems.
Problem with 11.0-RC3 definitely disappears after uninstalling both KB3172614 and KB3179574 updates. Why FreeBSD 10.3 works when these updates installed? On which layer this should be fixed: FreeBSD or Hyper-V?
(In reply to Hongjiang from comment #20) Same mountroot panic. virtual disk is enumerated as da0, then immediately detached.
(In reply to Hongjiang from comment #20) If I uninstall KB3172614 from both my Hyper-V hosts I'm able to successfully install and then boot into FreeBSD 11.0-RC3, as well as successfully boot into the VHDX you supplied. The Windows 8.1 and Windows Server 2012 R2 update history page (https://support.microsoft.com/en-au/help/24717/windows-8-1-windows-server-2012-r2-update-history) calls out the following Hyper-V related change: * When you try to configure connecting a SCSI storage device to a Windows Hyper-V Host, the Host will not recognize the SCSI storage device when Logical Unit (LUN) 0 is not present. I suspect this change (along with the Hyper-V code changes in FreeBSD since 10.3) is triggering the virtual disk being detached from the bus. As a reference, FreeBSD 10.3 release installs correctly with or without KB3172614 installed. So there's definitely a regression in the Hyper-V codebase in FreeBSD 11.0-RC3.
On my local 2012r2, I did not have applied either KB3172614 or KB3179574 updates. Thanks for your investigation.
delete the following VHDX since they are not used. https://honzhancustomer.blob.core.windows.net/freebsd11rc3issues/FreeBSD11RC3.vhdx https://honzhancustomer.blob.core.windows.net/freebsd11rc3issues/f11r_2012r2.vhdx
This seems what I'm hitting on Windows 2012 R2 with all current updates. I get: storvsc0: <Hyper-V IDE Storage Interface> on vmbus0 (probe0:blkvsc0:0:0:0): storvsc scsi_status = 2 (probe0:blkvsc0:0:1:1): invalid LUN 1 ... (probe0:blkvsc0:0:0:1): invalid LUN 1 da0 at blkvsc0 bus 0 scbus1 target 0 lun 0 da0: <specs> da0 at blkvsc0 bus 0 scbus1 target 0 lun 0 da0: <Msft Virtual Disk 1.0> detached I first tried HardenedBSD 11-STABLE, did not work. Then I tried HardenedBSD 10-STABLE, that did not work either. Then I tried FreeBSD 10.1-RELEASE and that did work, as did 10.2-REL and 10.3-REL. 11.0-REL does not work, nor does 11.0-RC3. Finally I installed from 10.3-REL CD and updated to 10-STABLE and the system didn't boot anymore after rebuilding GENERIC. I think it's safe to assume some kind of change in FreeBSD since 11.0-RC1 (2016/08/13) but before 11-RC2 (2016/08/25) causes this. Unless it's known what change causes this I'll start building & testing different dates of STABLE to pinpoint the exact date of the change.
I just noticed that RC3 gives me an error that I don't get on 10-STABLE or 11-REL: (da0:blkvsc0:0:0:0): fatal error, could not acquire reference count It does not recognize and then detach da0, it just doesn't recognize it at all. The other (error) messages are the same (storvsc scsi_status and invalid LUN).
Okay, I looked at the changes to hyper-v files in the window, then I tested a candidate and this change seems to have introduced the bug: https://svnweb.freebsd.org/base/stable/10/sys/dev/hyperv/storvsc/hv_storvsc_drv_freebsd.c?revision=304581&view=markup 10-STABLE r304580 works fine and r304581 and up does not. Could someone more knowledgeable tell me how to work around this problem? Can I just use the most recent revision of 10-STABLE and only revert the files in sys/dev/hyperv/storvsc (that were changed in r304581) back to 304580? Or are there more files that I should also revert to make it work? I need a HardenedBSD 10-STABLE running in h-v tomorrow, so I'm hoping I can just fiddle a bit with the files and get it working until a fix is committed. Removing the MS update on the host would be more of a pain unfortunately. Thanks!
(In reply to Terrence Koeman from comment #31) Thanks for your investigation. We have pending patch for another issue but it can also fix your issue(https://reviews.freebsd.org/D7693). The simplest patch for you to fix it is attached.
Created attachment 175166 [details] Patch to fix the issue
You can try my VHDX which also included this fix. https://honzhanbug212721.blob.core.windows.net/honzhan212721/hz_testBSD11RC3.vhdx
(In reply to Hongjiang from comment #32) I can successfully apply the patch at https://reviews.freebsd.org/D7693 to 10-STABLE, however building GENERIC then fails. I'm currently rebuilding with -j1 to see what's up. The attached patch doesn't apply (the original code in 10-STABLE is missing the if statement just after the comment).
Created attachment 175183 [details] screenshot of errors compiling GENERIC w/ proposed patch on 10-STABLE Screenshot of the first errors compiling GENERIC with the proposed patch applied on 10-STABLE.
(In reply to Terrence Koeman from comment #35) (In reply to Terrence Koeman from comment #36) Hi Terrence, are you using the latest stable/10 branch? The patch context in stable/10 and the HEAD branch should be the same. If it can't apply it cleanly to stable/10 somehow, you may try manually updating the code by replacing the line in stable/10 ccb->ccb_h.status |= CAM_SEL_TIMEOUT; with the related new block of code if (storvsc_get_storage_type(sc->hs_dev) == DRIVER_STORVSC) ccb->ccb_h.status |= CAM_SEL_TIMEOUT; else ccb->ccb_h.status |= CAM_DEV_NOT_THERE; . In stable/10, we also need to replace the sc->hs_dev with sc->hs_dev->device. Hope this can fix the compiling issue.
(In reply to Dexuan Cui from comment #37) Thanks, I did use the latest 10-STABLE. It looked like everything was successfully applied, but the compile just wouldn't work. At the moment I commented out the line "ccb->ccb_h.status |= CAM_SEL_TIMEOUT;", and that seems to have solved the problem for me. I figured that's what was causing the detach. Is it safe to leave it like that until a fix is committed to the tree? Thanks.
(In reply to Terrence Koeman from comment #38) We'll get Hongjiang's fix committed to the HEAD and later will MFC it to stable/10. I suppose this would take a week or so.
(In reply to Terrence Koeman from comment #38) Hi Terrence, To fix this issue for 10-stable, you don't need to apply the whole patch at https://reviews.freebsd.org/D7693; instead, you only need to apply the patch in Comment 33 and change the "sc->hs_dev" to "sc->hs_dev->device": this should build fine according to my test against today's 10-stable code.
Created attachment 175289 [details] Lock order reversal on 11-STABLE with proposed patch applied (In reply to Dexuan Cui from comment #37) I've made the changes you recommended, and I'm noticing frequent lock order reversals. Especially at boot and shutdown. Attached is a screenshot of one, it looks like it has to do with the storage subsystem. Doesn't halt execution, and I haven't seen any corruption. It also does not appear that these reversals become more frequent when disk I/O increases. Is this an indication of something wrong, or just a debugging notice I should ignore or disable?
(In reply to Terrence Koeman from comment #41) This is a well known LOR.
It will be MFCed to 10-stable and 11-stable.
MFC to stable/10 and stable/11 are done.
is this working? i'm sorry, i have't try this on FreebBSD 11. but it works for me on PFSense 2.4 (FreeBSD 11) On the installer boot prompt, select 3. Escape to the loader prompt set hw.ata.disk_enable=1 boot after installation do it again. after boot exit to shell with 8. ee /boot/loader.conf add the line hw.ata.disk_enable="1"
10-stable and 11-stable have fixed this issue. You can try 11-stable if you are using 11-release and encountered this issue.
This issue is causing errors with FreeBSD 11.0-RELEASE (up to p4) running on Azure as well. I've seen multiple instances of VMs failing to boot because of this bug.
It is fixed now (FreeBSD-EN-17:03.hyperv) https://www.freebsd.org/security/advisories/FreeBSD-EN-17:03.hyperv.asc