Bug 206978
| Summary: | sysutils/fusefs-ntfs: Disable UBLIO as it breaks mkntfs | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Ports & Packages | Reporter: | Zane C. Bowers-Hadley <vvelox> | ||||||
| Component: | Individual Port(s) | Assignee: | freebsd-ports-bugs (Nobody) <ports-bugs> | ||||||
| Status: | Open --- | ||||||||
| Severity: | Affects Many People | CC: | bsdpr, danfe, darin, emaste, freebsd, grahamperrin, hodong, junovitch, jwb, rajhlinux, rank1seeker, samy.mahmoudi, w.schwarzenfeld, ybungalobill | ||||||
| Priority: | --- | Keywords: | needs-qa | ||||||
| Version: | Latest | Flags: | linimon:
maintainer-feedback?
(freebsd) |
||||||
| Hardware: | Any | ||||||||
| OS: | Any | ||||||||
| See Also: | https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194526 | ||||||||
| Bug Depends on: | |||||||||
| Bug Blocks: | 194526 | ||||||||
| Attachments: |
|
||||||||
I have a work in progress update for this port, which removes UBLIO support given the number of issues that have been reported for it. Would you be able to test it if I attach a patch? Preferably with a backed up or test NTFS volume/dataset (In reply to Kubilay Kocak from comment #1) Most definitely. Would be most happy to help. :) Thank you Zane, I'll have something attached here soon :) *** Bug 198566 has been marked as a duplicate of this bug. *** Add new maintainer to CC. fusefs-nf has now 2017.3.23. I think this is overcome by events and could be closed. Is this really still in progress ? 11.2-RELEASE-p4 i386 with fusefs-ntfs-2017.3.23 & libublio-20070103_2 (deinstalled BOTH and recompiled JUST NOW)
Compiling sysutils/fusefs-ntfs AUTO installs devel/libublio (no option)
According to test and fixes(?) below, there are TWO possible reasons for failure:
1) Too big partition/slice size
2) Partition/slice's length spreading to the end of device
Failure always occurs by hanging at 'Creating backup boot sector.', so that is a place to look at in the source code.
FAILS:
======
# gpart show -p ada0
=> 40 312581728 ada0 GPT (149G)
40 312581728 ada0p1 ms-basic-data (149G)
# mkntfs -vIf -p 40 -H 16 -S 63 -L 'NTFS_slice' /dev/ada0p1
Cluster size has been automatically set to 4096 bytes.
Creating NTFS volume structures.
Creating root directory (mft record 5)
Creating $MFT (mft record 0)
Creating $MFTMirr (mft record 1)
Creating $LogFile (mft record 2)
Creating $AttrDef (mft record 4)
Creating $Bitmap (mft record 6)
Creating $Boot (mft record 7)
Creating backup boot sector.
FIX: ==> Not actually. ;)
====
# gpart delete -i 1 ada0
# gpart add -s 20g -t ms-basic-data ada0
# gpart show -p ada0
=> 40 312581728 ada0 GPT (149G)
40 41943040 ada0p1 ms-basic-data (20G)
41943080 270638688 - free - (129G)
# mkntfs -vIf -p 40 -H 16 -S 63 -L 'NTFS_slice' /dev/ada0p1
Cluster size has been automatically set to 4096 bytes.
Creating NTFS volume structures.
Creating root directory (mft record 5)
Creating $MFT (mft record 0)
Creating $MFTMirr (mft record 1)
Creating $LogFile (mft record 2)
Creating $AttrDef (mft record 4)
Creating $Bitmap (mft record 6)
Creating $Boot (mft record 7)
Creating backup boot sector.
Creating $Volume (mft record 3)
Creating $BadClus (mft record 8)
Creating $Secure (mft record 9)
Creating $UpCase (mft record 0xa)
Creating $Extend (mft record 11)
Creating system file (mft record 0xc)
Creating system file (mft record 0xd)
Creating system file (mft record 0xe)
Creating system file (mft record 0xf)
Creating $Quota (mft record 24)
Creating $ObjId (mft record 25)
Creating $Reparse (mft record 26)
Syncing root directory index record.
Syncing $Bitmap.
Syncing $MFT.
Updating $MFTMirr.
Syncing device.
mkntfs completed successfully. Have a nice day.
Make it FAIL/HANG again:
==
# gpart add -t ms-basic-data ada0
# gpart show -p ada0
=> 40 312581728 ada0 GPT (149G)
40 41943040 ada0p1 ms-basic-data (20G)
41943080 270638688 ada0p2 ms-basic-data (129G)
# mkntfs -vIf -p 41943080 -H 16 -S 63 -L 'NTFS_slice_2' /dev/ada0p2
Cluster size has been automatically set to 4096 bytes.
Creating NTFS volume structures.
Creating root directory (mft record 5)
Creating $MFT (mft record 0)
Creating $MFTMirr (mft record 1)
Creating $LogFile (mft record 2)
Creating $AttrDef (mft record 4)
Creating $Bitmap (mft record 6)
Creating $Boot (mft record 7)
Creating backup boot sector.
FIX again: ==> Not actually. ;)
====
# gpart delete -i 2 ada0
# gpart add -s 120g -t ms-basic-data ada0
# gpart show -p ada0
=> 40 312581728 ada0 GPT (149G)
40 41943040 ada0p1 ms-basic-data (20G)
41943080 251658240 ada0p2 ms-basic-data (120G)
293601320 18980448 - free - (9.0G)
# mkntfs -vIf -p 41943080 -H 16 -S 63 -L 'NTFS_slice_2' /dev/ada0p2
Cluster size has been automatically set to 4096 bytes.
Creating NTFS volume structures.
Creating root directory (mft record 5)
Creating $MFT (mft record 0)
Creating $MFTMirr (mft record 1)
Creating $LogFile (mft record 2)
Creating $AttrDef (mft record 4)
Creating $Bitmap (mft record 6)
Creating $Boot (mft record 7)
Creating backup boot sector.
Creating $Volume (mft record 3)
Creating $BadClus (mft record 8)
Creating $Secure (mft record 9)
Creating $UpCase (mft record 0xa)
Creating $Extend (mft record 11)
Creating system file (mft record 0xc)
Creating system file (mft record 0xd)
Creating system file (mft record 0xe)
Creating system file (mft record 0xf)
Creating $Quota (mft record 24)
Creating $ObjId (mft record 25)
Creating $Reparse (mft record 26)
Syncing root directory index record.
Syncing $Bitmap.
Syncing $MFT.
Updating $MFTMirr.
Syncing device.
mkntfs completed successfully. Have a nice day.
Domagoj Smolčić
11.2-RELEASE-p9 i386 with fusefs-ntfs-2017.3.23 and WITHOUT libublio (at a time of a previous post, there was no option to choose)
Without libublio, ALL issues from my previous post are gone! ;)
Tested on exactly the same HW with same CMDs.
Reasons of failure:
devel/libublio
libublio doesn't ONLY breaks mkntfs, but it also makes I/O operations on mounted ntfs unreliable.
Regarding See Also: Bug 194526 (sysutils/fusefs-ntfs: ntfs-3g with libublio lost files)
I.e; During copy of >100 GB files from RO mounted ntfs:
----
cp: /mnt/ntfs/file.r05: No such file or directory
Second copy attempt of above file succeeded.
In worse case scenario, some files completed without emitted errors, but i.e; file of 4G ended up being copied as 1.3G file
So I had to use diff tool for verification and it too would sometimes had an issues, like:
diff: /mnt/ntfs/file.mp4: Operation timed out
Second or higher attempts would catch silent errors: $ diff -rq ... ...
----
Only in .:
Files ... and ... differ
.
.
.
----
So I would have to manually ..., ok you get the point.
Based on above facts, this Bug 206978 can be closed, as well as Bug 194526 and all other submitted bugs relating to libublio if existing.
Then new bug dedicated to devel/libublio should be opened and resolved.
Regarding it, all points to it's obsolescence:
Port:
--
Link in pkg-descr:WWW: is invalid
Port has been last touched at 2017-01-18
It:
--
Library of only 17K in size used by BSD and derivatives including Mac OS X.
Exists only because BSD doesn't have a block, but character devices.
Regarding it's code it is all from *BSD, so no linux "translations" required.
It uses BSD style data structure macros via queue.h(taken from OpenBSD as is) and tree.h(taken from FreeBSD with some mods)
Sugar on top is it's size of only 17K, so no one is going to get lost in code.
Now best/shocking part is it's OLDNESS:
libublio has been created at 2006 and file has in it name 2007
Used queue.h,v 1.31 2005/11/25 (OpenBSD)
Used tree.h,v 1.5 2006/01/19 (FreeBSD)
To sum it up, I think devel/libublio doesn't need fixing/patching but RECREATION/FORK with current port as a good template.
PS/HINT: Ports using libublio:
--
sysutils/fusefs-ntfs
sysutils/fusefs-hfsfuse
sysutils/fusefs-exfat
sysutils/exfat-utils
--
I'm going over my WIP for UBLIO option removal again for QA, but will be leaving the ublio bits in in case anyone wants to easily have a play with bringing back UBLIO to a maintained stable state The WIP also includes additions for UUID and DECRYPT After the change lands, it would be great to: 1) Get wider testing of this port with the recent updates in HEAD (13.0-CURRENT) to fuse [1] thanks to Alan Somers 2) Investigate block device caching possibilities in base, if its still relevent/valuable given (1). [1] https://lists.freebsd.org/pipermail/freebsd-current/2019-August/073980.html For quite a while (several years), I've been occasionally getting weird problems with writing to NTFS partitions: rsync(1) would abort file copy with infamous "Input/output error (5)" message. I think it occurs more often when partition becomes closer to full. Today I've hit it again and tried to google some solution. At first, I've found this rather old Debian bug[*] which suggested that downgrading to version 2012.1.15 might "fix" it. I've tried that but the problem did not go away. Then I've searched our Bugzilla and stumbled upon this PR, which suggested to disable UBLIO, albeit for another reason. Surprisingly, it helped! I was able to fill the NTFS partition until "No space left on device (28)" without getting "Input/output error (5)". This is still on 2012.1.15, I didn't try the latest version after the old version worked, but I suspect the problem is indeed with the UBLIO and thus it should be disabled by default (at least) or perhaps even annotated as BROKEN. [*] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=774330 I'm seeing the performance issue also. Oddly, mkntfs works fine on /dev/da1 (and I can mount with ntfs-g3), but hangs on /dev/da1s1. Is this related? root@ghost:~ # mkntfs -fC -L WINBLOWS1909 /dev/da0p1 The partition start sector was not specified for /dev/da0p1 and it could not be obtained automatically. It has been set to 0. The number of sectors per track was not specified for /dev/da0p1 and it could not be obtained automatically. It has been set to 0. The number of heads was not specified for /dev/da0p1 and it could not be obtained automatically. It has been set to 0. Cluster size has been automatically set to 4096 bytes. To boot from a device, Windows needs the 'partition start sector', the 'sectors per track' and the 'number of heads' to be set. Windows will not be able to boot from this device. Creating NTFS volume structures. I confirm that libublio is a culprit. I attempted to make a ntfs on a 916 GiB partition using mkntfs and interrupted the process after *four hours* of activity. I eventually found the current PR, rebuilt fuse-ntfs without the UBLIO option, and eventually managed to format that partition in a blink. This option definitely needs to be disabled, or libublio be fixed, or libublio be replaced with an automatically created gcache instance. Created attachment 226356 [details] terminal output I found this bug after encountering repeated difficulty at <https://forums.FreeBSD.org/threads/81231/post-521947>. There's a screenshot <https://forums.freebsd.org/attachments/1625937980425-png.10549/> of: gstat -p – with apparently endless high read activity for the targeted device. Maintainer reset. I also get this error where it hangs at: "Creating backup boot sector." Using FreeBSD 13.1 # mkntfs --verbose --fast --label 2TB_HD /dev/ada1p1 The partition start sector was not specified for /dev/ada1p1 and it could not be obtained automatically. It has been set to 0. The number of sectors per track was not specified for /dev/ada1p1 and it could not be obtained automatically. It has been set to 0. The number of heads was not specified for /dev/ada1p1 and it could not be obtained automatically. It has been set to 0. Cluster size has been automatically set to 4096 bytes. To boot from a device, Windows needs the 'partition start sector', the 'sectors per track' and the 'number of heads' to be set. Windows will not be able to boot from this device. Creating NTFS volume structures. Creating root directory (mft record 5) Creating $MFT (mft record 0) Creating $MFTMirr (mft record 1) Creating $LogFile (mft record 2) Creating $AttrDef (mft record 4) Creating $Bitmap (mft record 6) Creating $Boot (mft record 7) Creating backup boot sector. Solution provided by "Domagoj Smolčić" works for me by explicitly specifying the "-s" parameter of the sector size on "gpart", then perform the "mkntfs" operation: Here is what worked for me now: View Partitions/Drive: # gpart show Destroy Partition/Drive: # gpart destroy -F ada1 Create a GUID Partition Scheme: # gpart create -s GPT ada1 Create a partition type (ntfs) and add the size (bytes, using “g” for GB): # gpart add -t ms-basic-data -s 1800g ada1 The drive needs a NTFS Filesystem and perform a fast format: # mkntfs --verbose --fast --label 2TB_HD /dev/ada1p1 Mount the Drive: # ntfs-3g /dev/ada1p1 /mnt/2_TB_HD ^Triage: notify current maintainer. It looks as though the "In Progress" state became incorrect when the previous maintainership was reset. |
Created attachment 166662 [details] sysutils_fusefs_ntfs_Makefile.diff This disables UBLIO by default as it breaks mkntfs. It also notes it in UBLIO_DESC at the end as breaking mkntfs. UBLIO hangs for some reason when mkntfs is tries to create the backup boot block.