Bug 206978 - sysutils/fusefs-ntfs: Disable UBLIO as it breaks mkntfs
Summary: sysutils/fusefs-ntfs: Disable UBLIO as it breaks mkntfs
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: freebsd-ports-bugs (Nobody)
URL:
Keywords: needs-patch, needs-qa
: 198566 (view as bug list)
Depends on:
Blocks: 194526
  Show dependency treegraph
 
Reported: 2016-02-06 15:07 UTC by Zane C. Bowers-Hadley
Modified: 2024-01-21 15:17 UTC (History)
14 users (show)

See Also:
linimon: maintainer-feedback? (freebsd)


Attachments
sysutils_fusefs_ntfs_Makefile.diff (668 bytes, patch)
2016-02-06 15:07 UTC, Zane C. Bowers-Hadley
no flags Details | Diff
terminal output (3.01 KB, text/plain)
2021-07-10 17:40 UTC, Graham Perrin
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Zane C. Bowers-Hadley 2016-02-06 15:07:36 UTC
Created attachment 166662 [details]
sysutils_fusefs_ntfs_Makefile.diff

This disables UBLIO by default as it breaks mkntfs. It also notes it in UBLIO_DESC at the end as breaking mkntfs.

UBLIO hangs for some reason when mkntfs is tries to create the backup boot block.
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2016-02-06 15:17:53 UTC
I have a work in progress update for this port, which removes UBLIO support given the number of issues that have been reported for it.

Would you be able to test it if I attach a patch? Preferably with a backed up or test NTFS volume/dataset
Comment 2 Zane C. Bowers-Hadley 2016-02-06 16:40:30 UTC
(In reply to Kubilay Kocak from comment #1)

Most definitely. Would be most happy to help. :)
Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2016-02-06 17:42:17 UTC
Thank you Zane, I'll have something attached here soon :)
Comment 4 VK freebsd_triage 2016-05-22 23:28:50 UTC
*** Bug 198566 has been marked as a duplicate of this bug. ***
Comment 5 Jason Unovitch freebsd_committer freebsd_triage 2016-06-05 22:54:13 UTC
Add new maintainer to CC.
Comment 6 Walter Schwarzenfeld freebsd_triage 2018-01-13 20:10:09 UTC
fusefs-nf has now 2017.3.23. I think this is overcome by events and could be closed.
Comment 7 Samy Mahmoudi 2018-08-29 12:18:54 UTC
Is this really still in progress ?
Comment 8 Domagoj Smolčić 2018-10-30 16:58:49 UTC
11.2-RELEASE-p4 i386 with fusefs-ntfs-2017.3.23 & libublio-20070103_2 (deinstalled BOTH and recompiled JUST NOW)
Compiling sysutils/fusefs-ntfs AUTO installs devel/libublio (no option)

According to test and fixes(?) below, there are TWO possible reasons for failure:
    1) Too big partition/slice size
    2) Partition/slice's length spreading to the end of device


Failure always occurs by hanging at 'Creating backup boot sector.', so that is a place to look at in the source code.


FAILS:
======
# gpart show -p ada0
=>       40  312581728    ada0  GPT  (149G)
         40  312581728  ada0p1  ms-basic-data  (149G)

# mkntfs -vIf -p 40 -H 16 -S 63 -L 'NTFS_slice' /dev/ada0p1
Cluster size has been automatically set to 4096 bytes.
Creating NTFS volume structures.
Creating root directory (mft record 5)
Creating $MFT (mft record 0)
Creating $MFTMirr (mft record 1)
Creating $LogFile (mft record 2)
Creating $AttrDef (mft record 4)
Creating $Bitmap (mft record 6)
Creating $Boot (mft record 7)
Creating backup boot sector.


FIX:  ==>  Not actually. ;)
====
# gpart delete -i 1 ada0
# gpart add -s 20g -t ms-basic-data ada0
# gpart show -p ada0
=>       40  312581728    ada0  GPT  (149G)
         40   41943040  ada0p1  ms-basic-data  (20G)
   41943080  270638688          - free -  (129G)

# mkntfs -vIf -p 40 -H 16 -S 63 -L 'NTFS_slice' /dev/ada0p1
Cluster size has been automatically set to 4096 bytes.
Creating NTFS volume structures.
Creating root directory (mft record 5)
Creating $MFT (mft record 0)
Creating $MFTMirr (mft record 1)
Creating $LogFile (mft record 2)
Creating $AttrDef (mft record 4)
Creating $Bitmap (mft record 6)
Creating $Boot (mft record 7)
Creating backup boot sector.
Creating $Volume (mft record 3)
Creating $BadClus (mft record 8)
Creating $Secure (mft record 9)
Creating $UpCase (mft record 0xa)
Creating $Extend (mft record 11)
Creating system file (mft record 0xc)
Creating system file (mft record 0xd)
Creating system file (mft record 0xe)
Creating system file (mft record 0xf)
Creating $Quota (mft record 24)
Creating $ObjId (mft record 25)
Creating $Reparse (mft record 26)
Syncing root directory index record.
Syncing $Bitmap.
Syncing $MFT.
Updating $MFTMirr.
Syncing device.
mkntfs completed successfully. Have a nice day.


Make it FAIL/HANG again:
==
# gpart add -t ms-basic-data ada0
# gpart show -p ada0
=>       40  312581728    ada0  GPT  (149G)
         40   41943040  ada0p1  ms-basic-data  (20G)
   41943080  270638688  ada0p2  ms-basic-data  (129G)

# mkntfs -vIf -p 41943080 -H 16 -S 63 -L 'NTFS_slice_2' /dev/ada0p2
Cluster size has been automatically set to 4096 bytes.
Creating NTFS volume structures.
Creating root directory (mft record 5)
Creating $MFT (mft record 0)
Creating $MFTMirr (mft record 1)
Creating $LogFile (mft record 2)
Creating $AttrDef (mft record 4)
Creating $Bitmap (mft record 6)
Creating $Boot (mft record 7)
Creating backup boot sector.


FIX again:  ==>  Not actually. ;)
====
# gpart delete -i 2 ada0
# gpart add -s 120g -t ms-basic-data ada0
# gpart show -p ada0
=>       40  312581728    ada0  GPT  (149G)
         40   41943040  ada0p1  ms-basic-data  (20G)
   41943080  251658240  ada0p2  ms-basic-data  (120G)
  293601320   18980448          - free -  (9.0G)

# mkntfs -vIf -p 41943080 -H 16 -S 63 -L 'NTFS_slice_2' /dev/ada0p2
Cluster size has been automatically set to 4096 bytes.
Creating NTFS volume structures.
Creating root directory (mft record 5)
Creating $MFT (mft record 0)
Creating $MFTMirr (mft record 1)
Creating $LogFile (mft record 2)
Creating $AttrDef (mft record 4)
Creating $Bitmap (mft record 6)
Creating $Boot (mft record 7)
Creating backup boot sector.
Creating $Volume (mft record 3)
Creating $BadClus (mft record 8)
Creating $Secure (mft record 9)
Creating $UpCase (mft record 0xa)
Creating $Extend (mft record 11)
Creating system file (mft record 0xc)
Creating system file (mft record 0xd)
Creating system file (mft record 0xe)
Creating system file (mft record 0xf)
Creating $Quota (mft record 24)
Creating $ObjId (mft record 25)
Creating $Reparse (mft record 26)
Syncing root directory index record.
Syncing $Bitmap.
Syncing $MFT.
Updating $MFTMirr.
Syncing device.
mkntfs completed successfully. Have a nice day.


Domagoj Smolčić
Comment 9 Domagoj Smolčić 2019-04-29 10:49:37 UTC
11.2-RELEASE-p9 i386 with fusefs-ntfs-2017.3.23 and WITHOUT libublio (at a time of a previous post, there was no option to choose)

Without libublio, ALL issues from my previous post are gone! ;)
Tested on exactly the same HW with same CMDs.
Reasons of failure:
    devel/libublio


libublio doesn't ONLY breaks mkntfs, but it also makes I/O operations on mounted ntfs unreliable.

Regarding See Also: Bug 194526 (sysutils/fusefs-ntfs: ntfs-3g with libublio lost files)
I.e; During copy of >100 GB files from RO mounted ntfs:
----
cp: /mnt/ntfs/file.r05: No such file or directory
Second copy attempt of above file succeeded.

In worse case scenario, some files completed without emitted errors, but i.e; file of 4G ended up being copied as 1.3G file
So I had to use diff tool for verification and it too would sometimes had an issues, like:
diff: /mnt/ntfs/file.mp4: Operation timed out
Second or higher attempts would catch silent errors: $ diff -rq ... ...
----
Only in .:
Files ... and ... differ
.
.
.
----
So I would have to manually ..., ok you get the point.


Based on above facts, this Bug 206978 can be closed, as well as Bug 194526 and all other submitted bugs relating to libublio if existing.
Then new bug dedicated to devel/libublio should be opened and resolved.

Regarding it, all points to it's obsolescence:
Port:
--
Link in pkg-descr:WWW: is invalid
Port has been last touched at 2017-01-18

It:
--
Library of only 17K in size used by BSD and derivatives including Mac OS X.
Exists only because BSD doesn't have a block, but character devices.
Regarding it's code it is all from *BSD, so no linux "translations" required.
It uses BSD style data structure macros via queue.h(taken from OpenBSD as is) and tree.h(taken from FreeBSD with some mods)
Sugar on top is it's size of only 17K, so no one is going to get lost in code.

Now best/shocking part is it's OLDNESS:
libublio has been created at 2006 and file has in it name 2007
Used queue.h,v 1.31 2005/11/25 (OpenBSD)
Used tree.h,v 1.5 2006/01/19 (FreeBSD)

To sum it up, I think devel/libublio doesn't need fixing/patching but RECREATION/FORK with current port as a good template.

PS/HINT: Ports using libublio:
--
sysutils/fusefs-ntfs
sysutils/fusefs-hfsfuse
sysutils/fusefs-exfat
sysutils/exfat-utils
--
Comment 10 Kubilay Kocak freebsd_committer freebsd_triage 2019-08-09 04:18:21 UTC
I'm going over my WIP for UBLIO option removal again for QA, but will be leaving the ublio bits in in case anyone wants to easily have a play with bringing back UBLIO to a maintained stable state

The WIP also includes additions for UUID and DECRYPT
Comment 11 Kubilay Kocak freebsd_committer freebsd_triage 2019-08-09 04:20:54 UTC
After the change lands, it would be great to:

1) Get wider testing of this port with the recent updates in HEAD (13.0-CURRENT) to fuse [1] thanks to Alan Somers

2) Investigate block device caching possibilities in base, if its still relevent/valuable given (1).

[1] https://lists.freebsd.org/pipermail/freebsd-current/2019-August/073980.html
Comment 12 Alexey Dokuchaev freebsd_committer freebsd_triage 2019-11-28 10:07:16 UTC
For quite a while (several years), I've been occasionally getting weird problems with writing to NTFS partitions: rsync(1) would abort file copy with infamous "Input/output error (5)" message.  I think it occurs more often when partition becomes closer to full.  Today I've hit it again and tried to google some solution.

At first, I've found this rather old Debian bug[*] which suggested that downgrading to version 2012.1.15 might "fix" it.  I've tried that but the problem did not go away.

Then I've searched our Bugzilla and stumbled upon this PR, which suggested to disable UBLIO, albeit for another reason.  Surprisingly, it helped!  I was able to fill the NTFS partition until "No space left on device (28)" without getting "Input/output error (5)".

This is still on 2012.1.15, I didn't try the latest version after the old version worked, but I suspect the problem is indeed with the UBLIO and thus it should be disabled by default (at least) or perhaps even annotated as BROKEN.

[*] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=774330
Comment 13 Jason W. Bacon freebsd_committer freebsd_triage 2019-12-18 23:03:18 UTC
I'm seeing the performance issue also.

Oddly, mkntfs works fine on /dev/da1 (and I can mount with ntfs-g3), but hangs on /dev/da1s1.
Comment 14 Darin Luckie 2020-04-26 02:53:59 UTC
Is this related?

root@ghost:~ # mkntfs -fC -L WINBLOWS1909 /dev/da0p1
The partition start sector was not specified for /dev/da0p1 and it could not be obtained automatically.  It has been set to 0.
The number of sectors per track was not specified for /dev/da0p1 and it could not be obtained automatically.  It has been set to 0.
The number of heads was not specified for /dev/da0p1 and it could not be obtained automatically.  It has been set to 0.
Cluster size has been automatically set to 4096 bytes.
To boot from a device, Windows needs the 'partition start sector', the 'sectors per track' and the 'number of heads' to be set.
Windows will not be able to boot from this device.
Creating NTFS volume structures.
Comment 15 Bertrand Petit 2020-05-16 01:34:49 UTC
I confirm that libublio is a culprit. I attempted to make a ntfs on a 916 GiB partition using mkntfs and interrupted the process after *four hours* of activity. I eventually found the current PR, rebuilt fuse-ntfs without the UBLIO option, and eventually managed to format that partition in a blink. This option definitely needs to be disabled, or libublio be fixed, or libublio be replaced with an automatically created gcache instance.
Comment 16 Graham Perrin freebsd_committer freebsd_triage 2021-07-10 17:40:01 UTC
Created attachment 226356 [details]
terminal output

I found this bug after encountering repeated difficulty at <https://forums.FreeBSD.org/threads/81231/post-521947>. 

There's a screenshot <https://forums.freebsd.org/attachments/1625937980425-png.10549/> of: 

gstat -p

– with apparently endless high read activity for the targeted device.
Comment 17 Rene Ladan freebsd_committer freebsd_triage 2022-03-07 19:55:01 UTC
Maintainer reset.
Comment 18 rajhlinux 2023-08-16 21:21:22 UTC
I also get this error where it hangs at: "Creating backup boot sector."
Using FreeBSD 13.1

# mkntfs --verbose --fast --label 2TB_HD /dev/ada1p1

The partition start sector was not specified for /dev/ada1p1 and it could not be obtained automatically.  It has been set to 0.
The number of sectors per track was not specified for /dev/ada1p1 and it could not be obtained automatically.  It has been set to 0.
The number of heads was not specified for /dev/ada1p1 and it could not be obtained automatically.  It has been set to 0.
Cluster size has been automatically set to 4096 bytes.
To boot from a device, Windows needs the 'partition start sector', the 'sectors per track' and the 'number of heads' to be set.
Windows will not be able to boot from this device.
Creating NTFS volume structures.
Creating root directory (mft record 5)
Creating $MFT (mft record 0)
Creating $MFTMirr (mft record 1)
Creating $LogFile (mft record 2)
Creating $AttrDef (mft record 4)
Creating $Bitmap (mft record 6)
Creating $Boot (mft record 7)
Creating backup boot sector.
Comment 19 rajhlinux 2023-08-16 21:39:22 UTC
Solution provided by "Domagoj Smolčić" works for me by explicitly specifying the "-s" parameter of the sector size on "gpart", then perform the "mkntfs" operation:


Here is what worked for me now:


View Partitions/Drive:

# gpart show


Destroy Partition/Drive:

# gpart destroy -F ada1


Create a GUID Partition Scheme:

# gpart create -s GPT ada1


Create a partition type (ntfs) and add the size (bytes, using “g” for GB):

# gpart add -t ms-basic-data -s 1800g ada1
  

The drive needs a NTFS Filesystem and perform a fast format:

# mkntfs --verbose --fast --label 2TB_HD /dev/ada1p1


Mount the Drive:

# ntfs-3g /dev/ada1p1 /mnt/2_TB_HD
Comment 20 Mark Linimon freebsd_committer freebsd_triage 2024-01-21 15:17:46 UTC
^Triage: notify current maintainer.

It looks as though the "In Progress" state became incorrect when the previous maintainership was reset.