Hi,

this is not a problem of the burn program or ISO 9660 formatter
but of the "cd9660" driver of FreeBSD. See for details:
  https://lists.freebsd.org/pipermail/freebsd-hackers/2012-April/038552.html

You will see the affected file as single file if you mount the medium
on GNU/Linux, or if you inspect it by xorriso:

  xorriso -indev /dev/cd0 -lsdl /test10g --

You may extract it to disk as single file on FreeBSD by:

  xorriso -osirrox on -indev /dev/cd0 -extract /test10g ./test10g

The fact that FreeBSD shows better results, if mkisofs option -udf is
used, is to blame on the better implementation of the UDF driver.
(growisofs forwards most of its options to mkisofs.)


Have a nice day :)

Thomas

Hi,

i now see that your file path in the ISO is probably /test/test10g.
So the proposed xorriso commands to inspect and extract it should also use
that ISO path, of course:

  xorriso -indev /dev/cd0 -lsdl /test/test10g --

  xorriso -osirrox on -indev /dev/cd0 -extract /test/test10g ./test10g


Have a nice day :)

Thomas

Hi,

(It's sunday. Thought trickles slowly ...)

Urm, do i get it right that not even in UDF the big file is presented
properly on FreeBSD ?

How did you mount it ?
If not by mount_udf(8): Do you get better results with mount_udf ?


> # growisofs -udf -iso-level 3 -Z /dev/cd0 -J -R --graft-points /test/=test10g

Except "-Z /dev/cd0" these options get forwarded to mkisofs (or genisoimage
or xorrisofs, depending on environment variable MKISOFS or GENISOIMAGE).

-iso-level 3 enables production of multiple extents per file. This is
essential for being able to store files larger than 4 GiB - 1 byte in
ISO 9660 and Joliet.
-R enables Rock Ridge info in the ISO 9660 filesystem tree.
-J enables production of an additional Joliet filesystem.
-udf enables production of an UDF filesystem.

The resulting UDF filesystem of the image should be mountable by
mount_udf(8).
ISO 9660 and Joliet are covered by mount_cd9660(8).


> P.S. I can write 4gb+ file by 'growisofs -udf -iso-level 3 -Z /dev/cd0i
> -J -R -allow-limited-size --graft-points /test/=test10g' on ubuntu.

-allow-limited-size limits the data files in ISO 9660 and Joliet to 2 GiB
(minus 1 byte ?). So only the UDF aspect of the image bears the full size.
The fact that you have to specify it, is a bug in genisoimage:
  File file_of_4gb+1m is larger than 4GiB-1.
  -allow-limited-size was not specified. There is no way do represent this file size. Aborting.
To be consistently buggy it then does not what its man page promises, but
rather rolls over the filesize by 4 GiB.
To see this, mount on a Linux kernel by
  mount -t iso9660 ...

To your luck (and the luck of genisoimage) the preference on Linux is
UDF over Rock Ridge over Joliet over plain ISO 9660.

I assume you have an original mkisofs on your FreeBSD machine which can
create a proper multi-extent data file in ISO 9660. At least it did
not complain about missing -allow-limited-size.
(If you can afford to give up option -udf, then xorrisofs would be a nice
 companion for growisofs. But i still assume that with FreeBSD as target
 system you are better off with working UDF and broken ISO 9660.)


Have a nice day :)

Thomas

(In reply to scdbackup from comment #3)

I found that the order of arguments is important.
Since growisofs assumes that the first two arguments are not arguments to mkisofs, it seems that -iso-level 3 did not pass to mkisofs. In linux, I seem to have saved by accident because I added -allow-limited-size to the beginning of the argument. So growisofs now works correctly if we specify arguments of mkisofs after the third argument with -Z or -M for the first argument and /dev/cd0 for the second argument.

i.e.
(1) 'growisofs -udf -iso-level 3 -Z /dev/cd0 -J -R --graft-points /test/=test10g' is not working.
(2) 'growisofs -Z /dev/cd0 -udf -iso-level 3 -J -R --graft-points /test/=test10g'
is working well, perhaps.

P.S. I will write a movie file of 35 gb on the BD-RW DL from now on, check the sha 512, confirm that it was written correctly and report the result. However, my BD drive is very slow, so reporting will take a while.

(In reply to Masachika ISHIZUKA from comment #4)

As I wrote in #1 as 'growisofs -udf -iso-level 3 -Z /dev/cd0 -J -R -allow-limited-size --graft-points /test/=test10g' on ubuntu, but order of arguments may not be correct.
Because I did not think the order of arguments is not important and I wrote by hand.

(In reply to scdbackup from comment #3)

Thank you very much.
It works well with mount_udf.

> I found that the order of arguments is important.
> Since growisofs assumes that the first two arguments are not arguments to
> mkisofs, it seems that -iso-level 3 did not pass to mkisofs.

The above was completely wrong.
The order of growisofs's arguments had no effect.

When I wrote #1, I was using mount_cd9660 and I thought that it could not be written properly. I think it could not read correctly on ubuntu, but it was my misunderstanding. I can read it correctly both by ubuntu and by mount_udf of freebsd.

Sorry to you.

Hi,

so we conclude that we found a valid use case for mkisofs option -udf
outside of the DVD video world.

mkisofs and xorriso would be able to create large files in ISO 9660.
genisoimage is not able to do so.
FreeBSD is not able to represent large files properly when the ISO is 
mounted.
So UDF by mkisofs or genisoimage is the workaround for FreeBSD which
is supposed to work elsewhere, too.

This workaround is valid only for single session ISOs. Adding a session
so that it works with UDF is not possible with mkisofs or genisoimage,
afaik.


Have a nice day :)

Thomas

(In reply to scdbackup from comment #7)

> This workaround is valid only for single session ISOs. Adding a session
> so that it works with UDF is not possible with mkisofs or genisoimage,
> afaik.

Thank you for important information.
I tried to append 1.3gb file to udf/iso9660 disc.
(1) on freebsd
# growisofs -udf -iso-level 3 -M /dev/cd0 -J -R --graft-points /test/=test1.3g
# mount -t udf /dev/cd0 /cdrom
# ls /cdrom/test
(shown only first session's files)
# umount /cdrom
# mount -t cd9660 /dev/cd0 /cdrom
# ls /cdrom/test
(all files are shown but 4gb+ files are shown multiple with incorrect size)

(2) on ubuntu
# growisofs -udf -iso-level 3 -M /dev/cd0 -J -R --graft-points /test/=test1.3g
(snip)
genisoimage: Error: 'test/test10g' and 'test/test10g' have the same Rock Ridge name 'test/test10g'
(snip)
Unable to sort directory test
:-( genisoimage has failed: 1
(abend)

Is it impossible to add files to the udf disc due to udf specification restrictions?

Hi,

> (1) on freebsd
> # mount -t udf /dev/cd0 /cdrom
> # ls /cdrom/test
> (shown only first session's files)

This is probably due to the fact that growisofs formats BD-R to be
Pseudo-Overwritable. Afterwards the medium shows only one track and
thus is not recognized by the operating system as multi-session medium.
Since the UDF "anchor" blocks were not updated, they still point to
the directory tree of the first session.

The formatting default can be suppressed by growisofs option
  -use-the-force-luke=spare=none
Formatting for Defect Management without Pseudo-Overwrite can be achieved
by a separate formatting run of cdrskin or xorriso before using growisofs
for burning. (I am unsure whether dvd+rw-format can be kept from adding
the Pseudo-Overwrite feature. So in this case my programs are safer.)

I am not aware whether Linux or FreeBSD try to apply the multi-session
convention if they mount the UDF aspect of a medium with multiple sessions.
It might be interesting to see how growisofs, mkisofs -udf, and mount_udf
behave with the second session on a BD-R that is not pseudo-overwritable.


> # mount -t cd9660 /dev/cd0 /cdrom
> # ls /cdrom/test
> (all files are shown but 4gb+ files are shown multiple with incorrect size)

The ISO-9660 Primary Volume Descriptor was updated by growisofs to
point to the directory tree of the add-on session. It refers to data
files in both sessions. Pity that the cd9660 driver cannot properly
represent the big file.

Background on ISO 9660 multi-session:
The convention for readers is to start at the superblock (Primary Volume
Descriptor) at block 16 of the first track of the last session on the
optical medium.
Since overwritable optical media only have one session and track, this
prescription does not work for them. But other than with multi-session
media, the old superblock can be overwritten to point to the directory
tree of the newly appended pseudo-session.
This way, growisofs and xorriso can do ISO 9660 multi-session on all
optical media types. (xorriso can do it on image files, too.)

Obviously, growisofs has no similar update provision for the UDF anchors.


> (2) on ubuntu
> genisoimage: Error: 'test/test10g' and 'test/test10g' have the same Rock Ridge name 'test/test10g'

Another known bug of genisoimage. Maybe you should install real mkisofs.
Either from Joerg Schillings cdrtools source tarball or from Brandon
Snider's cdrtools PPA.


> Is it impossible to add files to the udf disc due to udf specification
> restrictions?

UDF is complicated and has multiple variations ("builds").
Afaik, mkisofs produces UDF 1.x "Plain build". It is used for creating
video which Wikipedia mentions as UDF 1.02.

Although "plain" is characterized as "random read/write" i assume that
the architecture of mkisofs does not invite for modification of existing
blocks in the image. After all it shall be able to deliver its result
as stream to a pipe.

UDF has its own provisions for being modified. I guess this deferred
all attempts to implement multi-session in parallel with ISO 9660,
although this seems possible.
(I myself was completely deferred by reading ECMA-167 and UDF-2.60.
 Afterwards i had to clean my eyes with a tooth brush.)


Have a nice day :)

Thomas

(In reply to scdbackup from comment #9)

Thank you for telling me a lot of important information.
Since I can not understand immediately unless I experiment a lot, I will also try various things.

(In reply to Masachika ISHIZUKA from comment #10)

I tested 'mount -t udf /dev/cd0 /cdrom' without growisofs on FreeBSD, NetBSD and ubuntu Linux.

(1) UDF 1.50
It can read/write on NetBSD and ubuntu Linux.
It can read, but can not write on FreeBSD.

(2) UDF 2.01
It can read/write on NetBSD and ubuntu Linux.
It can not mount on FreeBSD.
I can not make mount_udf2 on head r338306, with the following errors.
| /usr/src/sys/modules/udf2/../../fs/udf2/udf_vnops.c:1160:16: error: no member named 'a_reqpage' in 'struct vop_getpages_args'
|         if (pages[ap->a_reqpage]->valid == VM_PAGE_BITS_ALL) {
|                   ~~  ^
| /usr/src/sys/modules/udf2/../../fs/udf2/udf_vnops.c:1162:17: error: no member named 'a_reqpage' in 'struct vop_getpages_args'
|                         if (i != ap->a_reqpage) {
|                                  ~~  ^
| /usr/src/sys/modules/udf2/../../fs/udf2/udf_vnops.c:1163:5: error: use of undeclared identifier 'PA_LOCK_COUNT'

(3) UDF 2.50
It can read/write on NetBSD.
It can read, but can not write on ubuntu Linux.
It can not mount on FreeBSD.

So if we use UDF 1.50, it can append dirs/files on NetBSD/ubuntu Linux.

On NetBSD, I use UDF 1.50 discs as follows.
netbsd# newfs_udf -v 1.50 /dev/rcd0 <=== Create UDF 1.50 udf filesystem on BD-R.
netbsd# mount -t udf /dev/cd0 /mnt <=== Mount BD-r for packet write/read.
netbsd# cp -pr dirs/files /mnt/  <=== We can copy dirs/files with normal cp command.

We can format UDF 1.50 with 'mkudffs -r 1.50 /dev/sr0' instead of newfs_udf on ubuntu Linux.

(In reply to Masachika ISHIZUKA from comment #11)

The above was tested with BD-RE, and it was totally useless if it was BD-R.
With BD-R, NetBSD 8.0 will freeze when you umount and become BD-R media that can not be used at all. On ubuntu Linux 18.04.1 it seems that BD-R can be mounted only as read only.

Hi,

so i can cancel my plans to explore Linux UDF capabilities with sequential
media. Your previous success report was not unplausible, because UDF indeed
defines usage models for such media. But i am not aware of an implemented
UDF driver in a free operating system which works according to that model,
which Wikipedia calls "VAT build".

https://people.freebsd.org/~scottl/udf/index.html
calls in its TODO for volunteers to implement VAT.

Customers for VAT would be the sequential media or media states: 
CD-R, unformatted CD-RW, DVD-R, DVD-R DL, DVD+R, DVD+R DL, 
unformatted DVD-RW, BD-R not formatted to Pseudo-Overwrite.
Traditionally one needs a burn program to write to them.

The other family is called "overwritable": formatted CD-RW, DVD-RAM, DVD+RW,
formatted DVD-RW, Pseudo-Overwrite formatted BD-R, BD-RE.
If the operating system recognizes it as overwritable then the medium is
usable like a slow hard disk or a slow USB stick. But it might not
last long until the medium dies under the random access load of a
hard-disk-oriented read-write filesystem. Burn programs are usually less
torturous to drive and medium, because they mainly work sequential.

(I also fail to find for FreeBSD something like Linux pktsetup(1) which
would be needed for using formatted DVD-RW like a hard disk.)

Have a nice day :)

Thomas

(In reply to scdbackup from comment #13)

I tried writing to BD-R DL on Windows10.
If we use Windows10, we can see that it can be created/appended to UDF 2.60 BD-R DL.
I didn't consider to use Windows10 at first because Windows10 home was unable to use the nfs client function and Windows10 pro's nfs client does not support UTF-8. But I found we can browse from the 'network locations' of explorer of Windows 10 home with FreeBSD's minidlna, i.e. it is ok without using nfs or samba. We can mount_udf the BD-R DL which was created/appended by this method on the ubuntu machine with read only mode. I was checked that this media was written properly by checking sha512.
Currently, when appending 4gb or more files to BD-R, I think this method is better.

Created attachment 197110 [details]
mount_udf2 for 10.4R

Created attachment 197111 [details]
mount_udf2 for current

I found that mount_udf2 from NetBSD can mount all revisions of udf on FreeBSD to read, but it can not mount to write.
I verified to be read DB-R DL udf 1.02, 1.50, 2.01, 2.50 and 2.60 on 10.4-RELEASE-p3 and 12.0-ALPHA6.

MARKED AS SPAM

MARKED AS SPAM