Comment 1 Ed Maste 2025-05-09 13:50:59 UTC

This is a different issue. Maybe it already existed, but was not observed if pycdlib already exited due to the other issue. Which specific image are you testing?

Comment 2 Ed Maste 2025-05-09 14:09:58 UTC

I was able to open FreeBSD-14.3-BETA1-amd64-bootonly.iso with pycdlib and walk the contents without any errors reported.

Comment 3 Ed Maste 2025-05-09 17:53:10 UTC

OK, was able to reproduce with FreeBSD-14.3-BETA1-amd64-dvd1.iso

>>> iso.open("/home/emaste/images/FreeBSD-14.3-BETA1-amd64-dvd1.iso")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/emaste/src/pycdlib/pycdlib/pycdlib.py", line 4066, in open
    self._open_fp(fp)
  File "/home/emaste/src/pycdlib/pycdlib/pycdlib.py", line 2235, in _open_fp
    ic_level, lastbyte = self._walk_directories(self.pvd, extent_to_ptr,
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/emaste/src/pycdlib/pycdlib/pycdlib.py", line 1051, in _walk_directories
    rr = new_record.parse(vd, data[offset:offset + lenbyte],
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/emaste/src/pycdlib/pycdlib/dr.py", line 282, in parse
    if xa_rec.parse(record[record_offset:], self.len_fi):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/emaste/src/pycdlib/pycdlib/dr.py", line 86, in parse
    raise pycdlibexception.PyCdlibInvalidISO('Unused fields should be 0')
pycdlib.pycdlibexception.PyCdlibInvalidISO: Unused fields should be 0

It would be nice if pycdlib produced more detailed error messages, e.g. "Unused fields should be 0 in XA record at file offset 123456".

Comment 4 Ed Maste 2025-05-09 19:20:32 UTC

It looks like this is a pycdlib bug.

Here is a hexdump of the ISO content from the offending extent
$ dd if=FreeBSD-14.3-BETA1-amd64-dvd1.iso bs=2k iseek=1756652 | hexdump -Cv
...
000b7800  a8 00 56 d4 1a 00 00 1a  d4 56 08 0a 00 00 00 00  |..V......V......|
000b7810  0a 08 7d 05 02 0c 2f 34  00 00 00 00 01 00 00 01  |..}.../4........|
000b7820  1e 53 53 4c 5f 53 45 53  53 49 4f 4e 5f 47 45 54  |.SSL_SESSION_GET|
000b7830  30 5f 54 49 43 4b 45 54  2e 33 5f 47 5a 3b 31 00  |0_TICKET.3_GZ;1.|
000b7840  50 58 2c 01 24 81 00 00  00 00 81 24 03 00 00 00  |PX,.$......$....|
000b7850  00 00 00 03 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000b7860  00 00 00 00 58 41 00 00  00 00 41 58 54 46 1a 01  |....XA....AXTF..|
000b7870  0e 7d 05 02 0c 2f 34 00  7d 05 02 0c 2f 34 00 7d  |.}.../4.}.../4.}|
000b7880  05 02 0c 30 08 00 4e 4d  21 01 00 53 53 4c 5f 53  |...0..NM!..SSL_S|
000b7890  45 53 53 49 4f 4e 5f 67  65 74 30 5f 74 69 63 6b  |ESSION_get0_tick|
000b78a0  65 74 2e 33 2e 67 7a 00  b2 00 9b d3 1a 00 00 1a  |et.3.gz.........|
000b78b0  d3 9b a0 10 00 00 00 00  10 a0 7d 05 02 0c 2f 33  |..........}.../3|
...

pycdlib's XARecord::parse does:

        # In a "typical" XA record, the record immediately follows the DR
        # record (but comes before the Rock Ridge record, if this is a Rock
        # Ridge ISO).  However, we have seen ISOs (Windows 98 SE) that put some
        # padding between the end of the DR record and the XA record.  As far
        # as I can tell, that padding is the size of the file identifier,
        # but rounded up to the nearest even number.  We check both places for
        # the XA record.
        for offset in (0, even_size):
            parse_str = xastr[offset:]
            if len(parse_str) < struct.calcsize(self.FMT):
                return False

            (self._group_id, self._user_id, self._attributes, signature,
             self._filenum, unused) = struct.unpack_from(self.FMT, parse_str, 0)
            if signature != b'XA':
                continue

            if unused != b'\x00\x00\x00\x00\x00':
                print("offset=" + str(offset))
                print(" ".join(f"{byte:02x}" for byte in unused))
                raise pycdlibexception.PyCdlibInvalidISO('Unused fields should be 0')

            self._pad_size = offset
            break

The "PX" at 000b7840 is the beginning of the Rock Ridge entry (PX is the POSIX attributes record). XARecord::parse is called before parsing the RR entry, and happens to find the two bytes "XA" in the RR entry (in this case, the inode/serial number 0x4158).

Comment 5 Ed Maste 2025-05-09 19:40:33 UTC

I have opened a bug report for pycdlib at https://github.com/clalancette/pycdlib/issues/136

You could apply a change like this to pycdlib to work around the issue:

diff --git a/pycdlib/dr.py b/pycdlib/dr.py
index 1bc4236..42d6d75 100644
--- a/pycdlib/dr.py
+++ b/pycdlib/dr.py
@@ -72,7 +72,7 @@ class XARecord:
         # as I can tell, that padding is the size of the file identifier,
         # but rounded up to the nearest even number.  We check both places for
         # the XA record.
-        for offset in (0, even_size):
+        for offset in [0]:
             parse_str = xastr[offset:]
             if len(parse_str) < struct.calcsize(self.FMT):
                 return False

(In reply to Ed Maste from comment #1)
We are testing freebsd-14.3-BETA1.

Comment 7 Ed Maste 2025-05-14 12:32:30 UTC

> We are testing freebsd-14.3-BETA1.

But which specific ISO image?

I tested FreeBSD-14.3-BETA1-amd64-bootonly.iso and could not reproduce an issue. I tested FreeBSD-14.3-BETA1-amd64-dvd1.iso and reproduced an issue, which is a bug in pycdlib.

(In reply to Ed Maste from comment #7)
Hi Ed,

We tested FreeBSD-14.3-BETA1-amd64-dvd1.iso.

Comment 9 Ed Maste 2025-05-15 23:25:20 UTC

(In reply to Yanhui He from comment #8)
Ok, thanks - so that is the same one I was able to reproduce an issue with. This isn't a bug in our ISO image and needs to be fixed in pycdlib. This issue may or may not disappear with new image builds (e.g. BETA2), just depending on arbitrary filesystem ordering. You might be able to use disc1 instead of dvd1, if you don't need any of the extra content that's only on dvd1.