Bug 220870 - sysutils/py-diffoscope: Fix test suite failures
Summary: sysutils/py-diffoscope: Fix test suite failures
Status: In Progress
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Guangyuan Yang
URL:
Keywords:
Depends on: 220868
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-20 00:35 UTC by Ed Maste
Modified: 2019-01-27 16:42 UTC (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ed Maste freebsd_committer 2017-07-20 00:35:46 UTC
Testing v84 on my stable/10 desktop. make test reports in the final summary:
14 failed, 257 passed, 85 skipped, 1 warnings in 118.02 seconds

tests/test_presenters.py:126: AssertionError
tests/comparators/test_device.py:50: AssertionError
tests/comparators/test_device.py:55: AssertionError
tests/comparators/test_elf.py:54: AssertionError
tests/comparators/test_elf.py:59: AssertionError
tests/comparators/test_elf.py:88: AssertionError
tests/comparators/test_elf.py:102: AssertionError
tests/comparators/test_iso9660.py:47: IndexError
tests/comparators/test_iso9660.py:52: IndexError
tests/comparators/test_iso9660.py:56: IndexError
tests/comparators/test_iso9660.py:62: IndexError
tests/comparators/test_iso9660.py:72: IndexError
tests/comparators/test_pcap.py:49: AssertionError
tests/comparators/test_rlib.py:69: IndexError

This is really an upstream issue, but submitting this PR for tracking and reference as we import newer diffoscope versions (while still working to incorporate fixes upstream).
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2017-07-20 00:58:57 UTC
Are we creating/tracking issues upstream as well or doing it here instead due to upstream not being easy to submit issues for?
Comment 2 Ed Maste freebsd_committer 2017-07-20 02:55:55 UTC
> Are we creating/tracking issues upstream as well or doing it here instead
> due to upstream not being easy to submit issues for?

Charlie or I will submit upstream bugs if appropriate as we triage the individual issues, although most likely we'll just be submitting patches. For example, the first issue we've triaged is due to an unfixed bug in our cdrtools port/package, described in PR220868.
Comment 3 Guangyuan Yang freebsd_committer 2017-07-20 14:57:18 UTC
Use sysutils/cdrtools-devel instead of the default sysutil/cdrtools can fix the following cases:
tests/comparators/test_iso9660.py::test_symlink PASSED                                                                                                                                   
tests/comparators/test_iso9660.py::test_compressed_files PASSED                                                                                                                          
tests/comparators/test_iso9660.py::test_compare_non_existing PASSED

And can receive more info on these cases:
tests/comparators/test_iso9660.py::test_iso9660_content FAILED                                                                                                                           
tests/comparators/test_iso9660.py::test_iso9660_rockridge FAILED                                                                                                                         

See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220868 for more info.
Comment 4 Ed Maste freebsd_committer 2017-07-20 17:44:49 UTC
koobs, do you have any advice on handling cdrtools vs. cdrtools-devel? we'd rather have the latter installed when using diffoscope.
Comment 5 Guangyuan Yang freebsd_committer 2017-07-21 18:31:27 UTC
The following test failures have been fixed:

tests/comparators/test_device.py:50
tests/comparators/test_device.py:55

Related commit: https://anonscm.debian.org/git/reproducible/diffoscope.git/commit/?id=1390b76
Comment 6 Kubilay Kocak freebsd_committer freebsd_triage 2017-07-23 08:14:36 UTC
(In reply to Ed Maste from comment #4)

Preference for using released versions of any dependent software (with backports of fixes if necessary) and encouraging upstream to release new bugfix versions, over -devel versions unless unavoidable.
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2017-07-23 08:16:00 UTC
CC cdrtools/cdrtools-devel maintainer (marius)
Comment 8 Kubilay Kocak freebsd_committer freebsd_triage 2017-07-23 08:19:15 UTC
(In reply to Kubilay Kocak from comment #6)

I note that the debian diffoscope package doesn't depend on cdrtools )removed from repository a while ago). Is there something else we could/should be using or is there more to this story (say for example: utility compatibility with freebsd) ?
Comment 9 Guangyuan Yang freebsd_committer 2017-07-23 19:06:40 UTC
(In reply to Kubilay Kocak from comment #8)

From my knowledge, on Debian they use genisoimage. Need to confirm its compatibility on FreeBSD.
Comment 10 Guangyuan Yang freebsd_committer 2017-07-24 16:19:10 UTC
I did a bit more investigation to find out that other than the problems mentioned above, the rest failed cases are all caused by the different handlings between bsddiff and gnudiff (using gdiff instead will fix all the rest cases).
Comment 11 Ed Maste freebsd_committer 2017-07-24 16:47:17 UTC
(In reply to Guangyuan (Charlie) Yang from comment #10)
bapt@, what do you think we should do about this wrt BSD diff? We have a few possible paths for this specific port:

- work on making BSD diff make the same choices about diff hunk selection as GNU diff
- have diffoscope check for and prefer gdiff if it exists
- have diffoscope require GNU diffutils be installed and use it unconditionally on FreeBSD
Comment 12 Kubilay Kocak freebsd_committer freebsd_triage 2017-07-25 03:13:45 UTC
(In reply to Ed Maste from comment #11)

(3) first, with (1) (gnudiff compat) as a longer term target
Comment 13 Guangyuan Yang freebsd_committer 2017-07-25 20:23:49 UTC
I will add code to upstream diffoscope to force using gdiff (GNU Diff) tool on FreeBSD then. Dependency list of sysutils/py-diffoscope should also be updated along with the upstream diffoscope update.

Since PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220868 is Closed/FIXED, I don't see any other failed cases on my machine except for the ones described here: https://github.com/yzgyyang/diffoscope/issues/2.
Comment 14 Kubilay Kocak freebsd_committer freebsd_triage 2017-07-26 02:27:34 UTC
(In reply to Guangyuan (Charlie) Yang from comment #13)

The ability to set the diff (and other) tools via some setting, environment variable or similar would be more desirable and make for a more portable/usable diffoscope. Then the consumer (in this case, the port) can use/switch/test utilities without needing to patch code or create more upstream work
Comment 15 Ed Maste freebsd_committer 2017-07-28 21:17:00 UTC
(In reply to Kubilay Kocak from comment #14)
I think that might be more effort than it's worth. diffoscope was written to expect GNU diff, so we ought to just use GNU diff if possible.
Comment 16 Guangyuan Yang freebsd_committer 2017-08-14 20:42:32 UTC
A quick update regarding this issue:

Tests should all pass except for the ones that expect GNU diff results instead of BSD diff. A bug report with patch is submitted and is awaiting upstream's response. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871029

> The ability to set the diff (and other) tools via some setting, environment variable or similar would be more desirable and make for a more portable/usable diffoscope.

Had discussions with the developers of diffoscope, they agreed that an external config dictionary or similar should be added. More investigations are to be done, and no plan is available for now. See a similar issue here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=869868
Comment 17 Guangyuan Yang freebsd_committer 2017-08-17 16:16:43 UTC
Multiple commits have been made on upstream to "prefer tools that start with 'g' on a non-GNU system". So all the failed cases should be gone (at least on my computer, can't say about skipped tests though) upon the ports update of the next diffoscope release.

One last thing: add textproc/diffutils to the dependency list.
Comment 18 Ed Maste freebsd_committer 2017-08-17 18:24:33 UTC
Thanks Charlie - I assume that now we just need to wait for the next diffoscope release and then update the port.
Comment 19 commit-hook freebsd_committer 2017-09-27 21:19:18 UTC
A commit references this bug:

Author: ygy
Date: Wed Sep 27 21:19:00 UTC 2017
New revision: 450791
URL: https://svnweb.freebsd.org/changeset/ports/450791

Log:
  sysutils/py-diffoscope: Update to 86

  PR:	220870
  Approved by:	koobs (maintainer)
  Differential Revision:	https://reviews.freebsd.org/D12328

Changes:
  head/sysutils/py-diffoscope/Makefile
  head/sysutils/py-diffoscope/distinfo
Comment 20 Guangyuan Yang freebsd_committer 2018-01-29 02:47:11 UTC
Reported and fixed readelf version parsing errors in the upstream Version 90.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=886963

As the codebase iterates, I may close this one and open another PR for latest errors and fixes.
Comment 21 Ed Maste freebsd_committer 2018-10-03 17:34:03 UTC
On my laptop I currently see two test failures:

% pytest-3.6
============================= test session starts ==============================
platform freebsd12 -- Python 3.6.5, pytest-3.6.4, py-1.6.0, pluggy-0.6.0
rootdir: /usr/home/emaste/src/diffoscope, inifile:
collected 382 items / 2 errors                                                 

==================================== ERRORS ====================================
________________ ERROR collecting tests/comparators/test_elf.py ________________
tests/comparators/test_elf.py:77: in <module>
    @skip_if_binutils_does_not_support_x86()
tests/utils/tools.py:101: in skip_if_binutils_does_not_support_x86
    'elf64-x86-64' not in get_supported_elf_formats(),
tests/utils/tools.py:109: in get_supported_elf_formats
    ('objdump', '--info'),
/usr/local/lib/python3.6/subprocess.py:336: in check_output
    **kwargs).stdout
/usr/local/lib/python3.6/subprocess.py:418: in run
    output=stdout, stderr=stderr)
E   subprocess.CalledProcessError: Command '('objdump', '--info')' returned non-zero exit status 1.
------------------------------- Captured stderr --------------------------------
objdump: Unknown command line argument '--info'.  Try: 'objdump -help'
objdump: Did you mean '-df'?
_______________ ERROR collecting tests/comparators/test_rlib.py ________________
tests/comparators/test_rlib.py:78: in <module>
    @skip_if_binutils_does_not_support_x86()
tests/utils/tools.py:101: in skip_if_binutils_does_not_support_x86
    'elf64-x86-64' not in get_supported_elf_formats(),
tests/utils/tools.py:109: in get_supported_elf_formats
    ('objdump', '--info'),
/usr/local/lib/python3.6/subprocess.py:336: in check_output
    **kwargs).stdout
/usr/local/lib/python3.6/subprocess.py:418: in run
    output=stdout, stderr=stderr)
E   subprocess.CalledProcessError: Command '('objdump', '--info')' returned non-zero exit status 1.
------------------------------- Captured stderr --------------------------------
objdump: Unknown command line argument '--info'.  Try: 'objdump -help'
objdump: Did you mean '-df'?
!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!
=========================== 2 error in 3.08 seconds ============================
Comment 22 Ed Maste freebsd_committer 2018-10-03 17:54:54 UTC
Oops, tests are not being run when those errors are encountered.

PATH=/usr/local/bin:$PATH /usr/bin/time pytest reports:

============= 8 failed, 272 passed, 120 skipped in 482.29 seconds ==============
      484.13 real       218.76 user       654.70 sys

tests/test_main.py:161: AssertionError
tests/comparators/test_berkeley_db.py:33: AssertionError
tests/comparators/test_cpio.py:50: AssertionError
tests/comparators/test_device.py:59: AssertionError
tests/comparators/test_device.py:68: AssertionError
tests/comparators/test_ppu.py:50: AssertionError
tests/comparators/test_ppu.py:66: IndexError
tests/comparators/test_ppu.py:71:
tests/utils/nonexisting.py:32: AssertionError
Comment 23 Ed Maste freebsd_committer 2019-01-27 16:42:31 UTC
Tried building in Cirrus-CI,
https://cirrus-ci.com/task/5110258826477568

============= 9 failed, 254 passed, 153 skipped in 188.06 seconds ==============
      199.92 real        99.76 user       166.00 sys