Bug 148695 - [bsd.port.mk] [patch] Easier / automatic regression-test target for ports
Summary: [bsd.port.mk] [patch] Easier / automatic regression-test target for ports
Status: In Progress
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: Any Any
: Normal Affects Only Me
Assignee: Steve Wills
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-07-16 22:30 UTC by Christopher Key
Modified: 2014-11-09 22:25 UTC (History)
1 user (show)

See Also:


Attachments
ports-regression-test.patch (1.10 KB, patch)
2010-07-16 22:30 UTC, Christopher Key
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Christopher Key 2010-07-16 22:30:07 UTC
At the moment, many ported applications/libraries provide some form of regression testing, but very few ports actually expose this via the standard regression-test target.  The regression-test target will be called by tinderbox if available, and should also make debugging misbehaving ports easier.

Attached is a patch which aims to remedy this situation.

Firstly, it simplifies the creation of a regression-test target, with the addition of a REGRESSION_TEST_TARGET variable.  For audio/flac for example, this means that:
#v+
     regression-test: build
             @cd ${WRKSRC}; ${SETENV} ${MAKE_ENV} ${MAKE} check
#v-

can be replaced with:
#v+
     REGRESSION_TEST_TARGET= check
#v-

Secondly, rather more aggressively, it assumes that anything using GNU_CONFIGURE or PERL_CONFIGURE implements a check or test target respectively, and sets REGRESSION_TEST_TARGET accordingly.  As far as I am aware, these target are always present, although it is not guaranteed that any tests will actually be run.  Most perl modules do provide tests, but gnu software seems less likely to do so.  If no useful testing will be performed, the automatic creation of a regression-test target can be overridden by setting REGRESSION_TEST_TARGET to an empty string.

The attached patch probably isn't doing things in the correct places, or doing them in them in the correct way, but does work to illustrate the intent.
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2010-07-18 16:17:55 UTC
Responsible Changed
From-To: freebsd-ports-bugs->portmgr

bsd.port.mk patch.
Comment 2 Pav Lucistnik freebsd_committer 2010-08-12 15:49:19 UTC
State Changed
From-To: open->suspended

This cannot be added because many packages will fail the tests (false-positive). 
It needs to be enabled on port-by-port basis.
Comment 3 Erwin Lansing freebsd_committer 2012-05-12 02:45:58 UTC
Responsible Changed
From-To: portmgr->swills

Hand over to swills who is working on a testing framework and may 
pick some good ideas from this PR to incorporate.
Comment 4 Jan Beich freebsd_committer 2014-07-19 20:36:35 UTC
regression-test isn't documented, standardised[1] and only run by tinderbox but not poudriere. And this bug maybe obsolete by the addition of option helpers in bsd.options.mk. For example, to run devel/libffi tests the following is enough

  OPTIONS_DEFINE=	TEST

  TEST_BUILD_DEPENDS=runtest:${PORTSDIR}/misc/dejagnu
  TEST_ALL_TARGET=check

and if maintainer has high confidence[1] in passing tests also add

  OPTIONS_DEFAULT=	TEST

while user can globally disable via make.conf to shave build time

  OPTIONS_UNSET=	TEST

[1] examples being archivers/lzo2 and graphics/png each requiring a different way to disable running tests