Bug 164896 - [exp-run] add support for testing perl modules
Summary: [exp-run] add support for testing perl modules
Status: In Progress
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: Any Any
: Normal Affects Only Me
Assignee: Bryan Drewery
URL:
Keywords: patch
: 198494 (view as bug list)
Depends on:
Blocks:
 
Reported: 2012-02-08 04:10 UTC by Steve Wills
Modified: 2018-12-05 12:01 UTC (History)
3 users (show)

See Also:


Attachments
file.diff (1.31 KB, patch)
2012-02-08 04:10 UTC, Steve Wills
no flags Details | Diff
test_framework.diff (10.62 KB, patch)
2012-04-26 03:05 UTC, Steve Wills
no flags Details | Diff
test_framework.diff (10.62 KB, patch)
2012-04-26 04:24 UTC, Steve Wills
no flags Details | Diff
ports-test-framework.patch3.txt (10.50 KB, text/plain)
2012-05-22 04:45 UTC, Steve Wills
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Steve Wills freebsd_committer 2012-02-08 04:10:15 UTC
I'm working on testing perl modules. This patch is the first part of a more complete solution for testing perl modules. This will make it possible to run tests. The second part will need to add support for installing TEST_DEPENDS, either in a separate "stage" or as a normal target. Feedback on the approach would be appreciated. In the mean time, an exp-run for this patch is requested.

Fix: Patch attached with submission follows:
Comment 1 Philip M. Gollucci freebsd_committer 2012-02-08 20:13:04 UTC
Responsible Changed
From-To: freebsd-ports-bugs->portmgr
Comment 2 Mark Linimon freebsd_committer freebsd_triage 2012-03-24 07:54:37 UTC
State Changed
From-To: open->analyzed
Comment 3 Mark Linimon freebsd_committer freebsd_triage 2012-03-24 07:54:37 UTC
Responsible Changed
From-To: portmgr->linimon
Comment 4 Steve Wills freebsd_committer 2012-04-26 03:05:37 UTC
Here's an updated patch. I believe this may be much closer to ready.

Steve
Comment 5 Steve Wills freebsd_committer 2012-04-26 04:24:06 UTC
On 04/25/12 22:05, Steve Wills wrote:
> Here's an updated patch. I believe this may be much closer to ready.
> 
> Steve

Slight change: USE_PERL_TESTS -> WITH_PERL_TESTS

Steve
Comment 6 Steve Wills freebsd_committer 2012-05-22 04:45:21 UTC
Here's a further updated version of the patch. It has a few minor
changes such as not including tests if TEST_DEPENDS are defined.

Would still like to get feedback from someone who knows bsd.port.mk
well. My basic question is, am I better off with this sort of "bolted
on" approach to testing, or should I take a different approach and
integrate the TEST_DEPENDS so they are treated like other DEPENDS, for
example showing up in "make describe" and creating a
"pretty-print-test-depends-list" target.

Thanks,
Steve
Comment 7 Mark Linimon freebsd_committer freebsd_triage 2012-06-02 04:50:29 UTC
Responsible Changed
From-To: linimon->swills

This test was completed some time ago.  Sorry to have put off doing the 
analysis until now. 

The results are in: 

http://pointyhat-west.isc.freebsd.org/errorlogs/amd64-errorlogs/e.9.20120324081752.pointyhat-west/index-reason.html 

There were no failures with the ports infrastructure. 

All 13 package errors marked 'mtree' seem to be regressions introduced 
by the patch.  Please advise.
Comment 8 Steve Wills freebsd_committer 2012-06-11 02:57:24 UTC
On Jun 1, 2012, at 11:54 PM, linimon@FreeBSD.org wrote:

> Synopsis: [exp-run] add support for testing perl modules
> 
> Responsible-Changed-From-To: linimon->swills
> Responsible-Changed-By: linimon
> Responsible-Changed-When: Sat Jun 2 03:50:29 UTC 2012
> Responsible-Changed-Why: 
> This test was completed some time ago.  Sorry to have put off doing the
> analysis until now.
> 
> The results are in:
> 
>  http://pointyhat-west.isc.freebsd.org/errorlogs/amd64-errorlogs/e.9.20120324081752.pointyhat-west/index-reason.html
> 
> There were no failures with the ports infrastructure.
> 
> All 13 package errors marked 'mtree' seem to be regressions introduced
> by the patch.  Please advise.
> 
> http://www.freebsd.org/cgi/query-pr.cgi?pr=164896

The 13 package errors marked 'mtree' are due to the patch creating a
"regression-test" target that calls the "test" target, in the case that
the regression-test target doesn't exist. These ports have "test"
targets which create extra files, which creates the mtree issue. I've
seen that happen with many perl ports during my own testing. Since
pointyhat calls the "regression-test" target, the mtree issue is
triggered due to this patch where it wouldn't be before because the
regression-test call wasn't calling the test target.

We can solve these issues several ways. Perhaps the simplest way would
be to have the test patch changed so that the default test target is
only created if both the test and the regression-test targets do not
exist. Another would be to standardize on only "test" or
"regression-test". Or we might have pointyhat not call the test target.

Yet another way to solve the problem might be to have the ports changed
so that their tests don't leave leftover files. I'm guessing there are a
small number of ports which have regression-test targets which are
called by pointyhat and which had this "test creates mtree issue" issue
in the past which were fixed.

All but the simplest solution would require more discussion... I think
we need to standardize on the naming and decide where/when tests will be
run in any case, so perhaps now's the time....

Steve

(re-sending to get this into the PR)
Comment 9 Steve Wills freebsd_committer 2012-06-11 03:09:37 UTC
Responsible Changed
From-To: swills->portmgr

- Back to portmgr for discussion
Comment 10 Bryan Drewery freebsd_committer 2014-03-28 22:49:23 UTC
Responsible Changed
From-To: portmgr->bdrewery

I've been deep into bsd.port.mk deps lately, I will see this in. I will 
move bsd.test.mk into a USES as well. I agree with the "bolted on" 
approach as the general case right now is no tests"
Comment 11 Mathieu Arnold 2014-03-28 23:53:02 UTC
Do note that perl.mk has a regression-test target (and a test one), look at
the end :-)

-- 
Mathieu Arnold
Comment 12 Bryan Drewery freebsd_committer 2015-03-10 16:14:22 UTC
*** Bug 198494 has been marked as a duplicate of this bug. ***