Bug 203640
| Summary: | devel/ice: fix build, switch to new test framework | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Ports & Packages | Reporter: | Dmitry Marakasov <amdmi3> | ||||
| Component: | Individual Port(s) | Assignee: | Dmitry Marakasov <amdmi3> | ||||
| Status: | Closed FIXED | ||||||
| Severity: | Affects Many People | CC: | freebsd, grembo | ||||
| Priority: | --- | ||||||
| Version: | Latest | ||||||
| Hardware: | Any | ||||||
| OS: | Any | ||||||
| Attachments: |
|
||||||
|
Description
Dmitry Marakasov
2015-10-08 14:46:55 UTC
I'm aware of the build failures. The build doesn't fail on every time, but only occasionally. As I failed to reproduce the problem *at all* (tried a multitude of setups, 9, 10, 11, vm, bare metal etc., ran the tests for days) and as I also didn't get any response from the nice people running the build hosts about setup details, that would allow me to reproduce the issues. I added some debug code to the unit tests in question, which allowed me to understand where it breaks, but that's not enough to analyse the issue. I was about the disable the unit tests, but that's something I'm reluctant to do. All unit tests are network based, so there might be a configuration/resource problem there. It might also point to a specific problem in Ice or in FreeBSD, as the problems happen in asynchronous tests (looks like some sort of deadlock). It's always the same two tests failing, so figuring out what's going on would be great (it seems timing sensitive). So you suggest to rename "TESTS" to "TEST" (diff), by removing the description, the requirement to python this introduced is hidden from the user, do you think this is good? Note that test is currently not run in the build process, but before staging. So if I switched to the new tests framework (do you have any information/links on it?), what would that essentially mean? Also, is there anything you can provide that would allow me to create a setup like the current build environment, so I can reproduce the issue and potentially fix it? > All unit tests are network based I'm not sure of cluster setup, but ports are generally not allowed to access any network after fetch phase. My first guess was that it was enforced on the cluster, thus the failures. However if the failures only happen sporadically I'm not that sure. > So you suggest to rename "TESTS" to "TEST" (diff), by removing the description, the requirement to python this introduced is hidden from the user, do you think this is good? Yes, I don't see the need to list dependencies in option descriptions. TEST option already implies that tests require some extra depends which should be installed before build. > Note that test is currently not run in the build process, but before staging. It's the part of (package) build process. > So if I switched to the new tests framework (do you have any information/links on it?), what would that essentially mean? It doesn't seem to be documented in PHB yet, however you could check out this review: https://reviews.freebsd.org/D3680 That would mean that tests are no longer run during the package builds and would thus no longer break them, but you can still run them by calling test target. bdrewery@ should be working on test target support in poudriere, so at some point we'll have separate poudriere setup which runs tests for all ports, so no failure comes unnoticed. Also this way no-network limitation could not be enforced for tests. > Also, is there anything you can provide that would allow me to create a setup like the current build environment, so I can reproduce the issue and potentially fix it? You could ask antoine@ about the cluster setup. >> All unit tests are network based > I'm not sure of cluster setup, but ports are generally not allowed to access any network after fetch phase. My first guess was that it was enforced on the cluster, thus the failures. However if the failures only happen sporadically I'm not that sure. Well, by network I mean 127.0.0.1/::1. Afaik this is ok. >> So you suggest to rename "TESTS" to "TEST" (diff), by removing the description, the requirement to python this introduced is hidden from the user, do you think this is good? > Yes, I don't see the need to list dependencies in option descriptions. TEST option already implies that tests require some extra depends which should be installed before build. OK >> Note that test is currently not run in the build process, but before staging. > It's the part of (package) build process. > So if I switched to the new tests framework (do you have any information/links on it?), what would that essentially mean? >> It doesn't seem to be documented in PHB yet, however you could check out this review: > https://reviews.freebsd.org/D3680 I will look into this. >> That would mean that tests are no longer run during the package builds and would thus no longer break them, but you can still run them by calling test target. bdrewery@ should be working on test target support in poudriere, so at some point we'll have separate poudriere setup which runs tests for all ports, so no failure comes unnoticed. Also this way no-network limitation could not be enforced for tests. Like I said, it's only localhost and should be fine. Having a test host would be most valuable, as in the past those tests actually helped discover bugs in the kernel while testing Ice. >> Also, is there anything you can provide that would allow me to create a setup like the current build environment, so I can reproduce the issue and potentially fix it? > You could ask antoine@ about the cluster setup. Will do (asked peter wemm and bdrewery@ among others in the past). So, any comments? Not yet. Actually yes, do you know when the test builders will be up and running? Otherwise I would just remove the offending tests for the time being and the rest, once we have test builders in place. (In reply to Michael Gmelin from comment #6) > Actually yes, do you know when the test builders will be up and running? No. > Otherwise I would just remove the offending tests for the time being and the > rest, once we have test builders in place. You should not remove any tests. This PR can be committed right away, it doesn't depend on anything. (In reply to Dmitry Marakasov from comment #7) > You should not remove any tests. This PR can be committed right away, it > doesn't depend on anything. What I've meant is that it would fix the build right away, and test failures may be investigated as soon as test-aware poudriere is ready. Hm, while there is no timeline for test builders at all, I would prefer to disable the two affected tests when they run on the package builder (easy to check for), that way I'll know of breakage early. (In reply to Michael Gmelin from comment #9) > Hm, while there is no timeline for test builders at all, I would prefer to > disable the two affected tests when they run on the package builder (easy to > check for), that way I'll know of breakage early. Tests should not run during the package build in any case. I just updated all three Ice ports to 3.6.1 (r404582) and used that opportunity to incorporate those changes (devel/py-ice also needed changes, so the new options work correctly). I did testing using poudriere testport -i and then running "make test" manually within the same session. Do you have any update on the status of a test host or at least automatic testing when running "poudriere testport"? (In reply to Michael Gmelin from comment #11) > I just updated all three Ice ports to 3.6.1 (r404582) and used that > opportunity to incorporate those changes (devel/py-ice also needed changes, > so the new options work correctly). Thank you. > I did testing using poudriere testport -i and then running "make test" > manually within the same session. > > Do you have any update on the status of a test host or at least automatic > testing when running "poudriere testport"? No. |