Attachment #244542 for bug #273483

View | Details | Raw Unified | Return to bug 273483
Collapse All | Expand All

Lines 661-666 variables must be specified in poudriere's [.filename]#make.conf# and not in Link Here

(-)b/documentation/content/en/books/porters-handbook/testing/_index.adoc (-5 / +3 lines)
661	Refer to package:ports-mgmt/poudriere[] documentation for details.	661	Refer to package:ports-mgmt/poudriere[] documentation for details.
662	====	662	====
663		663
664	Please refer to the debugging information in the	664	Please refer to the extref:{developers-handbook}tools/#debugging[debugging information in the
665	extref:{developers-handbook}[Developer's Handbook, debugging] for more details	665	Developer's Handbook] for more details
666	about the debugging tools available.	666	about the tools available.
667	-
668	--
669	.../porters-handbook/testing/_index.adoc \| 49 ++++++-------------	667	.../porters-handbook/testing/_index.adoc \| 49 ++++++-------------
670	1 file changed, 15 insertions(+), 34 deletions(-)	668	1 file changed, 15 insertions(+), 34 deletions(-)




[[make-test]]
== Running `make test`

Even if the port builds fine, it is a good idea to ensure that the software correctly does what it is supposed to do.
If the original upstream project provides tests along with the software, it is a good idea to run them and check everything works as expected.
If the original upstream project provides tests along with the software, it is a
good idea to run them and check everything works as expected.

A port can enable tests automatically by using the `TEST_TARGET` variable.
When set, this variable contains the name of the testing target of the port.
This is usually just `test` but other names include `tests`, `check` or for specific cases things like `run_tests.py`.
specific cases things like `run_tests.py`.

In addition to the `TEST_TARGET` variable the framework provides the following variables to control the tests execution:
variables to control the tests execution:

* `TEST_WRKSRC` is the directory to do the tests in.
* `TEST_ENV` contains additional variables to be passed to the test stage.
* `TEST_ARGS` contains any extra arguments passed to  the test stage.

Examples of use of these variables can be found in package:cad/xyce[], package:www/libjwt[] and others.
package:www/libjwt[] and others.

[NOTE]
====

It often has newer interesting features.
A ports committer will want to use the development version because it is what is used in production, and has all the new features that will make sure everything is exactly right.
A contributor will not necessarily need those as the most important fixes are backported to released version.
The main reason for the use of the development version to build the official package is because it is faster, in a way that will shorten a full build from 18 hours to 17 hours when using a high end 32 CPU server with 128GB of RAM.
in a way that will shorten a full build from 18 hours to 17 hours when using a high end 32 CPU server with 128GB of RAM.
Those optimizations will not matter a lot when building ports on a desktop machine.

[[testing-poudriere-setup]]

# poudriere jail -c -j 131Ramd64 -v 13.1-RELEASE -a amd64
....

Fetch a `13.1-RELEASE` for `amd64` from the FTP server given by `FREEBSD_HOST` in [.filename]#poudriere.conf#, create the zfs file system `tank/poudriere/jails/131Ramd64`, mount it on [.filename]#/poudriere/jails/131Ramd64# and extract the `13.1-RELEASE` tarballs into this file system.
create the zfs file system `tank/poudriere/jails/131Ramd64`,
mount it on [.filename]#/poudriere/jails/131Ramd64# and extract the `13.1-RELEASE` tarballs into this file system.

[source,shell]
....
# poudriere jail -c -j 12i386 -v stable/12 -a i386 -m git+https
....

Create `tank/poudriere/jails/12i386`, mount it on [.filename]#/poudriere/jails/12i386#, then check out the tip of the Git branch of `FreeBSD-12-STABLE` from `GIT_HOST` in [.filename]#poudriere.conf# or the default `git.freebsd.org` into [.filename]#/poudriere/jails/12i386/usr/src#, then complete a `buildworld` and install it into [.filename]#/poudriere/jails/12i386#.
then check out the tip of the Git branch of `FreeBSD-12-STABLE` from `GIT_HOST` in [.filename]#poudriere.conf# or the default `git.freebsd.org` into [.filename]#/poudriere/jails/12i386/usr/src#,
then complete a `buildworld` and install it into [.filename]#/poudriere/jails/12i386#.

[NOTE]
====


[TIP]
====
When testing ports where [.filename]#pkg-plist# is altered during build depending on the selected options, it is recommended to perform a test run with all options selected _and_ one with all options deselected.
it is recommended to perform a test run with all options selected _and_ one with all options deselected.
====

[[testing-poudriere-sets]]

# poudriere distclean -p portstree
....

will scan the distfiles folder, `DISTFILES_CACHE` in [.filename]#poudriere.conf#, versus the ports tree given by the `-p _portstree_` argument and prompt for removal of those distfiles.
versus the ports tree given by the `-p _portstree_` argument and prompt for removal of those distfiles.
To skip the prompt and remove all unused files unconditionally, the `-y` argument can be added:

[source,shell]


Sometimes things go wrong and the port fails at run time.
The framework provides some facilities to help in debugging ports.
These helpers are limited since the way of debugging a port heavily depends on the technology used.
the technology used.
The following variables help with debugging ports:

* `WITH_DEBUG`. If set, ports are built with debugging symbols.
* `WITH_DEBUG_PORTS`. Specifies a list of ports to be built with `WITH_DEBUG` set.
* `DEBUG_FLAGS`. Used to specify additional flags to `CFLAGS`. Defaults to `-g`.

When `WITH_DEBUG` is set, either globally or for a list of ports, the resulting binaries are not stripped.
binaries are not stripped.

These variables can be specified in [.filename]#make.conf# or in the command line:
line:

[source,shell]
....


[NOTE]
====
If the port is built using package:ports-mgmt/poudriere[] the debugging variables must be specified in poudriere's [.filename]#make.conf# and not in [.filename]#/etc/make.conf#.
variables must be specified in poudriere's [.filename]#make.conf# and not in
[.filename]#/etc/make.conf#.
Refer to package:ports-mgmt/poudriere[] documentation for details.
====

Please refer to the extref:{developers-handbook}tools/#debugging[debugging information in the Developer's Handbook] for more details about the tools available.
Developer's Handbook] for more details
about the tools available.
- 
--
.../content/en/books/porters-handbook/testing/_index.adoc       | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

Lines 108-114 For a new port, `portlint -A` is the most thorough; for an existing port, `portl Link Here

(-)b/documentation/content/en/books/porters-handbook/testing/_index.adoc (-2 / +1 lines)
108	Since `portlint` uses heuristics to try to figure out errors, it can produce false positive warnings.	108	Since `portlint` uses heuristics to try to figure out errors, it can produce false positive warnings.
109	In addition, occasionally something that is flagged as a problem really cannot be done in any other way due to limitations in the ports framework.	109	In addition, occasionally something that is flagged as a problem really cannot be done in any other way due to limitations in the ports framework.
110	pass:[<!-- vale Vale.Terms = NO -->]	110	pass:[<!-- vale Vale.Terms = NO -->]
111	When in doubt, the best thing to do is ask on {freebsd-ports}.	111	When in doubt, the best thing to do is ask on the {freebsd-ports}.
112	pass:[<!-- vale Vale.Terms = YES -->]	112	pass:[<!-- vale Vale.Terms = YES -->]
113		113
114	[[testing-porttools]]	114	[[testing-porttools]]
115	-

Return to bug 273483

Lines 68-92 Also note that running a recent version of `portlint` (as specified in the next Link Here

(-)b/documentation/content/en/books/porters-handbook/testing/_index.adoc (-36 / +15 lines)
68	[[make-test]]	68	[[make-test]]
69	== Running `make test`	69	== Running `make test`
70		70
71	Even if the port builds fine, it is a good idea to ensure that the software	71	Even if the port builds fine, it is a good idea to ensure that the software correctly does what it is supposed to do.
72	correctly does what it is supposed to do.	72	If the original upstream project provides tests along with the software, it is a good idea to run them and check everything works as expected.
73	If the original upstream project provides tests along with the software, it is a
74	good idea to run them and check everything works as expected.
75		73
76	A port can enable tests automatically by using the `TEST_TARGET` variable.	74	A port can enable tests automatically by using the `TEST_TARGET` variable.
77	When set, this variable contains the name of the testing target of the port.	75	When set, this variable contains the name of the testing target of the port.
78	This is usually just `test` but other names include `tests`, `check` or for	76	This is usually just `test` but other names include `tests`, `check` or for specific cases things like `run_tests.py`.
79	specific cases things like `run_tests.py`.
80		77
81	In addition to the `TEST_TARGET` variable the framework provides the following	78	In addition to the `TEST_TARGET` variable the framework provides the following variables to control the tests execution:
82	variables to control the tests execution:
83		79
84	* `TEST_WRKSRC` is the directory to do the tests in.	80	* `TEST_WRKSRC` is the directory to do the tests in.
85	* `TEST_ENV` contains additional variables to be passed to the test stage.	81	* `TEST_ENV` contains additional variables to be passed to the test stage.
86	* `TEST_ARGS` contains any extra arguments passed to the test stage.	82	* `TEST_ARGS` contains any extra arguments passed to the test stage.
87		83
88	Examples of use of these variables can be found in package:cad/xyce[],	84	Examples of use of these variables can be found in package:cad/xyce[], package:www/libjwt[] and others.
89	package:www/libjwt[] and others.
90		85
91	[NOTE]	86	[NOTE]
92	====	87	====
Lines 246-253 This development version is used for the official FreeBSD package builds, so it Link Here
246	It often has newer interesting features.	241	It often has newer interesting features.
247	A ports committer will want to use the development version because it is what is used in production, and has all the new features that will make sure everything is exactly right.	242	A ports committer will want to use the development version because it is what is used in production, and has all the new features that will make sure everything is exactly right.
248	A contributor will not necessarily need those as the most important fixes are backported to released version.	243	A contributor will not necessarily need those as the most important fixes are backported to released version.
249	The main reason for the use of the development version to build the official package is because it is faster,	244	The main reason for the use of the development version to build the official package is because it is faster, in a way that will shorten a full build from 18 hours to 17 hours when using a high end 32 CPU server with 128GB of RAM.
250	in a way that will shorten a full build from 18 hours to 17 hours when using a high end 32 CPU server with 128GB of RAM.
251	Those optimizations will not matter a lot when building ports on a desktop machine.	245	Those optimizations will not matter a lot when building ports on a desktop machine.
252		246
253	[[testing-poudriere-setup]]	247	[[testing-poudriere-setup]]
Lines 295-312 Create the base jails which poudriere will use for building: Link Here
295	# poudriere jail -c -j 131Ramd64 -v 13.1-RELEASE -a amd64	289	# poudriere jail -c -j 131Ramd64 -v 13.1-RELEASE -a amd64
296	....	290	....
297		291
298	Fetch a `13.1-RELEASE` for `amd64` from the FTP server given by `FREEBSD_HOST` in [.filename]#poudriere.conf#,	292	Fetch a `13.1-RELEASE` for `amd64` from the FTP server given by `FREEBSD_HOST` in [.filename]#poudriere.conf#, create the zfs file system `tank/poudriere/jails/131Ramd64`, mount it on [.filename]#/poudriere/jails/131Ramd64# and extract the `13.1-RELEASE` tarballs into this file system.
299	create the zfs file system `tank/poudriere/jails/131Ramd64`,
300	mount it on [.filename]#/poudriere/jails/131Ramd64# and extract the `13.1-RELEASE` tarballs into this file system.
301		293
302	[source,shell]	294	[source,shell]
303	....	295	....
304	# poudriere jail -c -j 12i386 -v stable/12 -a i386 -m git+https	296	# poudriere jail -c -j 12i386 -v stable/12 -a i386 -m git+https
305	....	297	....
306		298
307	Create `tank/poudriere/jails/12i386`, mount it on [.filename]#/poudriere/jails/12i386#,	299	Create `tank/poudriere/jails/12i386`, mount it on [.filename]#/poudriere/jails/12i386#, then check out the tip of the Git branch of `FreeBSD-12-STABLE` from `GIT_HOST` in [.filename]#poudriere.conf# or the default `git.freebsd.org` into [.filename]#/poudriere/jails/12i386/usr/src#, then complete a `buildworld` and install it into [.filename]#/poudriere/jails/12i386#.
308	then check out the tip of the Git branch of `FreeBSD-12-STABLE` from `GIT_HOST` in [.filename]#poudriere.conf# or the default `git.freebsd.org` into [.filename]#/poudriere/jails/12i386/usr/src#,
309	then complete a `buildworld` and install it into [.filename]#/poudriere/jails/12i386#.
310		300
311	[NOTE]	301	[NOTE]
312	====	302	====
Lines 511-518 Testing dependent ports with non-default options can be accomplished using sets, Link Here
511		501
512	[TIP]	502	[TIP]
513	====	503	====
514	When testing ports where [.filename]#pkg-plist# is altered during build depending on the selected options,	504	When testing ports where [.filename]#pkg-plist# is altered during build depending on the selected options, it is recommended to perform a test run with all options selected _and_ one with all options deselected.
515	it is recommended to perform a test run with all options selected _and_ one with all options deselected.
516	====	505	====
517		506
518	[[testing-poudriere-sets]]	507	[[testing-poudriere-sets]]
Lines 620-627 The command Link Here
620	# poudriere distclean -p portstree	609	# poudriere distclean -p portstree
621	....	610	....
622		611
623	will scan the distfiles folder, `DISTFILES_CACHE` in [.filename]#poudriere.conf#,	612	will scan the distfiles folder, `DISTFILES_CACHE` in [.filename]#poudriere.conf#, versus the ports tree given by the `-p _portstree_` argument and prompt for removal of those distfiles.
624	versus the ports tree given by the `-p _portstree_` argument and prompt for removal of those distfiles.
625	To skip the prompt and remove all unused files unconditionally, the `-y` argument can be added:	613	To skip the prompt and remove all unused files unconditionally, the `-y` argument can be added:
626		614
627	[source,shell]	615	[source,shell]
Lines 634-652 To skip the prompt and remove all unused files unconditionally, the `-y` argumen Link Here
634		622
635	Sometimes things go wrong and the port fails at run time.	623	Sometimes things go wrong and the port fails at run time.
636	The framework provides some facilities to help in debugging ports.	624	The framework provides some facilities to help in debugging ports.
637	These helpers are limited since the way of debugging a port heavily depends on	625	These helpers are limited since the way of debugging a port heavily depends on the technology used.
638	the technology used.
639	The following variables help with debugging ports:	626	The following variables help with debugging ports:
640		627
641	* `WITH_DEBUG`. If set, ports are built with debugging symbols.	628	* `WITH_DEBUG`. If set, ports are built with debugging symbols.
642	* `WITH_DEBUG_PORTS`. Specifies a list of ports to be built with `WITH_DEBUG` set.	629	* `WITH_DEBUG_PORTS`. Specifies a list of ports to be built with `WITH_DEBUG` set.
643	* `DEBUG_FLAGS`. Used to specify additional flags to `CFLAGS`. Defaults to `-g`.	630	* `DEBUG_FLAGS`. Used to specify additional flags to `CFLAGS`. Defaults to `-g`.
644		631
645	When `WITH_DEBUG` is set, either globally or for a list of ports, the resulting	632	When `WITH_DEBUG` is set, either globally or for a list of ports, the resulting binaries are not stripped.
646	binaries are not stripped.
647		633
648	These variables can be specified in [.filename]#make.conf# or in the command	634	These variables can be specified in [.filename]#make.conf# or in the command line:
649	line:
650		635
651	[source,shell]	636	[source,shell]
652	....	637	....
Lines 655-666 line: Link Here
655		640
656	[NOTE]	641	[NOTE]
657	====	642	====
658	If the port is built using package:ports-mgmt/poudriere[] the debugging	643	If the port is built using package:ports-mgmt/poudriere[] the debugging variables must be specified in poudriere's [.filename]#make.conf# and not in [.filename]#/etc/make.conf#.
659	variables must be specified in poudriere's [.filename]#make.conf# and not in
660	[.filename]#/etc/make.conf#.
661	Refer to package:ports-mgmt/poudriere[] documentation for details.	644	Refer to package:ports-mgmt/poudriere[] documentation for details.
662	====	645	====
663		646
664	Please refer to the extref:{developers-handbook}tools/#debugging[debugging information in the	647	Please refer to the extref:{developers-handbook}tools/#debugging[debugging information in the Developer's Handbook] for more details about the tools available.
665	Developer's Handbook] for more details
666	about the tools available.
667	-
668	--
669	.../content/en/books/porters-handbook/testing/_index.adoc \| 2 +-	648	.../content/en/books/porters-handbook/testing/_index.adoc \| 2 +-
670	1 file changed, 1 insertion(+), 1 deletion(-)	649	1 file changed, 1 insertion(+), 1 deletion(-)