Bug 235200

Summary: lib.libc.sys.sendfile_test.* fail
Product: Base System Reporter: Li-Wen Hsu <lwhsu>
Component: testsAssignee: Enji Cooper <ngie>
Status: Closed FIXED    
Severity: Affects Some People CC: asomers
Priority: --- Flags: ngie: mfc-stable11+
ngie: mfc-stable12+
Version: CURRENT   
Hardware: Any   
OS: Any   
See Also: https://reviews.freebsd.org/D18969
https://reviews.freebsd.org/D19026
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234809
https://reviews.freebsd.org/D19188

Description Li-Wen Hsu freebsd_committer 2019-01-25 19:08:54 UTC
These tests all failed with message:

/usr/src/lib/libc/tests/sys/sendfile_test.c:117: error != 0: getaddrinfo failed: Success

lib.libc.sys.sendfile_test.fd_negative_bad_fd_v4
lib.libc.sys.sendfile_test.fd_negative_bad_fd_v6
lib.libc.sys.sendfile_test.fd_positive_file_v4
lib.libc.sys.sendfile_test.fd_positive_file_v6
lib.libc.sys.sendfile_test.fd_positive_shm_v4
lib.libc.sys.sendfile_test.fd_positive_shm_v6
lib.libc.sys.sendfile_test.flags_v4
lib.libc.sys.sendfile_test.flags_v6
lib.libc.sys.sendfile_test.hdtr_negative_bad_pointers_v4
lib.libc.sys.sendfile_test.hdtr_negative_bad_pointers_v6
lib.libc.sys.sendfile_test.offset_negative_value_less_than_zero_v4
lib.libc.sys.sendfile_test.offset_negative_value_less_than_zero_v6
lib.libc.sys.sendfile_test.s_negative_not_connected_socket_v4
lib.libc.sys.sendfile_test.s_negative_not_connected_socket_v6
lib.libc.sys.sendfile_test.s_negative_udp_socket_v4
lib.libc.sys.sendfile_test.s_negative_udp_socket_v6
lib.libc.sys.sendfile_test.sbytes_negative_v4
lib.libc.sys.sendfile_test.sbytes_negative_v6
lib.libc.sys.sendfile_test.sbytes_positive_v4
lib.libc.sys.sendfile_test.sbytes_positive_v6
Comment 1 commit-hook freebsd_committer 2019-01-26 03:44:15 UTC
A commit references this bug:

Author: ngie
Date: Sat Jan 26 03:43:12 UTC 2019
New revision: 343461
URL: https://svnweb.freebsd.org/changeset/base/343461

Log:
  Fix reporting errors with `gai_strerror(..)`

  The return value (`err`) should be checked; not the `errno` value.

  PR:		235200
  Approved by:	emaste (mentor)
  Reviewed by:	asomers, lwhsu
  MFC after:	28 days
  MFC with:	r343362, r343365, r343367-r343368
  Differential Revision: https://reviews.freebsd.org/D18969

Changes:
  head/lib/libc/tests/sys/sendfile_test.c
Comment 2 Li-Wen Hsu freebsd_committer 2019-01-26 09:17:51 UTC
These tests still fail because the testing VM doesn't have Internet connection. I'll work on that.  In the meanwhile, do you think it's reasonable to mark these cases "skip" when there is no Internet connection or make it not depending on public Internet connection?
Comment 3 Enji Cooper freebsd_committer 2019-01-29 23:20:51 UTC
(In reply to Li-Wen Hsu from comment #2)

I opened up https://reviews.freebsd.org/D19026 to remove the DNS lookup piece, as it's not important to the overall test. I just wanted to avoid hardcoding the address (c'est la vie -- I didn't anticipate this... *shrugs*).
Comment 4 commit-hook freebsd_committer 2019-02-04 19:12:53 UTC
A commit references this bug:

Author: ngie
Date: Mon Feb  4 19:12:45 UTC 2019
New revision: 343751
URL: https://svnweb.freebsd.org/changeset/base/343751

Log:
  Avoid the DNS lookup for "localhost"

  ci.FreeBSD.org does not have access to a DNS resolver/network (unlike my test
  VM), so in order for the test to pass on the host, it needs to avoid the DNS
  lookup by using the numeric host address representation.

  PR:		235200
  Reviewed by:	asomers, lwhsu
  Approved by:	emaste (mentor)
  MFC after:	2 weeks
  MFC with:	r343362, r343365, r343367-r343368, r343461
  Differential Revision: https://reviews.freebsd.org/D19026

Changes:
  head/lib/libc/tests/sys/sendfile_test.c
Comment 5 Li-Wen Hsu freebsd_committer 2019-02-05 20:46:18 UTC
After r343751, the cases in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235200#c0 is fixed but now we have lib.libc.sys.sendfile_test.hdtr_positive_v{4,6} failing.
Comment 7 Enji Cooper freebsd_committer 2019-02-06 00:20:46 UTC
(In reply to Li-Wen Hsu from comment #6)

Funny... I must have fixed some of the underlying issues in prior commits. Weird why it was failing for me on my host when testing out that commit. I'll try again :)!
Comment 8 Enji Cooper freebsd_committer 2019-02-06 00:23:09 UTC
(In reply to Enji Cooper from comment #7)

Nope. Someone else must have fixed something... (this kernel is 4 days old)?

$ kyua test -k /usr/tests/lib/libc/sys/Kyuafile sendfile_test
sendfile_test:fd_negative_bad_fd_v4  ->  passed  [0.007s]
sendfile_test:fd_negative_bad_fd_v6  ->  passed  [0.009s]
sendfile_test:fd_positive_file_v4  ->  passed  [0.005s]
sendfile_test:fd_positive_file_v6  ->  passed  [0.006s]
sendfile_test:fd_positive_shm_v4  ->  passed  [0.006s]
sendfile_test:fd_positive_shm_v6  ->  passed  [0.006s]
sendfile_test:flags_v4  ->  passed  [0.021s]
sendfile_test:flags_v6  ->  passed  [0.019s]
sendfile_test:hdtr_negative_bad_pointers_v4  ->  passed  [0.003s]
sendfile_test:hdtr_negative_bad_pointers_v6  ->  passed  [0.003s]
sendfile_test:hdtr_positive_v4  ->  expected_failure: The header/trailer testcases fail today with a data mismatch; bug # 234809: /usr/src/contrib/atf/atf-c/utils.c:431: exitstatus != WEXITSTATUS(status)  [0.005s]
sendfile_test:hdtr_positive_v6  ->  expected_failure: The header/trailer testcases fail today with a data mismatch; bug # 234809: /usr/src/contrib/atf/atf-c/utils.c:431: exitstatus != WEXITSTATUS(status)  [0.006s]
sendfile_test:offset_negative_value_less_than_zero_v4  ->  passed  [0.002s]
sendfile_test:offset_negative_value_less_than_zero_v6  ->  passed  [0.002s]
sendfile_test:s_negative_not_connected_socket_v4  ->  passed  [0.003s]
sendfile_test:s_negative_not_connected_socket_v6  ->  passed  [0.002s]
sendfile_test:s_negative_not_descriptor  ->  passed  [0.002s]
sendfile_test:s_negative_not_socket_file_descriptor  ->  passed  [0.003s]
sendfile_test:s_negative_udp_socket_v4  ->  passed  [0.002s]
sendfile_test:s_negative_udp_socket_v6  ->  passed  [0.002s]
sendfile_test:sbytes_negative_v4  ->  expected_failure: bug 232210: EFAULT assert fails because copyout(9) call is not checked: /usr/src/lib/libc/tests/sys/sendfile_test.c:945: Expected true value in error == -1  [0.003s]
sendfile_test:sbytes_negative_v6  ->  expected_failure: bug 232210: EFAULT assert fails because copyout(9) call is not checked: /usr/src/lib/libc/tests/sys/sendfile_test.c:945: Expected true value in error == -1  [0.002s]
sendfile_test:sbytes_positive_v4  ->  passed  [0.003s]
sendfile_test:sbytes_positive_v6  ->  passed  [0.002s]

Results file id is usr_tests_lib_libc_sys.20190206-002147-086051
Results saved to /home/ngie/.kyua/store/results.usr_tests_lib_libc_sys.20190206-002147-086051.db

24/24 passed (0 failed)
$ uname -a
FreeBSD pinklady-fbsd-current.local 13.0-CURRENT FreeBSD 13.0-CURRENT r343668+c0d866da71a6(master) GENERIC-NODEBUG  amd64
Comment 9 commit-hook freebsd_committer 2019-02-19 22:20:20 UTC
A commit references this bug:

Author: ngie
Date: Tue Feb 19 22:19:32 UTC 2019
New revision: 344310
URL: https://svnweb.freebsd.org/changeset/base/344310

Log:
  Make `server_cat(..)` handle short receives

  In short, the prior code was far too simplistic when it came to calling recv(2)
  and failed intermittently (or in the case of Jenkins, deterministically).

  Handle short recv(2)s by checking the return code and incrementing the window
  into the buffer by the number of received bytes. If the number of received
  bytes <= 0, then bail out of the loop, and test the total number of received
  bytes vs the expected number of bytes sent for equality, and base whether or
  not the test passes/fails on that fact.

  Remove the expected failure, now that the hdtr testcases deterministically pass
  on my host after this change [1].

  PR:		234809 [1], 235200
  Reviewed by:	asomers
  Approved by:	emaste (mentor)
  MFC after:	1 week
  Differential Revision: https://reviews.freebsd.org/D19188

Changes:
  head/lib/libc/tests/sys/sendfile_test.c
Comment 10 commit-hook freebsd_committer 2019-03-10 16:13:44 UTC
A commit references this bug:

Author: ngie
Date: Sun Mar 10 16:13:01 UTC 2019
New revision: 344977
URL: https://svnweb.freebsd.org/changeset/base/344977

Log:
  MFC r343362,r343365,r343367,r343368,r343461,r343751,r344310:

  r343362:

  Add [initial] functional tests for sendfile(2) as lib/libc/sys/sendfile

  These testcases exercise a number of functional requirements for sendfile(2).

  The testcases use IPv4 and IPv6 domain sockets with TCP, and were confirmed
  functional on UFS and ZFS. UDP address family sockets cannot be used per the
  sendfile(2) contract, thus using UDP sockets is outside the scope of
  testing the syscall in positive cases. As seen in
  `:s_negative_udp_socket_test`, UDP is used to test the sendfile(2) contract
  to ensure that EINVAL is returned by sendfile(2).

  The testcases added explicitly avoid testing out `SF_SYNC` due to the
  complexity of verifying that support. However, this is a good next logical
  item to verify.

  The `hdtr_positive*` testcases work to a certain degree (the header
  testcases pass), but the trailer testcases do not work (it is an expected
  failure). In particular, the value received by the mock server doesn't match
  the expected value, and instead looks something like the following (using
  python array notation):

  `trailer[:]message[1:]`

  instead of:

  `message[:]trailer[:]`

  This makes me think there's a buffer overrun issue or problem with the
  offset somewhere in the sendfile(2) system call, but I need to do some
  other testing first to verify that the code is indeed sane, and my
  assumptions/code isn't buggy.

  The `sbytes_negative` testcases that check `sbytes` being set to an
  invalid value resulting in `EFAULT` fails today as the other change
  (which checks `copyout(9)`) has not been committed [1]. Thus, it
  should remain an expected failure (see bug 232210 for more details
  on this item).

  Next steps for testing sendfile(2):
  1. Fix the header/trailer testcases so that they pass.
  2. Setup if_tap interface and test with it, instead of using "localhost", per
     @asomers's suggestion.
  3. Handle short recv(2)'s in `server_cat(..)`.
  4. Add `SF_SYNC` support.
  5. Add some more negative tests outside the scope of the functional contract.

  PR: 		232210

  r343365:

  Unbreak the gcc build with sendfile_test after r343362

  gcc 8.x is more pedantic than clang 7.x with format strings and the tests
  passed `void*` variables while supplying `%s` (which is technically
  incorrect).

  Make the affected `void*` variables use `char*` storage instead to address
  this issue, as the compiler will upcast the values to `char*`.

  MFC with:	r343362

  r343367:

  Unbreak the build on architectures where size_t isn't synonymous with uintmax_t

  I should have used `%zu` instead of `%ju` with `size_t` types.

  MFC with:	r343362, r343365
  Pointyhat to:	ngie

  r343368:

  Fix up r343367

  I should have only changed the format qualifier with the `size_t` value,
  `length`, not the other [`off_t`] value, `dest_file_size`.

  MFC with:	r343362, r343365, r343367

  r343461:

  Fix reporting errors with `gai_strerror(..)`

  The return value (`err`) should be checked; not the `errno` value.

  PR:		235200
  MFC with:	r343362, r343365, r343367-r343368

  r343751:

  Avoid the DNS lookup for "localhost"

  ci.FreeBSD.org does not have access to a DNS resolver/network (unlike my test
  VM), so in order for the test to pass on the host, it needs to avoid the DNS
  lookup by using the numeric host address representation.

  PR:		235200
  MFC with:	r343362, r343365, r343367-r343368, r343461

  r344310:

  Make `server_cat(..)` handle short receives

  In short, the prior code was far too simplistic when it came to calling recv(2)
  and failed intermittently (or in the case of Jenkins, deterministically).

  Handle short recv(2)s by checking the return code and incrementing the window
  into the buffer by the number of received bytes. If the number of received
  bytes <= 0, then bail out of the loop, and test the total number of received
  bytes vs the expected number of bytes sent for equality, and base whether or
  not the test passes/fails on that fact.

  Remove the expected failure, now that the hdtr testcases deterministically pass
  on my host after this change [1].

  PR:		234809 [1], 235200

  Approved by:	emaste (mentor)
  Differential Revision: https://reviews.freebsd.org/D19524

Changes:
_U  stable/11/
  stable/11/lib/libc/tests/sys/Makefile
  stable/11/lib/libc/tests/sys/sendfile_test.c
Comment 11 commit-hook freebsd_committer 2019-03-10 16:13:46 UTC
A commit references this bug:

Author: ngie
Date: Sun Mar 10 16:13:01 UTC 2019
New revision: 344977
URL: https://svnweb.freebsd.org/changeset/base/344977

Log:
  MFC r343362,r343365,r343367,r343368,r343461,r343751,r344310:

  r343362:

  Add [initial] functional tests for sendfile(2) as lib/libc/sys/sendfile

  These testcases exercise a number of functional requirements for sendfile(2).

  The testcases use IPv4 and IPv6 domain sockets with TCP, and were confirmed
  functional on UFS and ZFS. UDP address family sockets cannot be used per the
  sendfile(2) contract, thus using UDP sockets is outside the scope of
  testing the syscall in positive cases. As seen in
  `:s_negative_udp_socket_test`, UDP is used to test the sendfile(2) contract
  to ensure that EINVAL is returned by sendfile(2).

  The testcases added explicitly avoid testing out `SF_SYNC` due to the
  complexity of verifying that support. However, this is a good next logical
  item to verify.

  The `hdtr_positive*` testcases work to a certain degree (the header
  testcases pass), but the trailer testcases do not work (it is an expected
  failure). In particular, the value received by the mock server doesn't match
  the expected value, and instead looks something like the following (using
  python array notation):

  `trailer[:]message[1:]`

  instead of:

  `message[:]trailer[:]`

  This makes me think there's a buffer overrun issue or problem with the
  offset somewhere in the sendfile(2) system call, but I need to do some
  other testing first to verify that the code is indeed sane, and my
  assumptions/code isn't buggy.

  The `sbytes_negative` testcases that check `sbytes` being set to an
  invalid value resulting in `EFAULT` fails today as the other change
  (which checks `copyout(9)`) has not been committed [1]. Thus, it
  should remain an expected failure (see bug 232210 for more details
  on this item).

  Next steps for testing sendfile(2):
  1. Fix the header/trailer testcases so that they pass.
  2. Setup if_tap interface and test with it, instead of using "localhost", per
     @asomers's suggestion.
  3. Handle short recv(2)'s in `server_cat(..)`.
  4. Add `SF_SYNC` support.
  5. Add some more negative tests outside the scope of the functional contract.

  PR: 		232210

  r343365:

  Unbreak the gcc build with sendfile_test after r343362

  gcc 8.x is more pedantic than clang 7.x with format strings and the tests
  passed `void*` variables while supplying `%s` (which is technically
  incorrect).

  Make the affected `void*` variables use `char*` storage instead to address
  this issue, as the compiler will upcast the values to `char*`.

  MFC with:	r343362

  r343367:

  Unbreak the build on architectures where size_t isn't synonymous with uintmax_t

  I should have used `%zu` instead of `%ju` with `size_t` types.

  MFC with:	r343362, r343365
  Pointyhat to:	ngie

  r343368:

  Fix up r343367

  I should have only changed the format qualifier with the `size_t` value,
  `length`, not the other [`off_t`] value, `dest_file_size`.

  MFC with:	r343362, r343365, r343367

  r343461:

  Fix reporting errors with `gai_strerror(..)`

  The return value (`err`) should be checked; not the `errno` value.

  PR:		235200
  MFC with:	r343362, r343365, r343367-r343368

  r343751:

  Avoid the DNS lookup for "localhost"

  ci.FreeBSD.org does not have access to a DNS resolver/network (unlike my test
  VM), so in order for the test to pass on the host, it needs to avoid the DNS
  lookup by using the numeric host address representation.

  PR:		235200
  MFC with:	r343362, r343365, r343367-r343368, r343461

  r344310:

  Make `server_cat(..)` handle short receives

  In short, the prior code was far too simplistic when it came to calling recv(2)
  and failed intermittently (or in the case of Jenkins, deterministically).

  Handle short recv(2)s by checking the return code and incrementing the window
  into the buffer by the number of received bytes. If the number of received
  bytes <= 0, then bail out of the loop, and test the total number of received
  bytes vs the expected number of bytes sent for equality, and base whether or
  not the test passes/fails on that fact.

  Remove the expected failure, now that the hdtr testcases deterministically pass
  on my host after this change [1].

  PR:		234809 [1], 235200

  Approved by:	emaste (mentor)
  Differential Revision: https://reviews.freebsd.org/D19524

Changes:
_U  stable/11/
  stable/11/lib/libc/tests/sys/Makefile
  stable/11/lib/libc/tests/sys/sendfile_test.c
Comment 12 commit-hook freebsd_committer 2019-03-10 16:13:48 UTC
A commit references this bug:

Author: ngie
Date: Sun Mar 10 16:13:01 UTC 2019
New revision: 344977
URL: https://svnweb.freebsd.org/changeset/base/344977

Log:
  MFC r343362,r343365,r343367,r343368,r343461,r343751,r344310:

  r343362:

  Add [initial] functional tests for sendfile(2) as lib/libc/sys/sendfile

  These testcases exercise a number of functional requirements for sendfile(2).

  The testcases use IPv4 and IPv6 domain sockets with TCP, and were confirmed
  functional on UFS and ZFS. UDP address family sockets cannot be used per the
  sendfile(2) contract, thus using UDP sockets is outside the scope of
  testing the syscall in positive cases. As seen in
  `:s_negative_udp_socket_test`, UDP is used to test the sendfile(2) contract
  to ensure that EINVAL is returned by sendfile(2).

  The testcases added explicitly avoid testing out `SF_SYNC` due to the
  complexity of verifying that support. However, this is a good next logical
  item to verify.

  The `hdtr_positive*` testcases work to a certain degree (the header
  testcases pass), but the trailer testcases do not work (it is an expected
  failure). In particular, the value received by the mock server doesn't match
  the expected value, and instead looks something like the following (using
  python array notation):

  `trailer[:]message[1:]`

  instead of:

  `message[:]trailer[:]`

  This makes me think there's a buffer overrun issue or problem with the
  offset somewhere in the sendfile(2) system call, but I need to do some
  other testing first to verify that the code is indeed sane, and my
  assumptions/code isn't buggy.

  The `sbytes_negative` testcases that check `sbytes` being set to an
  invalid value resulting in `EFAULT` fails today as the other change
  (which checks `copyout(9)`) has not been committed [1]. Thus, it
  should remain an expected failure (see bug 232210 for more details
  on this item).

  Next steps for testing sendfile(2):
  1. Fix the header/trailer testcases so that they pass.
  2. Setup if_tap interface and test with it, instead of using "localhost", per
     @asomers's suggestion.
  3. Handle short recv(2)'s in `server_cat(..)`.
  4. Add `SF_SYNC` support.
  5. Add some more negative tests outside the scope of the functional contract.

  PR: 		232210

  r343365:

  Unbreak the gcc build with sendfile_test after r343362

  gcc 8.x is more pedantic than clang 7.x with format strings and the tests
  passed `void*` variables while supplying `%s` (which is technically
  incorrect).

  Make the affected `void*` variables use `char*` storage instead to address
  this issue, as the compiler will upcast the values to `char*`.

  MFC with:	r343362

  r343367:

  Unbreak the build on architectures where size_t isn't synonymous with uintmax_t

  I should have used `%zu` instead of `%ju` with `size_t` types.

  MFC with:	r343362, r343365
  Pointyhat to:	ngie

  r343368:

  Fix up r343367

  I should have only changed the format qualifier with the `size_t` value,
  `length`, not the other [`off_t`] value, `dest_file_size`.

  MFC with:	r343362, r343365, r343367

  r343461:

  Fix reporting errors with `gai_strerror(..)`

  The return value (`err`) should be checked; not the `errno` value.

  PR:		235200
  MFC with:	r343362, r343365, r343367-r343368

  r343751:

  Avoid the DNS lookup for "localhost"

  ci.FreeBSD.org does not have access to a DNS resolver/network (unlike my test
  VM), so in order for the test to pass on the host, it needs to avoid the DNS
  lookup by using the numeric host address representation.

  PR:		235200
  MFC with:	r343362, r343365, r343367-r343368, r343461

  r344310:

  Make `server_cat(..)` handle short receives

  In short, the prior code was far too simplistic when it came to calling recv(2)
  and failed intermittently (or in the case of Jenkins, deterministically).

  Handle short recv(2)s by checking the return code and incrementing the window
  into the buffer by the number of received bytes. If the number of received
  bytes <= 0, then bail out of the loop, and test the total number of received
  bytes vs the expected number of bytes sent for equality, and base whether or
  not the test passes/fails on that fact.

  Remove the expected failure, now that the hdtr testcases deterministically pass
  on my host after this change [1].

  PR:		234809 [1], 235200

  Approved by:	emaste (mentor)
  Differential Revision: https://reviews.freebsd.org/D19524

Changes:
_U  stable/11/
  stable/11/lib/libc/tests/sys/Makefile
  stable/11/lib/libc/tests/sys/sendfile_test.c
Comment 13 commit-hook freebsd_committer 2019-03-10 16:14:53 UTC
A commit references this bug:

Author: ngie
Date: Sun Mar 10 16:14:43 UTC 2019
New revision: 344978
URL: https://svnweb.freebsd.org/changeset/base/344978

Log:
  MFC r343362,r343365,r343367,r343368,r343461,r343751,r344310:

  r343362:

  Add [initial] functional tests for sendfile(2) as lib/libc/sys/sendfile

  These testcases exercise a number of functional requirements for sendfile(2).

  The testcases use IPv4 and IPv6 domain sockets with TCP, and were confirmed
  functional on UFS and ZFS. UDP address family sockets cannot be used per the
  sendfile(2) contract, thus using UDP sockets is outside the scope of
  testing the syscall in positive cases. As seen in
  `:s_negative_udp_socket_test`, UDP is used to test the sendfile(2) contract
  to ensure that EINVAL is returned by sendfile(2).

  The testcases added explicitly avoid testing out `SF_SYNC` due to the
  complexity of verifying that support. However, this is a good next logical
  item to verify.

  The `hdtr_positive*` testcases work to a certain degree (the header
  testcases pass), but the trailer testcases do not work (it is an expected
  failure). In particular, the value received by the mock server doesn't match
  the expected value, and instead looks something like the following (using
  python array notation):

  `trailer[:]message[1:]`

  instead of:

  `message[:]trailer[:]`

  This makes me think there's a buffer overrun issue or problem with the
  offset somewhere in the sendfile(2) system call, but I need to do some
  other testing first to verify that the code is indeed sane, and my
  assumptions/code isn't buggy.

  The `sbytes_negative` testcases that check `sbytes` being set to an
  invalid value resulting in `EFAULT` fails today as the other change
  (which checks `copyout(9)`) has not been committed [1]. Thus, it
  should remain an expected failure (see bug 232210 for more details
  on this item).

  Next steps for testing sendfile(2):
  1. Fix the header/trailer testcases so that they pass.
  2. Setup if_tap interface and test with it, instead of using "localhost", per
     @asomers's suggestion.
  3. Handle short recv(2)'s in `server_cat(..)`.
  4. Add `SF_SYNC` support.
  5. Add some more negative tests outside the scope of the functional contract.

  PR: 		232210

  r343365:

  Unbreak the gcc build with sendfile_test after r343362

  gcc 8.x is more pedantic than clang 7.x with format strings and the tests
  passed `void*` variables while supplying `%s` (which is technically
  incorrect).

  Make the affected `void*` variables use `char*` storage instead to address
  this issue, as the compiler will upcast the values to `char*`.

  MFC with:	r343362

  r343367:

  Unbreak the build on architectures where size_t isn't synonymous with uintmax_t

  I should have used `%zu` instead of `%ju` with `size_t` types.

  MFC with:	r343362, r343365
  Pointyhat to:	ngie

  r343368:

  Fix up r343367

  I should have only changed the format qualifier with the `size_t` value,
  `length`, not the other [`off_t`] value, `dest_file_size`.

  MFC with:	r343362, r343365, r343367

  r343461:

  Fix reporting errors with `gai_strerror(..)`

  The return value (`err`) should be checked; not the `errno` value.

  PR:		235200
  MFC with:	r343362, r343365, r343367-r343368

  r343751:

  Avoid the DNS lookup for "localhost"

  ci.FreeBSD.org does not have access to a DNS resolver/network (unlike my test
  VM), so in order for the test to pass on the host, it needs to avoid the DNS
  lookup by using the numeric host address representation.

  PR:		235200
  MFC with:	r343362, r343365, r343367-r343368, r343461

  r344310:

  Make `server_cat(..)` handle short receives

  In short, the prior code was far too simplistic when it came to calling recv(2)
  and failed intermittently (or in the case of Jenkins, deterministically).

  Handle short recv(2)s by checking the return code and incrementing the window
  into the buffer by the number of received bytes. If the number of received
  bytes <= 0, then bail out of the loop, and test the total number of received
  bytes vs the expected number of bytes sent for equality, and base whether or
  not the test passes/fails on that fact.

  Remove the expected failure, now that the hdtr testcases deterministically pass
  on my host after this change [1].

  PR:		234809 [1], 235200

  Approved by:	emaste (mentor, implicit: MFC to ^/stable/11 in r344977)
  Differential Revision: https://reviews.freebsd.org/D19523

Changes:
_U  stable/12/
  stable/12/lib/libc/tests/sys/Makefile
  stable/12/lib/libc/tests/sys/sendfile_test.c
Comment 14 commit-hook freebsd_committer 2019-03-10 16:14:55 UTC
A commit references this bug:

Author: ngie
Date: Sun Mar 10 16:14:43 UTC 2019
New revision: 344978
URL: https://svnweb.freebsd.org/changeset/base/344978

Log:
  MFC r343362,r343365,r343367,r343368,r343461,r343751,r344310:

  r343362:

  Add [initial] functional tests for sendfile(2) as lib/libc/sys/sendfile

  These testcases exercise a number of functional requirements for sendfile(2).

  The testcases use IPv4 and IPv6 domain sockets with TCP, and were confirmed
  functional on UFS and ZFS. UDP address family sockets cannot be used per the
  sendfile(2) contract, thus using UDP sockets is outside the scope of
  testing the syscall in positive cases. As seen in
  `:s_negative_udp_socket_test`, UDP is used to test the sendfile(2) contract
  to ensure that EINVAL is returned by sendfile(2).

  The testcases added explicitly avoid testing out `SF_SYNC` due to the
  complexity of verifying that support. However, this is a good next logical
  item to verify.

  The `hdtr_positive*` testcases work to a certain degree (the header
  testcases pass), but the trailer testcases do not work (it is an expected
  failure). In particular, the value received by the mock server doesn't match
  the expected value, and instead looks something like the following (using
  python array notation):

  `trailer[:]message[1:]`

  instead of:

  `message[:]trailer[:]`

  This makes me think there's a buffer overrun issue or problem with the
  offset somewhere in the sendfile(2) system call, but I need to do some
  other testing first to verify that the code is indeed sane, and my
  assumptions/code isn't buggy.

  The `sbytes_negative` testcases that check `sbytes` being set to an
  invalid value resulting in `EFAULT` fails today as the other change
  (which checks `copyout(9)`) has not been committed [1]. Thus, it
  should remain an expected failure (see bug 232210 for more details
  on this item).

  Next steps for testing sendfile(2):
  1. Fix the header/trailer testcases so that they pass.
  2. Setup if_tap interface and test with it, instead of using "localhost", per
     @asomers's suggestion.
  3. Handle short recv(2)'s in `server_cat(..)`.
  4. Add `SF_SYNC` support.
  5. Add some more negative tests outside the scope of the functional contract.

  PR: 		232210

  r343365:

  Unbreak the gcc build with sendfile_test after r343362

  gcc 8.x is more pedantic than clang 7.x with format strings and the tests
  passed `void*` variables while supplying `%s` (which is technically
  incorrect).

  Make the affected `void*` variables use `char*` storage instead to address
  this issue, as the compiler will upcast the values to `char*`.

  MFC with:	r343362

  r343367:

  Unbreak the build on architectures where size_t isn't synonymous with uintmax_t

  I should have used `%zu` instead of `%ju` with `size_t` types.

  MFC with:	r343362, r343365
  Pointyhat to:	ngie

  r343368:

  Fix up r343367

  I should have only changed the format qualifier with the `size_t` value,
  `length`, not the other [`off_t`] value, `dest_file_size`.

  MFC with:	r343362, r343365, r343367

  r343461:

  Fix reporting errors with `gai_strerror(..)`

  The return value (`err`) should be checked; not the `errno` value.

  PR:		235200
  MFC with:	r343362, r343365, r343367-r343368

  r343751:

  Avoid the DNS lookup for "localhost"

  ci.FreeBSD.org does not have access to a DNS resolver/network (unlike my test
  VM), so in order for the test to pass on the host, it needs to avoid the DNS
  lookup by using the numeric host address representation.

  PR:		235200
  MFC with:	r343362, r343365, r343367-r343368, r343461

  r344310:

  Make `server_cat(..)` handle short receives

  In short, the prior code was far too simplistic when it came to calling recv(2)
  and failed intermittently (or in the case of Jenkins, deterministically).

  Handle short recv(2)s by checking the return code and incrementing the window
  into the buffer by the number of received bytes. If the number of received
  bytes <= 0, then bail out of the loop, and test the total number of received
  bytes vs the expected number of bytes sent for equality, and base whether or
  not the test passes/fails on that fact.

  Remove the expected failure, now that the hdtr testcases deterministically pass
  on my host after this change [1].

  PR:		234809 [1], 235200

  Approved by:	emaste (mentor, implicit: MFC to ^/stable/11 in r344977)
  Differential Revision: https://reviews.freebsd.org/D19523

Changes:
_U  stable/12/
  stable/12/lib/libc/tests/sys/Makefile
  stable/12/lib/libc/tests/sys/sendfile_test.c
Comment 15 commit-hook freebsd_committer 2019-03-10 16:14:59 UTC
A commit references this bug:

Author: ngie
Date: Sun Mar 10 16:14:44 UTC 2019
New revision: 344978
URL: https://svnweb.freebsd.org/changeset/base/344978

Log:
  MFC r343362,r343365,r343367,r343368,r343461,r343751,r344310:

  r343362:

  Add [initial] functional tests for sendfile(2) as lib/libc/sys/sendfile

  These testcases exercise a number of functional requirements for sendfile(2).

  The testcases use IPv4 and IPv6 domain sockets with TCP, and were confirmed
  functional on UFS and ZFS. UDP address family sockets cannot be used per the
  sendfile(2) contract, thus using UDP sockets is outside the scope of
  testing the syscall in positive cases. As seen in
  `:s_negative_udp_socket_test`, UDP is used to test the sendfile(2) contract
  to ensure that EINVAL is returned by sendfile(2).

  The testcases added explicitly avoid testing out `SF_SYNC` due to the
  complexity of verifying that support. However, this is a good next logical
  item to verify.

  The `hdtr_positive*` testcases work to a certain degree (the header
  testcases pass), but the trailer testcases do not work (it is an expected
  failure). In particular, the value received by the mock server doesn't match
  the expected value, and instead looks something like the following (using
  python array notation):

  `trailer[:]message[1:]`

  instead of:

  `message[:]trailer[:]`

  This makes me think there's a buffer overrun issue or problem with the
  offset somewhere in the sendfile(2) system call, but I need to do some
  other testing first to verify that the code is indeed sane, and my
  assumptions/code isn't buggy.

  The `sbytes_negative` testcases that check `sbytes` being set to an
  invalid value resulting in `EFAULT` fails today as the other change
  (which checks `copyout(9)`) has not been committed [1]. Thus, it
  should remain an expected failure (see bug 232210 for more details
  on this item).

  Next steps for testing sendfile(2):
  1. Fix the header/trailer testcases so that they pass.
  2. Setup if_tap interface and test with it, instead of using "localhost", per
     @asomers's suggestion.
  3. Handle short recv(2)'s in `server_cat(..)`.
  4. Add `SF_SYNC` support.
  5. Add some more negative tests outside the scope of the functional contract.

  PR: 		232210

  r343365:

  Unbreak the gcc build with sendfile_test after r343362

  gcc 8.x is more pedantic than clang 7.x with format strings and the tests
  passed `void*` variables while supplying `%s` (which is technically
  incorrect).

  Make the affected `void*` variables use `char*` storage instead to address
  this issue, as the compiler will upcast the values to `char*`.

  MFC with:	r343362

  r343367:

  Unbreak the build on architectures where size_t isn't synonymous with uintmax_t

  I should have used `%zu` instead of `%ju` with `size_t` types.

  MFC with:	r343362, r343365
  Pointyhat to:	ngie

  r343368:

  Fix up r343367

  I should have only changed the format qualifier with the `size_t` value,
  `length`, not the other [`off_t`] value, `dest_file_size`.

  MFC with:	r343362, r343365, r343367

  r343461:

  Fix reporting errors with `gai_strerror(..)`

  The return value (`err`) should be checked; not the `errno` value.

  PR:		235200
  MFC with:	r343362, r343365, r343367-r343368

  r343751:

  Avoid the DNS lookup for "localhost"

  ci.FreeBSD.org does not have access to a DNS resolver/network (unlike my test
  VM), so in order for the test to pass on the host, it needs to avoid the DNS
  lookup by using the numeric host address representation.

  PR:		235200
  MFC with:	r343362, r343365, r343367-r343368, r343461

  r344310:

  Make `server_cat(..)` handle short receives

  In short, the prior code was far too simplistic when it came to calling recv(2)
  and failed intermittently (or in the case of Jenkins, deterministically).

  Handle short recv(2)s by checking the return code and incrementing the window
  into the buffer by the number of received bytes. If the number of received
  bytes <= 0, then bail out of the loop, and test the total number of received
  bytes vs the expected number of bytes sent for equality, and base whether or
  not the test passes/fails on that fact.

  Remove the expected failure, now that the hdtr testcases deterministically pass
  on my host after this change [1].

  PR:		234809 [1], 235200

  Approved by:	emaste (mentor, implicit: MFC to ^/stable/11 in r344977)
  Differential Revision: https://reviews.freebsd.org/D19523

Changes:
_U  stable/12/
  stable/12/lib/libc/tests/sys/Makefile
  stable/12/lib/libc/tests/sys/sendfile_test.c