Bug 222632 - connect(2) not available in capability mode
Summary: connect(2) not available in capability mode
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Many People
Assignee: freebsd-bugs (Nobody)
URL: https://github.com/HardenedBSD/harden...
Keywords: needs-qa, patch, security
Depends on:
Reported: 2017-09-26 22:17 UTC by Shawn Webb
Modified: 2019-04-08 15:34 UTC (History)
5 users (show)

See Also:
koobs: mfc-stable11?
koobs: mfc-stable10?

Disallow connectat/bindat AT_FDCWD in capabilities mode (11.09 KB, patch)
2017-11-12 15:42 UTC, Jan Kokemüller
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Shawn Webb 2017-09-26 22:17:57 UTC
The CAP_CONNECT capability is useless without the connect(2) syscall being annotated with SYF_CAPENABLED. This means that Capsicumized applications expecting to be able to call connect(2) on a socket cannot.
Comment 2 Shawn Webb 2017-09-26 23:21:30 UTC
Based on research done by Robert Watson, which isn't referenced anywhere in FreeBSD's official Capsicum documentation, connect(2) isn't ready to be Capsicumized.

Note that having CAP_CONNECT documented and referenced with CAP_SOCK_CLIENT in FreeBSD's sys/capsicum.h leads one to believe connect(2) should be available in capabilities mode. This is in addition to the rights(4) manpage.

As such, I've reverted the referenced commit.

So that leads one to ask the question: how does one properly Capsicumize applications that call connect(2) on an on-demand basis?
Comment 3 Shawn Webb 2017-09-26 23:22:36 UTC
Robert Watson's whitepaper which outlines why connect(2) currently does not support Capsicum can be found here: https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-818.pdf
Comment 4 Brooks Davis freebsd_committer 2017-09-27 00:01:01 UTC
You need a helper process of some sort with ambient authority that you can request sockets from over.  If you're especially paranoid, you might create a small server program that you exec over your large program to reduce attach surface.
Comment 5 Shawn Webb 2017-09-27 00:04:58 UTC
(In reply to Brooks Davis from comment #4)
That's essentially what I'm doing. I'm writing a non-capsicumized child process that handles socket and file descriptor creation in addition to file system access calls (like unlink, rename, etc.). I'm using file descriptor passing over a socketpair between the non-Capsicumized child process and its Capsicumized parent.

It seems, though, that CAP_CONNECT shouldn't even exist if connect(2) doesn't support Capsicum. All of FreeBSD's official documentation leads developers like me to believe that calling connect(2) within a capabilities enabled process is possible.

It seems non-intuitive that I have to wrap connect(2) via the file descriptor passing mechanism I've designed, especially with CAP_CONNECT exists.
Comment 6 Brooks Davis freebsd_committer 2017-09-27 00:31:27 UTC
Hmm, seems to be some confusion in the implementation.  kern_connectat() checks CAP_CONNECT despite a CAP_CONNECTAT existing.  SCTP also uses CAP_CONNECT in ways that aren't immediately obvious to me.

It's not clear what the right thing is for connectat().
Comment 7 Shawn Webb 2017-09-27 00:59:08 UTC
(In reply to Brooks Davis from comment #6)
In my case, I'm not trying to use connectat(2). Rather, I'm trying to use connect(2).
Comment 8 Domagoj Stolfa 2017-09-27 13:10:54 UTC
I've not investigated this too much, but I can see why in some sense, connect(2) could be seen as gaining an additional capability and could have issues. The connectat(2) system call is the one that works with PF_LOCAL sockets, so you could use that instead for a Unix socket.

There does seem to be a discrepancy across the documentation and what is actually implemented, as CAP_CONNECT explicitly states that connect(2) is allowed and the check is in place in the actual implementation, but connect is not present in sys/kern/capabilities.conf, resulting in ECAPMODE.
Comment 9 Shawn Webb 2017-09-27 17:31:25 UTC
Turns out that Tor will at some point call SSL_connect from libssl. Since connect(2) isn't allowed in capmode, my efforts to apply Capsicum to Tor are in vain.

What would be the best way to handle this? I definitely want any network communication libssl might do to be Capsicumized. So calling SSL_connect from the non-capmode child process won't work.
Comment 10 Shawn Webb 2017-09-28 19:10:12 UTC
It turns out that SSL_connect doesn't call connect(2). SSL_connect assumes an already-connected socket. Its job is to set up the SSL/TLS side of things. Which is really good news.

I'm able to tell the non-capmode child process to perform the connect on behalf of the capmode parent. This is a doable workaround. However, long-term, we should figure out how to be able to call connect(2) in capmode. Until then, we should explicitly tell users that connect(2) is unavailable in capmode and that CAP_CONNECT is meaningless.
Comment 11 Ed Maste freebsd_committer 2017-10-02 09:00:58 UTC
(In reply to Shawn Webb from comment #10)
> However, long-term, we should figure out how to be able to call connect(2) in capmode.

As a kernel interface that provides access to global namespaces connect(2) cannot be called in capability mode. I looked at the capsicum(4) man page again now, and it certainly seems we need a more thorough explanation in there. This statement alludes to connect(2) not being available:

> capability mode
>   A process mode, entered by invoking cap_enter(2), in which access
>   to global OS namespaces (such as the file system and PID names‐
>   paces) is restricted

but it's certainly not a sufficient guide for refactoring existing applications for applying Capsicum.
Comment 12 Ed Maste freebsd_committer 2017-10-12 17:57:38 UTC
> There does seem to be a discrepancy across the documentation and what is
> actually implemented, as CAP_CONNECT explicitly states that connect(2) is
> allowed and the check is in place in the actual implementation, but connect
> is not present in sys/kern/capabilities.conf, resulting in ECAPMODE.

Indeed, it appears there is a discrepancy between what's implemented, what is documented, and what we believe the expected behaviour should be.

Descriptions from sys/capsicum.h:

/* Allows for connect(2). */
#define CAP_CONNECT             CAPRIGHT(0, 0x0000000080000000ULL)
/* Allows for connectat(2) on a directory descriptor. */
#define CAP_CONNECTAT           (CAP_LOOKUP | 0x0000010000000000ULL)


And reference "Declare more capability method rights" in r224255 and "Capsicum overhaul" in r247602.

Thus it seems the original design intent was to allow connect(2) in capability mode subject to CAP_CONNECT on the socket, but it was never added to capabilities.conf. With later refinement on thoughts on ambient authority connect() does not fit into that model and thus would not be allowed in capability mode.
Comment 13 Jan Kokemüller 2017-10-15 15:06:19 UTC
I've noticed that connectat(AT_FDCWD, ...) will even work on AF_INET sockets created in capabilities mode. Surely this is not intended to work? Am I missing something? bindat(AT_FDCWD, ...) also works.

	if (cap_enter() < 0) {
		err(1, "cap_enter");

	int sock = socket(AF_INET, SOCK_STREAM, 0);
	if (sock < 0) {
		err(1, "socket");

	struct sockaddr_in sin;
	sin.sin_family = AF_INET;
	sin.sin_port = htons(22);
	sin.sin_addr.s_addr = htonl(0x7F000001);

	if (connectat(AT_FDCWD, sock, (struct sockaddr *)(&sin),
	        sizeof(struct sockaddr_in)) < 0) {
		err(1, "connect");
Comment 14 Kubilay Kocak freebsd_committer freebsd_triage 2017-10-24 02:10:14 UTC
Thanks Shawn.

If you could attach the upstream commit as a patch here that would great
Comment 15 Ed Maste freebsd_committer 2017-10-24 02:12:40 UTC
(In reply to Kubilay Kocak from comment #14)
Which upstream commit are you referring to?
Comment 16 Kubilay Kocak freebsd_committer freebsd_triage 2017-10-24 09:54:10 UTC
(In reply to Ed Maste from comment #15)

Set in the URL field in this issue and Shawn referred to it in comment 1

I understand based on the discussion that it may or may not be the final set of changes required, but proposed changes are always preferred as attachments (unless they are in reviews, with the review url referenced here)
Comment 17 Shawn Webb 2017-10-24 13:01:15 UTC
(In reply to Kubilay Kocak from comment #16)
That particular change was reverted once I read more of Robert Watson's papers on Capsicum, in particular detailing why connect(2) was deliberately not allowed to be called in capmode. It would be great if the FreeBSD documentation reflected that.
Comment 18 Jan Kokemüller 2017-11-12 15:42:34 UTC
Created attachment 187942 [details]
Disallow connectat/bindat AT_FDCWD in capabilities mode

Here is a patch that disables connectat/bindat in capabilities mode when called with the AT_FDCWD parameter.

It also:
 - adds documentation that connect and connectat(AT_FDCWD,...) are equivalent and therefore connectat is not restricted to AF_UNIX sockets. Same for bindat.
 - adds documentation that CAP_BIND and CAP_CONNECT are useless in cap mode
 - adds some tests

Maybe there is still a use case for CAP_CONNECT/CAP_BIND: I think those rights can be used to lock down a raw, privileged socket created by a helper process.
Comment 19 Jan Kokemüller 2018-02-24 13:21:14 UTC
Has anybody had a chance to look at this yet (disabling connectat/bindat in capabilities mode when called with AT_FDCWD)? Should I open a new bug report?

Right now it's possible to communicate with other peers when in capabilities mode without any privileged helper processes.
Comment 20 Brooks Davis freebsd_committer 2018-02-25 18:28:36 UTC
(In reply to Jan Kokemüller from comment #19)

This patch did get buried.  I've looked it over and it seems correct, but I'm not a capsicum expert.  It might help to post to patch to reviews.freebsd.org.
Comment 21 Ed Maste freebsd_committer 2018-02-26 02:02:09 UTC
(In reply to Jan Kokemüller from comment #18)

* There's a capsicum-test project where this test could also be added. That said, until or unless we import capsicum-test into FreeBSD or are able to run it automatically I think it's beneficial to have a test in our own infrastructure, as you've done. https://github.com/google/capsicum-test

* Putting the patch in Phabricator might garner more interest, but it's been discovered now anyhow.

* I think the patch is good.
Comment 22 Jan Kokemüller 2018-04-28 06:12:51 UTC
Thank you for having a look!
I've rebased the patch to current head as jhb also added a test for capsicum in the meantime and the Makefile was conflicting.

I've posted a review here:
Comment 23 commit-hook freebsd_committer 2018-04-30 17:16:58 UTC
A commit references this bug:

Author: emaste
Date: Mon Apr 30 17:16:18 UTC 2018
New revision: 333119
URL: https://svnweb.freebsd.org/changeset/base/333119

  Clarify bindat/connectat use with AT_FDCWD

  Discovered during investigation into the PR - the description of
  AT_FDCWD was somewhat confusing.

  PR:		222632
  Submitted by:	Jan Kokem?ller <jan.kokemueller@gmail.com>
  MFC after:	1 week

Comment 24 commit-hook freebsd_committer 2018-04-30 17:31:12 UTC
A commit references this bug:

Author: emaste
Date: Mon Apr 30 17:31:07 UTC 2018
New revision: 333120
URL: https://svnweb.freebsd.org/changeset/base/333120

  Disable connectat/bindat with AT_FDCWD in capmode

  Previously it was possible to connect a socket (which had the
  CAP_CONNECT right) by calling "connectat(AT_FDCWD, ...)" even in
  capabilties mode.  This combination should be treated the same as a call
  to connect (i.e. forbidden in capabilities mode).  Similarly for bindat.

  Disable connectat/bindat with AT_FDCWD in capabilities mode, fix up the
  documentation and add tests.

  PR:		222632
  Submitted by:	Jan Kokem?ller <jan.kokemueller@gmail.com>
  Reviewed by:	Domagoj Stolfa
  MFC after:	1 week
  Relnotes:	Yes
  Differential Revision:	https://reviews.freebsd.org/D15221

Comment 25 Ed Maste freebsd_committer 2018-04-30 17:32:17 UTC
I think we still need to revisit CAP_BIND / CAP_CONNECT, but this should now be consistent and behave as expected.
Comment 26 Ed Maste freebsd_committer 2018-05-05 01:19:42 UTC
I'd like to see the new test also added to the capsicum-test project, https://github.com/google/capsicum-test

Jan, are you interested in adapting the test and submitting it as a pull request there?
Comment 27 Jan Kokemüller 2018-05-09 10:57:56 UTC
This is a good idea!
However, I'm not interested in signing the Google CLA at the moment.

One thing to be aware of: the test depends on some FreeBSD implementation details to see if the syscall was blocked by capsicum or the socket layer. The details could be different under Linux.
Comment 28 Ed Maste freebsd_committer 2019-04-08 15:34:51 UTC
Enji what do you thank about adding the test to capsicum-test?