The following test code (thanks R Watson) tries to call open with O_CREAT | O_DIRECTORY, on a path that references a broken symlink. The symlink is removed, a file is created, and the call to open returns with an error ENOTDIR. There are similar strange behaviours. For example, O_CREAT | O_DIRECTORY | O_EXCL on a symlink to a directory will result in the symlink being removed, a file being created, and the call returning with an error. POSIX says "If O_EXCL and O_CREAT are set, and path names a symbolic link, open() shall fail and set errno to [EEXIST], regardless of the contents of the symbolic link. " ( http://pubs.opengroup.org/onlinepubs/009695399/functions/open.html ) but I'm not sure whether FreeBSD aims for compliance here. Even if FreeBSD doesn't aim for compliance, this behaviour possibly breaks a possible POSIX invariant: calls which return with errors do not alter the underlying file system state (here, we get an error, but a symlink is removed and a new file is created). #include <err.h> #include <errno.h> #include <fcntl.h> #include <stdio.h> #include <string.h> #include <unistd.h> int main(int argc, char *argv[]) { int fd; /* Setup. */ if (unlink("broken") <= 0 && errno != ENOENT) err(-1, "unlink(broken)"); if (unlink("target") <= 0 && errno != ENOENT) err(-1, "unlink(target)"); if (symlink("target", "broken") < 0) err(-1, "symlink(target, broken)"); /* Test. */ fd = open("broken", O_CREAT | O_DIRECTORY, 0600); if (fd >= 0) errx(-1, "open(broken, O_CREAT | O_DIRECTORY) - no error"); else printf("Error on open() was %s\n", strerror(errno));  } Thanks
Per rwatson' answers, the biggest issue there is the O_DIRECTORY | O_CREAT handling, which should not create files. Patch attached fixes the issue to me. Also, the behaviour of the patch is same as on Linux, where open("broken symlink", O_CREAT) causes creation of the symlink target.
Created attachment 160720 [details] The supposed fix, do not create anything for O_DIRECTORY
What is the behaviour with a symlink to a non-existing target (in an existing directory) with flags O_CREAT | O_DIRECTORY | O_EXCL? (POSIX seems fairly clear that this should not create the target, but as you note, other platforms are not necessarily compliant).
(In reply to Tom Ridge from comment #3) After the patch, for O_DIRECTORY, the target is never created, regardless of other flags.
(In reply to Konstantin Belousov from comment #4) That said, the error for open("broken symlink", O_CREAT | O_EXCL | O_DIRECTORY) is ENOENT.
I've been viewing the problem as a case of an invalid argument -- in which case EINVAL might be preferred in the latter O_CREAT | O_EXCL | O_DIRECTORY case -- although perhaps it is impractical.
(In reply to Robert Watson from comment #6) The case of EXCL|CREAT|DIR can be special-cased. I updated the patch. I considered disabling O_CREAT|O_DIR at all, but IMO it should work for the case of existing directories.
Created attachment 160727 [details] The supposed fix, do not create anything for O_DIRECTORY, also return EINVAL for O_EXCL.
This approach seems reasonable to me. I can see arguments for also returning EINVAL even if O_EXCL isn't expressed (i.e., that O_DIRECTORY | O_CREAT is never a valid combination) but I don't feel strongly about it. Maybe Tom takes a view, but I think getting this patch in now makes sense.
A commit references this bug: Author: kib Date: Wed Sep 9 19:31:09 UTC 2015 New revision: 287599 URL: https://svnweb.freebsd.org/changeset/base/287599 Log: For open("name", O_DIRECTORY | O_CREAT), do not try to create the named node, open(2) cannot create directories. But do allow the flag combination to succeed if the directory already exists. Declare the open("name", O_DIRECTORY | O_CREAT | O_EXCL) always invalid for the same reason, since open(2) cannot create directory. Note that there is an argument that O_DIRECTORY | O_CREAT should be invalid always, regardless of the target directory existence or O_EXCL. The current fix is conservative and allows the call to succeed in the situation where it succeeded before the patch. Reported by: Tom Ridge <freebsd@tom-ridge.com> Reviewed by: rwatson PR: 202892 Sponsored by: The FreeBSD Foundation MFC after: 1 week Changes: head/sys/kern/vfs_vnops.c
A commit references this bug: Author: kib Date: Wed Sep 16 04:23:09 UTC 2015 New revision: 287847 URL: https://svnweb.freebsd.org/changeset/base/287847 Log: MFC r287599: Correct handling of open("name", O_DIRECTORY | O_CREAT). PR: 202892 Changes: _U stable/10/ stable/10/sys/kern/vfs_vnops.c