According to POSIX standard (http://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html), open function should fail to open a symlink with ELOOP when it is called with O_NOFOLLOW flag. However, it returns EMLINK on FreeBSD. It seems it was changed by this old commit: https://svnweb.freebsd.org/changeset/base/35088 This problem causes programs using GIO API to fail to open symlinks for writing (bug 212572).
This question has come up before in https://lists.freebsd.org/pipermail/freebsd-fs/2011-March/010952.html. The error for O_NOFOLLOW is [EMLINK] because that allows distinguishing a symlink loop in the non-final components of the path from a symlink in the final component of the path. The implementation of cmp(1) (-h option) depends on this and contrib/xz takes advantage of it to avoid a race condition. Perhaps an Austin group bug should be opened about this. By the way, NetBSD returns [EFTYPE] in this case.
(In reply to Jilles Tjoelker from comment #1) IMO the frequency of the issue raised, and the need of the appearance of the carrier of the tribal knowledge who can explain the consequences of the change, make it more reasonable to put this reasoning into the open(2) page. Jilles, would you, please, formulate your explanation to either BUGS or STANDARDS section of the man page ? It would be very useful for now, and even if both us and SUSvX adjust the behaviour, a historic reference is useful for very long time.
A commit references this bug: Author: jilles Date: Tue Nov 22 22:30:55 UTC 2016 New revision: 309026 URL: https://svnweb.freebsd.org/changeset/base/309026 Log: open(2): Clarify non-POSIX error when opening a symlink with O_NOFOLLOW. We return [EMLINK] instead of [ELOOP] when trying to open a symlink with O_NOFOLLOW, so that the original case of [ELOOP] can be distinguished. Code like cmp -h and xz takes advantage of this. PR: 214633 Reviewed by: kib, imp MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D8586 Changes: head/lib/libc/sys/open.2
I think I can close this bug now because the man page fix is already committed.
A commit references this bug: Author: jilles Date: Sun Dec 4 15:55:59 UTC 2016 New revision: 309533 URL: https://svnweb.freebsd.org/changeset/base/309533 Log: MFC r309026: open(2): Clarify non-POSIX error when opening a symlink with O_NOFOLLOW. We return [EMLINK] instead of [ELOOP] when trying to open a symlink with O_NOFOLLOW, so that the original case of [ELOOP] can be distinguished. Code like cmp -h and xz takes advantage of this. PR: 214633 Changes: stable/11/lib/libc/sys/open.2
A commit references this bug: Author: jilles Date: Sun Dec 4 16:00:25 UTC 2016 New revision: 309536 URL: https://svnweb.freebsd.org/changeset/base/309536 Log: MFC r309026: open(2): Clarify non-POSIX error when opening a symlink with O_NOFOLLOW. We return [EMLINK] instead of [ELOOP] when trying to open a symlink with O_NOFOLLOW, so that the original case of [ELOOP] can be distinguished. Code like cmp -h and xz takes advantage of this. PR: 214633 Changes: _U stable/10/ stable/10/lib/libc/sys/open.2
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=ea6a6b63e1fd304e790c8ed7627caf5e3ba52bc7 commit ea6a6b63e1fd304e790c8ed7627caf5e3ba52bc7 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2024-01-26 15:19:04 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2024-01-26 15:36:32 +0000 open: make non-POSIX errno value more apparent In the errno list, add an explicit note and reference to the note in the STANDARDS section. When O_NOFOLLOW is specified and the target is a symbolic link FreeBSD sets errno to a value different than that specified by POSIX. Commit 295159dfa3ed added a note to this effect, but I missed it when reading through the list of errno values. PR: 214633 Reviewed by: markj Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D43618 lib/libc/sys/open.2 | 5 +++++ 1 file changed, 5 insertions(+)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=c3c75c142c0cb9fe9bf166e3416195bc59ad0e12 commit c3c75c142c0cb9fe9bf166e3416195bc59ad0e12 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2024-01-26 15:19:04 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2024-03-22 13:30:30 +0000 open: make non-POSIX errno value more apparent In the errno list, add an explicit note and reference to the note in the STANDARDS section. When O_NOFOLLOW is specified and the target is a symbolic link FreeBSD sets errno to a value different than that specified by POSIX. Commit 295159dfa3ed added a note to this effect, but I missed it when reading through the list of errno values. PR: 214633 Reviewed by: markj Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D43618 (cherry picked from commit ea6a6b63e1fd304e790c8ed7627caf5e3ba52bc7) lib/libc/sys/open.2 | 5 +++++ 1 file changed, 5 insertions(+)