Hi all, I just upgraded my server (let's call it "server A") from FreeBSD 13.1 to FreeBSD 13.2. But now my system produces input/output errors when I try to make hard links in an SSHFS mounted directory. # Summary Example to reproduce on server A: ``` $ cd ~/Maildir # this is the SSHFS mount, on "server B" $ echo "foo" > foo $ ln foo bar ln: bar: Input/output error ``` It seems to create a new file with a new inode instead (see first column): ``` 771 -rw-r--r-- 1 user user 4 May 3 10:58 /home/user/Maildir/bar 774 -rw-r--r-- 1 user user 4 May 3 10:58 /home/user/Maildir/foo ``` Server A is using FreeBSD 13.2, on which I executed the above commands. Server B is using TrueNAS-13.0-U4 (based on FreeBSD 13.1-RELEASE-p7), on which the remote directory exists I'm mounting on the client. This is the /etc/fstab entry on server A, which worked using FreeBSD 13.1: ``` user@server-b.lan:/mnt/tank/home/user/Maildir /home/user/Maildir fusefs rw,allow_other,mountprog=/usr/local/bin/sshfs,port=1116,IdentityFile=/root/.ssh/id_ed25519,reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,late 0 0 ``` Creating hard links works on both the servers' own filesystems (ZFS), but not anymore when I upgraded server A to FreeBSD 13.2, using the same SSHFS mounted directory. ## Setup SSHFS is installed on my system: ``` $ pkg info | grep fusefs-sshfs fusefs-sshfs-3.7.3_1 Mount remote directories over ssh ``` I'm using the following entry in /boot/loader.conf: ``` $ grep fuse /boot/loader.conf fusefs_load="YES" ``` I can also see the FUSE kernel module being loaded (at boot): ``` $ kldstat | grep fuse 2 1 0xffffffff8213f000 262f0 fusefs.ko ``` And the SSHFS mount is still being mounted (at boot): ``` $ mount | grep home zroot/usr/home on /usr/home (zfs, local, noatime, nfsv4acls) user@server-b.lan:/mnt/tank/home/user/Maildir on /usr/home/user/Maildir (fusefs.sshfs) ``` Please note that /home is of course a symlink to /usr/home: ``` $ ls -ld /home lrwxr-xr-x 1 root wheel 8 Dec 5 12:58 /home@ -> usr/home ``` ## Background I found out the above when my MDA (Dovecot) on server A produced the following errors in the mail logs: ``` May 3 10:35:08 mail dovecot[3349]: imap(user)<73981><YPoBCcP6eJXAqLIB>: Error: Mailbox INBOX: link(/home/user/Maildir/cur/1683098837.M45130P81809.mail,S=44503,W=45217:2,, /home/user/Maildir/.Trash/tmp/1683102908.M640256P73981.mail) failed: Input/output error ``` These errors popped up after the upgrade to FreeBSD 13.2, and it is making it impossible to move mails around in my MUAs (e.g. Thunderbird). Interestingly, my MTA (Postfix) can still receive and store mails on the SSHFS mount. No problem at all. So it seems only Dovecot is hard linking files, which is now causing input/output errors when I upgraded to FreeBSD 13.2. Also, the Dovecot user still has permissions to perform read and write operations on the SSHFS mount. But hard linking is still troublesome: ``` # sudo -u dovecot touch /home/user/Maildir/foo # sudo -u dovecot rm /home/user/Maildir/foo # sudo -u dovecot touch /home/user/Maildir/foo # sudo -u dovecot ln /home/user/Maildir/foo /home/user/Maildir/bar ln: /home/user/Maildir/bar: Input/output error ``` Can anyone reproduce this by trying to make a hard link on an SSHFS mount using FreeBSD 13.2? Thanks in advance! P.S. I copied the above from my original post on [Reddit](https://www.reddit.com/r/freebsd/comments/136ewci/upgraded_to_freebsd_132_using_sshfs_hard_linking/). I have also since given up on SSHFS, and replaced it with NFS over WireGuard.