Bug 184340 - PATH_MAX not interoperable with Linux
Summary: PATH_MAX not interoperable with Linux
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: CURRENT
Hardware: Any Any
: Normal Affects Some People
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-11-27 23:10 UTC by dcundiff
Modified: 2017-09-17 20:38 UTC (History)
5 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description dcundiff 2013-11-27 23:10:00 UTC
PATH_MAX in is not interoperable with Linux when transferring files with Rsync. Since file paths can be much longer on Linux(4096 bytes) Rsync will crash as it sets up a buffer based on PATH_MAX. Also as unicode use increases a PATH_MAX of 1024 would limit paths to 256 32-bit unicode characters. Using 4096 bytes unicode paths could be 1024 characters.

Sure Rsync could change the value in their source, however, there could be other programs that are used between FreeBSD and Linux that use the value and will have trouble.
 
This is defined in the kernel headers

/usr/src/sys/sys/syslimits.h:#define     PATH_MAX                 1024   /* max bytes in pathname */

Fix: 

Change PATH_MAX in kernel to 4096 from 1024. Should be harmless and will fix the issue in any program that uses PATH_MAX from the kernel headers. Also would allow longer 32-bit unicode paths.
How-To-Repeat: Copy a file from a Linux machine to a FreeBSD machine where the pathname is greater than 1024 characters.
Comment 1 Brooks Davis freebsd_committer 2013-11-27 23:26:17 UTC
On Wed, Nov 27, 2013 at 11:03:31PM +0000, David Cundiff wrote:
> Change PATH_MAX in kernel to 4096 from 1024. Should be harmless and will fix the issue in any program that uses PATH_MAX from the kernel headers. Also would allow longer 32-bit unicode paths.

Blindly changing PATH_MAX would be far from harmless.  It would bloat
many internal structures and break ABIs and thus could not be done on
a stable branch without quite a bit of work.  This is probalby worth
fixing for 11.0, ideally by removing the limit entierly as suggested by
POSIX.

-- Brooks
Comment 2 dcundiff 2013-11-28 01:21:40 UTC
On 11/27/2013 6:26 PM, Brooks Davis wrote:
> On Wed, Nov 27, 2013 at 11:03:31PM +0000, David Cundiff wrote:
>> Change PATH_MAX in kernel to 4096 from 1024. Should be harmless and will fix the issue in any program that uses PATH_MAX from the kernel headers. Also would allow longer 32-bit unicode paths.
> Blindly changing PATH_MAX would be far from harmless.  It would bloat
> many internal structures and break ABIs and thus could not be done on
> a stable branch without quite a bit of work.  This is probalby worth
> fixing for 11.0, ideally by removing the limit entierly as suggested by
> POSIX.
>
> -- Brooks

Now that you mention it both of those would be an issue(I am not much of 
a programmer). We did just adjust it in the kernel source and several 
structs needed to be made larger as well. We tested it, everything seems 
to work, but who knows what other things we don't use broke. The ABI 
change seems safe enough for my use of the OS as a backup system, 
probably not for others.

I figured I'd mention it as Linux went to 4096. I imagine their 
reasoning was for allowing larger unicode paths. While 1024 characters 
for a path seems pretty excessive, 256(with 4 byte characters) seems a 
bit short.

Dave
Comment 3 Niall Douglas 2016-05-17 16:35:01 UTC
The excessively short PATH_MAX on FreeBSD is causing problems with stacking encryption layers like encfs on top of FreeBSD filing systems. Please see:

https://lists.freebsd.org/pipermail/freebsd-fs/2016-May/023250.html

Can we get motion on getting this limit raised please? Even 4096 bytes would make a big improvement.

Niall
Comment 4 Ben RUBSON 2016-05-17 16:55:55 UTC
Hello,

I'm also facing this issue where PATH_MAX is too small (FreeBSD 10.3).

Is there any plan to :
- increase this value to a higher value ?
- make this value user-tunable ?
- remove this limitation ?

As a workaround, where can we change it by ourselves ?
find /usr/src/ -type f -exec grep -i "define.*path.*1024" {} \; -ls
There are a lot of defines which are related to paths and hard-coded to 1024, so could be quite tricky to make this change without side-effects.

Thank you very much !

Ben
Comment 5 Jilles Tjoelker freebsd_committer 2016-05-22 18:37:52 UTC
The limit cannot be removed entirely without a severe rework because subsystems like ktrace and audit need a copy of the pathname that the user cannot modify concurrently.

The kernel rather likes allocating PATH_MAX or MAXPATHLEN sized buffers in general. This is already wasteful with PATH_MAX=1024 and even more so with 4096. Ideally, the majority of short pathnames would not waste so much memory.

For a local change I would change the one in sys/sys/syslimits.h. Then recompile everything including packages. There are a few nasty APIs like realpath() that implicitly depend on PATH_MAX.

Applications that insist on it can use longer pathnames by passing only short segments to system calls and using openat(2) and other *at functions. For example, find and rm from the base system do this (provided symlinks are not being followed, the current directory can be opened for reading and the pathnames passed to the utility themselves are not too long).
Comment 6 Niall Douglas 2016-05-22 22:03:04 UTC
(In reply to Jilles Tjoelker from comment #5)

I appreciate the detail in your reply. However, this is not a code development problem - as you mention, openat() is the correct solution to programming long paths. Rather it's a user problem - we are stuck with software written by others which was designed around a Linux PATH_MAX of 4096. Being that we cannot rewrite all this software to not be so stupid, it generates substantial gotchas for end users.

In terms of how to refactor the BSD kernel to handle this, I believe NT simply uses dynamic memory allocation for all paths, and therefore the 64Kb path limit is tractable except when frequently modifying paths as the win32 layer likes to do. An ideal solution for BSD would be some sort of variant storage which could be either 256 bytes of path or a dynamic memory allocation to a path. Perhaps a zero length path could mean "pointer to a dynamically allocated path follows", so something like:

union
{
  char path[256];        // for paths < 255
  struct
  {
    char _zero;             // lowest byte in memory is zero
    char _magic;            // magic marker byte to detect unioned path
    unsigned short length;  // length of path
    char *path;             // dynamically allocated pointer to path
  };
};

Anyway, I'm no expert in the BSD kernel, but I would find it nice to not have to install ZFS on Linux just so I can zfs snapshot the volume onto FreeBSD as my sole method of working around the BSD PATH_MAX limit.
Comment 7 Ben RUBSON 2017-09-17 20:38:22 UTC
On-going discussion & dev :
https://lists.freebsd.org/pipermail/freebsd-current/2017-September/066908.html