I run application which is sometimes opening >200k files at the same time. But, I get issue about "Too many open files". To test it, simply I create many dummy files and open it. Here the steps
Download source code to generate & load files:
# create files
> mkdir /tmp/1
> python dummyFileGenerator.py -r 1 -n 1000000 -d /tmp/1
# open all files
> python dummyFileLoader.py -r 1 -n 1000000 -d /tmp/1
Then, after the script open 32766 files, error occures like:
open file number: 30763
retry openfile 32766
Traceback (most recent call last):
File "dummyFileLoader.py", line 69, in <module>
File "dummyFileLoader.py", line 54, in openfile
IOError: [Errno 24] Too many open files: '/tmp/1/00/00/7f/fd'
It seems FreeBSD only alow each PID to open up to 32K files. Changing kern.maxfiles and kern.maxfilesperproc is also not solved the issue.
For additional, here my environtment:
# sysctl -a | grep maxfile
# ulimit -a
cpu time (seconds, -t) unlimited
file size (512-blocks, -f) unlimited
data seg size (kbytes, -d) 33554432
stack size (kbytes, -s) 524288
core file size (512-blocks, -c) unlimited
max memory size (kbytes, -m) unlimited
locked memory (kbytes, -l) unlimited
max user processes (-u) 8514
open files (-n) 117270
virtual mem size (kbytes, -v) unlimited
swap limit (kbytes, -w) unlimited
sbsize (bytes, -b) unlimited
pseudo-terminals (-p) unlimited
# uname -a
FreeBSD fbd1 10.1-RELEASE FreeBSD 10.1-RELEASE #0 r274401: Tue Nov 11 21:02:49 UTC 2014 email@example.com:/usr/obj/usr/src/sys/GENERIC amd64
Created attachment 162256 [details]
Generate many dummy files
Created attachment 162257 [details]
Load file which is generated by dummyFileGenerator.py
What do these sysctls say (these are values obtained from 11-CURRENT, BTW)?
$ sysctl kern.maxfilesperproc
$ sysctl kern.maxfiles
(In reply to Garrett Cooper,425-314-3911 from comment #3)
Oh wait. I missed it in your reply above.
Let me do some quick digging into this issue.
The issue is with libc's FILE having fds limited to short int. There was a patch for this problem in the works, cc-ing the author.
Did we fix this at $work yet?
(In reply to Garrett Cooper,425-314-3911 from comment #6)
Answering my own question about include/stdio.h: yes, we did (and I remember some of the context for issues that we ran into). In particular our version of perl was doing silly things when we were compiling in a chroot.
bdrewery fixed it in r461505 internally, but we didn't add a compat layer for the change. Let me see if I can gather all of the info about this and at least post the patch.
I have a long-lived patch to expand _file inside of FILE from short to int. However, there is some fallout in other ports (perl uses _file directly :() that needs to be fixed before it goes in.
The relevant patches (I should move these to a git branch) are at www.freebsd.org/~jhb/patches/stdio_file.patch as well as the stdio_file_exp*.patch follow up patches from when I was doing exp-runs on ports.
I had an earlier patch (stdio.patch) that made FILE completely opaque, but I think that broke too many things that expect FILE's definition to be available. In particular, I think GNU autoconf tests try to compile something with a static FILE object to make sure the FILE type exists.
The patch for perl sources is at https://people.freebsd.org/~oshogbo/perl-fdclose.patch All of the perl ports have to be patched before we can resume doing exp-runs of the source patch.
(In reply to John Baldwin from comment #9)
In case you plan commit this patch, it will be nice to expand
in the same way simultaneously. There is no room for additional flags left, so few code samples remains non-optimal time consuming.
(In reply to Garrett Cooper,425-314-3911 from comment #7)
Should I try this patch?
And may I know how to apply this patch?
I just pushed a new branch on github at github/bsdjhb/freebsd/stdio_file for HEAD (11, though might apply to 10 as well). It might break building various ports, and I don't know yet if there are other ports besides Perl that abuse FILE internals that need to be adjusted.
(In reply to Andrey A. Chernov from comment #11)
Expanding _flags is nearly impossible due to all the inlined uses of it in various binaries. We could add a new internal _flags2 field and be careful to never export it via inlined macros if we need more flags.
(In reply to John Baldwin from comment #14)
Ok. It will be nice to add _flags2 in the same time. This is for deep internals which never supposed to be visible, f.e. to eliminate fcntl(F_GETFL) call in ftello().
You can add _flags2 at any time. Long ago we changed our std* symbols so that growing FILE doesn't result in ABI breakage.
I tried to use patch from John but it didn't solve issue
Moreover, it seems the bug was reported since FreeBSD 7 as Bug #148581 but no more follow up. I just curious when it will be fixed.
Hope someone has concern and kindly fix it
For bugs matching the following conditions:
- Status == In Progress
- Assignee == "bugs@FreeBSD.org"
- Last Modified Year <= 2017
- Set Status to "Open"