Bug 203900 - Can not increase above 32K max open files descriptor per PID
Summary: Can not increase above 32K max open files descriptor per PID
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 10.1-STABLE
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-bugs (Nobody)
URL:
Keywords: patch
Depends on:
Blocks:
 
Reported: 2015-10-20 17:57 UTC by Rizal Muhammad Nur
Modified: 2018-05-20 23:52 UTC (History)
6 users (show)

See Also:


Attachments
Generate many dummy files (1.70 KB, text/plain)
2015-10-20 17:59 UTC, Rizal Muhammad Nur
no flags Details
Load file which is generated by dummyFileGenerator.py (1.74 KB, text/plain)
2015-10-20 18:00 UTC, Rizal Muhammad Nur
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Rizal Muhammad Nur 2015-10-20 17:57:20 UTC
I run application which is sometimes opening >200k files at the same time. But, I get issue about "Too many open files". To test it, simply I create many dummy files and open it. Here the steps

Download source code to generate & load files: 
 * https://gist.github.com/juniorh/3b2fb0a80cddb8e407b3
 * https://gist.github.com/juniorh/ef9273911dee551f1048

# create files
> mkdir /tmp/1
> python dummyFileGenerator.py -r 1 -n 1000000 -d /tmp/1

# open all files
> python dummyFileLoader.py -r 1 -n 1000000 -d /tmp/1 

Then, after the script open 32766 files, error occures like:
=================================
...
...
open file number: 30763
retry openfile 32766
Traceback (most recent call last):
  File "dummyFileLoader.py", line 69, in <module>
  File "dummyFileLoader.py", line 54, in openfile
IOError: [Errno 24] Too many open files: '/tmp/1/00/00/7f/fd'
=================================

It seems FreeBSD only alow each PID to open up to 32K files. Changing kern.maxfiles and kern.maxfilesperproc is also not solved the issue. 

For additional, here my environtment:
=================================
# sysctl -a | grep maxfile
kern.maxfiles: 130305
kern.maxfilesperproc: 117270

# ulimit -a
cpu time               (seconds, -t)  unlimited
file size           (512-blocks, -f)  unlimited
data seg size           (kbytes, -d)  33554432
stack size              (kbytes, -s)  524288
core file size      (512-blocks, -c)  unlimited
max memory size         (kbytes, -m)  unlimited
locked memory           (kbytes, -l)  unlimited
max user processes              (-u)  8514
open files                      (-n)  117270
virtual mem size        (kbytes, -v)  unlimited
swap limit              (kbytes, -w)  unlimited
sbsize                   (bytes, -b)  unlimited
pseudo-terminals                (-p)  unlimited

# uname -a
FreeBSD fbd1 10.1-RELEASE FreeBSD 10.1-RELEASE #0 r274401: Tue Nov 11 21:02:49 UTC 2014     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
=================================
Comment 1 Rizal Muhammad Nur 2015-10-20 17:59:45 UTC
Created attachment 162256 [details]
Generate many dummy files
Comment 2 Rizal Muhammad Nur 2015-10-20 18:00:24 UTC
Created attachment 162257 [details]
Load file which is generated by dummyFileGenerator.py
Comment 3 Enji Cooper freebsd_committer 2015-10-20 18:52:22 UTC
What do these sysctls say (these are values obtained from 11-CURRENT, BTW)?

$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 87570
$ sysctl kern.maxfiles       
kern.maxfiles: 97306
Comment 4 Enji Cooper freebsd_committer 2015-10-20 18:53:26 UTC
(In reply to Garrett Cooper,425-314-3911 from comment #3)

Oh wait. I missed it in your reply above.

Let me do some quick digging into this issue.
Comment 5 Mateusz Guzik freebsd_committer 2015-10-20 18:56:47 UTC
The issue is with libc's FILE having fds limited to short int. There was a patch for this problem in the works, cc-ing the author.
Comment 6 Enji Cooper freebsd_committer 2015-10-20 18:57:48 UTC
Did we fix this at $work yet?
Comment 7 Enji Cooper freebsd_committer 2015-10-20 19:29:49 UTC
(In reply to Garrett Cooper,425-314-3911 from comment #6)

Answering my own question about include/stdio.h: yes, we did (and I remember some of the context for issues that we ran into). In particular our version of perl was doing silly things when we were compiling in a chroot.

bdrewery fixed it in r461505 internally, but we didn't add a compat layer for the change. Let me see if I can gather all of the info about this and at least post the patch.
Comment 8 John Baldwin freebsd_committer freebsd_triage 2015-10-20 19:41:18 UTC
I have a long-lived patch to expand _file inside of FILE from short to int.  However, there is some fallout in other ports (perl uses _file directly :() that needs to be fixed before it goes in.
Comment 9 John Baldwin freebsd_committer freebsd_triage 2015-10-20 19:43:33 UTC
The relevant patches (I should move these to a git branch) are at www.freebsd.org/~jhb/patches/stdio_file.patch as well as the stdio_file_exp*.patch follow up patches from when I was doing exp-runs on ports.

I had an earlier patch (stdio.patch) that made FILE completely opaque, but I think that broke too many things that expect FILE's definition to be available.  In particular, I think GNU autoconf tests try to compile something with a static FILE object to make sure the FILE type exists.
Comment 10 John Baldwin freebsd_committer freebsd_triage 2015-10-20 19:44:49 UTC
The patch for perl sources is at https://people.freebsd.org/~oshogbo/perl-fdclose.patch  All of the perl ports have to be patched before we can resume doing exp-runs of the source patch.
Comment 11 Andrey A. Chernov freebsd_committer 2015-10-20 22:28:04 UTC
(In reply to John Baldwin from comment #9)
In case you plan commit this patch, it will be nice to expand
short	_flags;
in the same way simultaneously. There is no room for additional flags left, so few code samples remains non-optimal time consuming.
Comment 12 Rizal Muhammad Nur 2015-10-21 05:06:05 UTC
(In reply to Garrett Cooper,425-314-3911 from comment #7)

Should I try this patch?

www.freebsd.org/~jhb/patches/stdio_file.patch

And may I know how to apply this patch?
Comment 13 John Baldwin freebsd_committer freebsd_triage 2015-10-21 17:36:08 UTC
I just pushed a new branch on github at github/bsdjhb/freebsd/stdio_file for HEAD (11, though might apply to 10 as well).  It might break building various ports, and I don't know yet if there are other ports besides Perl that abuse FILE internals that need to be adjusted.
Comment 14 John Baldwin freebsd_committer freebsd_triage 2015-10-21 17:37:12 UTC
(In reply to Andrey A. Chernov from comment #11)
Expanding _flags is nearly impossible due to all the inlined uses of it in various binaries.  We could add a new internal _flags2 field and be careful to never export it via inlined macros if we need more flags.
Comment 15 Andrey A. Chernov freebsd_committer 2015-10-22 01:34:30 UTC
(In reply to John Baldwin from comment #14)
Ok. It will be nice to add _flags2 in the same time. This is for deep internals which never supposed to be visible, f.e. to eliminate fcntl(F_GETFL) call in ftello().
Comment 16 John Baldwin freebsd_committer freebsd_triage 2015-10-22 20:20:01 UTC
You can add _flags2 at any time.  Long ago we changed our std* symbols so that growing FILE doesn't result in ABI breakage.
Comment 17 Rizal Muhammad Nur 2015-11-05 18:05:30 UTC
I tried to use patch from John but it didn't solve issue

Moreover, it seems the bug was reported since FreeBSD 7 as Bug #148581 but no more follow up. I just curious when it will be fixed. 

Hope someone has concern and kindly fix it
Comment 18 Eitan Adler freebsd_committer freebsd_triage 2018-05-20 23:52:35 UTC
For bugs matching the following conditions:
- Status == In Progress
- Assignee == "bugs@FreeBSD.org"
- Last Modified Year <= 2017

Do
- Set Status to "Open"