See the commit log message for r350671. I can't seem to reproduce the hang by hand, but exceeding the limit does cause readelf to die with SIGPIPE, which is not right.
Kyle did some debugging and found that the issue comes from uipc_externalize() returning an error when it fails to allocate descriptors in the receiving process' fd table. I believe this results in an EMSGSIZE error from recvmsg().
(In reply to Mark Johnston from comment #1)
I think this might have been a red herring / faulty debugging somewhere -- my distilled test cases all seem to be doing the correct thing.
(In reply to Kyle Evans from comment #2)
"Correct" meaning that fileargs_open() fails with EMFILE?
Created attachment 206375 [details]
git(1) diff for some fixes
(In reply to Mark Johnston from comment #3)
In that context, I was trying to rewrite a smaller test case that reproduces the results -- that was a mistake.
There seem to be at least a couple of different problems here:
- fileargs_add_cache can fail, but not communicate that failure to its parent
- service_message should check nvlout for error state after taking a trip through service->s_command -- if nvlout happens to be an error state, cap_send_nvlist cannot succeed.
The SIGPIPE failure seems to originate from fileargs_add_cache hitting EMFILE at nvlist_add_nvlist(nvlout, fname, new), which then fails to travel because nvlout's in an error state.
Attached patch tries to clean some of this up, but doesn't do a very good job I don't think. Ultimately service_message should be catching any last-minute errors and trying to recover gracefully so we don't leave the other end in suspense, but the rest of the patch attempts to detect errors earlier and do something sane...