Bug 223413

Summary: writev() fails repeatedly with EAGAIN after sendfile() fails with EAGAIN
Product: Base System Reporter: YOUZHONG YANG <youzhong>
Component: kernAssignee: freebsd-bugs (Nobody) <bugs>
Status: New ---    
Severity: Affects Only Me CC: chris
Priority: ---    
Version: 11.1-RELEASE   
Hardware: Any   
OS: Any   

Description YOUZHONG YANG 2017-11-04 05:21:29 UTC
Recently I started evaluating FreeBSD, including the Samba software.

Unfortunately once I set 'use sendfile = yes' in Samba, and then tried to launch an application from the samba share, it hung. The application is not a small one, it loads thousands of files from the server to Windows client.

It appears the sendfile() failed with EAGAIN:

# dtrace -n 'syscall::sendfile:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}'
dtrace: description 'syscall::sendfile:return ' matched 1 probe
CPU     ID                    FUNCTION:NAME
 23  64274                  sendfile:return
              libc.so.7`__sys_sendfile+0xa
              libsmbd-base-samba4.so`vfswrap_sendfile+0x127
              libsmbd-base-samba4.so`smb_vfs_call_sendfile+0x6d
              libsmbd-base-samba4.so`smb2_sendfile_send_data+0x8e
              libtalloc.so.2.1.9`_tc_free_internal+0x152
              libtalloc.so.2.1.9`_tc_free_children_internal+0xac
              libtalloc.so.2.1.9`_tc_free_internal+0x331
              libtalloc.so.2.1.9`_talloc_free_internal+0xb2
              libtalloc.so.2.1.9`_talloc_free+0x114
              libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
              libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
              libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
              libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
              libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
              libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
              libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
              libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
              libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
              libsmbd-base-samba4.so`smbd_smb2_connection_handler+0x46
              libtevent.so.0.9.31`poll_event_loop_poll+0x75c
errno = 35

then Samba process 'smbd' tried to send data using writev(), but writev() failed repeatedly with EAGAIN:

# dtrace -n 'syscall::writev:return /arg1 == -1/ {ustack(); printf("errno = %d", errno);}'
 16  63748                    writev:return
              libc.so.7`_writev+0xa
              libsys-rw-samba4.so`sys_writev+0x21
              libsys-rw-samba4.so`write_data_iov+0x88
              libsys-rw-samba4.so`write_data+0x39
              libsmbd-base-samba4.so`fake_sendfile+0x16e
              libsmbd-base-samba4.so`smb2_sendfile_send_data+0x61a
              libtalloc.so.2.1.9`_tc_free_internal+0x152
              libtalloc.so.2.1.9`_tc_free_children_internal+0xac
              libtalloc.so.2.1.9`_tc_free_internal+0x331
              libtalloc.so.2.1.9`_talloc_free_internal+0xb2
              libtalloc.so.2.1.9`_talloc_free+0x114
              libsmbd-base-samba4.so`smbd_smb2_flush_send_queue+0x37a
              libsmbd-base-samba4.so`smbd_smb2_request_reply+0x1886
              libsmbd-base-samba4.so`smbd_smb2_request_done_ex+0x62b
              libsmbd-base-samba4.so`smbd_smb2_request_read_done+0x32f
              libtevent.so.0.9.31`_tevent_req_notify_callback+0x6c
              libsmbd-base-samba4.so`smbd_smb2_request_pending_queue+0x3f
              libsmbd-base-samba4.so`smbd_smb2_request_process_read+0x4ef
              libsmbd-base-samba4.so`smbd_smb2_request_dispatch+0x1f03
              libsmbd-base-samba4.so`smbd_smb2_io_handler+0x8e2
errno = 35

# dtrace -n 'syscall::sendfile:return { @["sendfile", execname, errno] = count();}' -n 'syscall::writev:return { @["writev", execname, errno] = count();}'
  writev          syslogd         0                1
  sendfile        smbd           35                3
  writev          unbound         0                5
  writev          winbindd        0               33
  writev          smbd            0             1725
  sendfile        smbd            0             5388
  writev          smbd           35        276979542

The FreeBSD version is 11.1-RELEASE, and Samba is 4.6.8. I don't know how to reproduce it without using Samba.

I can do dtrace. Please let me know what can be done next to find out the root cause of the issue. By the way, I am not familiar with the FreeBSD src code.