When using threads, the threads library (/usr/src/lib/libc_r/uthread/uthread_fd.c) sets O_NONBLOCK for each new opened file and fails to emulate the blocking write(2) correctly. This affects eg. when a program opens /dev/stdout for writing and thje reader process is slower than the writer. The writer gets EAGAIN error which shouldn't happen when the open call didn't specify O_NONBLOCK flag. How-To-Repeat: Run the following program (compiled with gcc -pthread -o writer writer.c) and feed its output to the shell script: ./writer | sh reader.sh reader.sh: ---------------------------------------- #! /bin/sh while : do dd of=/dev/null bs=1k count=1 sleep 1 done ---------------------------------------- writer.c ---------------------------------------- #include <stdio.h> #include <errno.h> #include <fcntl.h> main(int argc, char **argv) { char buf[1024]; int fd; int i; int total = 0; fd = open("/dev/stdout",O_WRONLY|O_CREAT|O_TRUNC,0666); if (fd == -1) { int e; e = errno; fprintf(stderr,"open, errno: %d\n",e); exit(1); } for(i = 0; i < sizeof(buf); i++) buf[i] = 'a' + (i % 26); for(;;) { int n; n = write(fd,buf,sizeof(buf)); if (n > 0) total += n; if (n != sizeof(buf)) { int e; e = errno; fprintf(stderr,"write, errno: %d\n",e); fprintf(stderr,"Total bytes written %d\n",total); exit(1); } } } ---------------------------------------- Run the following program (compiled with gcc -pthread -o writer writer.c) and feed its output to the shell script: ./writer | sh reader.sh reader.sh: ---------------------------------------- #! /bin/sh while : do dd of=/dev/null bs=1k count=1 sleep 1 done ---------------------------------------- writer.c ---------------------------------------- #include <stdio.h> #include <errno.h> #include <fcntl.h> main(int argc, char **argv) { char buf[1024]; int fd; int i; int total = 0; fd = open("/dev/stdout",O_WRONLY|O_CREAT|O_TRUNC,0666); if (fd == -1) { int e; e = errno; fprintf(stderr,"open, errno: %d\n",e); exit(1); } for(i = 0; i < sizeof(buf); i++) buf[i] = 'a' + (i % 26); for(;;) { int n; n = write(fd,buf,sizeof(buf)); if (n > 0) total += n; if (n != sizeof(buf)) { int e; e = errno; fprintf(stderr,"write, errno: %d\n",e); fprintf(stderr,"Total bytes written %d\n",total); exit(1); } } } ---------------------------------------- Run the following program (compiled with gcc -pthread -o writer writer.c) and feed its output to the shell script: ./writer | sh reader.sh reader.sh: ---------------------------------------- #! /bin/sh while : do dd of=/dev/null bs=1k count=1 sleep 1 done ---------------------------------------- writer.c ---------------------------------------- #include <stdio.h> #include <errno.h> #include <fcntl.h> main(int argc, char **argv) { char buf[1024]; int fd; int i; int total = 0; fd = open("/dev/stdout",O_WRONLY|O_CREAT|O_TRUNC,0666); if (fd == -1) { int e; e = errno; fprintf(stderr,"open, errno: %d\n",e); exit(1); } for(i = 0; i < sizeof(buf); i++) buf[i] = 'a' + (i % 26); for(;;) { int n; n = write(fd,buf,sizeof(buf)); if (n > 0) total += n; if (n != sizeof(buf)) { int e; e = errno; fprintf(stderr,"write, errno: %d\n",e); fprintf(stderr,"Total bytes written %d\n",total); exit(1); } } } ----------------------------------------
This problem doesn't occur for me on 4.6-stable. Does it still occur for you? If not, let me know and I'll close this bug. Thanks, -Archie __________________________________________________________________________ Archie Cobbs * Packet Design * http://www.packetdesign.com
I just tried it in 4.6 Release and 4.6.2 and it still occurs. This bug makes it for example quite impossible to use /dev/stdout in python scripts if python has been compiled with threads support. Are you sure that you compiled the writer.c with: gcc -pthread -o writer writer.c What happens is: alo@toivoton 123: ./writer | sh reader.sh write, errno: 35 Total bytes written 16384 1+0 records in 1+0 records out 1024 bytes transferred in 0.000032 secs (31932842 bytes/sec) 1+0 records in 1+0 records out 1024 bytes transferred in 0.000035 secs (29217465 bytes/sec) . . Repetead until: . 0+0 records in 0+0 records out 0 bytes transferred in 0.000018 secs (0 bytes/sec)
Antti Louko wrote: > I just tried it in 4.6 Release and 4.6.2 and it still occurs. I think this is a kernel bug, rather than a pthread bug. What's happening is that the file descriptor associated with /dev/stdout is "inheriting" the flags associated with file descriptor 0. This happens on both -stable and -current. This seems like broken behavior to me: flags should be associated with the file descriptor, not the underlying device or entity. But I don't know what the "official" semantics of /dev/stdout are supposed to be. The program below demonstrates the problem: $ cc -g -Wall -o flags flags.c $ ./flags O_NONBLOCK is set -Archie __________________________________________________________________________ Archie Cobbs * Packet Design * http://www.packetdesign.com #include <stdio.h> #include <fcntl.h> #include <err.h> int main(int argc, char **argv) { int flags; int fd; if ((flags = fcntl(0, F_GETFL, 0)) == -1) err(1, "fcntl"); if (fcntl(0, F_SETFL, flags | O_NONBLOCK) == -1) err(1, "fcntl"); if ((fd = open("/dev/stdout", O_WRONLY, 0)) == -1) err(1, "open"); if ((flags = fcntl(fd, F_GETFL, 0)) == -1) err(1, "fcntl"); printf("O_NONBLOCK is %s\n", (flags & O_NONBLOCK) ? "set" : "not set"); return (0); }
Sender: archie@dellroad.org Date: Tue, 10 Sep 2002 08:26:40 -0700 From: Archie Cobbs <archie@packetdesign.com> Organization: Packet Design X-Accept-Language: en CC: freebsd-gnats-submit@FreeBSD.org, deischen@freebsd.org Antti Louko wrote: > I just tried it in 4.6 Release and 4.6.2 and it still occurs. I think this is a kernel bug, rather than a pthread bug. What's happening is that the file descriptor associated with /dev/stdout is "inheriting" the flags associated with file descriptor 0. This happens on both -stable and -current. This seems like broken behavior to me: flags should be associated with the file descriptor, not the underlying device or entity. But I don't know what the "official" semantics of /dev/stdout are supposed to be. The program below demonstrates the problem: $ cc -g -Wall -o flags flags.c $ ./flags O_NONBLOCK is set But the O_NONBLOCK is not set in the kernel. It is probably set in /usr/src/lib/libc_r/uthread/uthread_fd.c (which sets O_NONBLOCK flags for fds in range 0..2) or somewhere else in libc_r because it doesn't happen without thread library. But someone who is familiar with the pthreads library should check what should happen. I don't feel competent to fix this. Regards, Antti
42943 is a duplicate of 41331, or at best an extended symptom. First of all, both 41331 and 42943 occur on purpose, and this is not a regression introduced by some other change, it is part of the original pthread code. We believe that this behavior is a side effect of the FreeBSD threads implementation, and that 41331 and 42943 are normal behavior. These bugs can only plausibly be addressed by rearchitecting the threads implementation, possibly in the context of 5.x. In FreeBSD, the pthread implementation is as a user-level thread package, in other words, all threads execute in the context of a single process, and the thread scheduler is hidden in the threaded version of libc. As a side-effect of this implementation, when a blocking system call is executed, the kernel only knows about the process, and this results in all the threads being blocked. To avoid blocking the entire thread group, the stdin, stdout and stderr file descriptors are made non-blocking. Note that while these two bugs were filed against the recent 4.x releases, there is nothing new in this area. The code governing this behavior has 2.x and 3.x labels in the CVS tree. In 41331, there is a test program, and as part of our investigation we tried this program on both Solaris and Linux. Neither one exhibits the 41331 behavior, however readers who are familiar with the threads implementation will realize that these are three radically different threads implementations, accounting for the discrepancy. If the 41331 behavior is intolerable, there are two possible approaches, either wait for a pthreads implementation on top of the kernel threads in FreeBSD 5.0 or install the linuxthreads port, which simulates the Linux way of handling threads as separate processes. NOTE: The following code excerpt is from 4.7-STABLE ! Here are two code snips which control this behavior. code snip from libc_r/uthread/pthread_private.h : /* * Standard I/O file descriptors need special flag treatment since * setting one to non-blocking does all on *BSD. Sigh. This array * is used to store the initial flag settings. */ SCLASS int _pthread_stdio_flags[3]; code snip from uthread_fd.c : /* Check if a stdio descriptor: */ if ((fd < 3) && (_pthread_stdio_flags[fd] != -1)) /* * Use the stdio flags read by * _pthread_init() to avoid * mistaking the non-blocking * flag that, when set on one * stdio fd, is set on all stdio * fds. */ entry->flags = _pthread_stdio_flags[fd]; /* * Make the file descriptor non-blocking. * This might fail if the device driver does * not support non-blocking calls, or if the * driver is naturally non-blocking. */ saved_errno = errno; _thread_sys_fcntl(fd, F_SETFL, entry->flags | O_NONBLOCK); errno = saved_errno; Rashmi Venkatesh, engineer Dorr H. Clark, advisor COEN 284 - Operating Systems Case Study Santa Clara University, Santa Clara CA.
Responsible Changed From-To: freebsd-bugs->freebsd-threads Assign to threads mailing list
State Changed From-To: open->closed New libpthread in FreeBSD 5.x does not exhibit this problem.