Bug 79208 - [nfs] Deadlock or starvation doing heavy NFS writes with writev
Summary: [nfs] Deadlock or starvation doing heavy NFS writes with writev
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 5.4-PRERELEASE
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2005-03-24 17:20 UTC by Marc Olzheim
Modified: 2007-11-17 03:19 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Marc Olzheim 2005-03-24 17:20:02 UTC
	Doing 60 MB of writev() over NFS, results in exactly 58638336
	bytes (57264K) ariving at the NFS server, _during_ which the
	machine becomes unresponsive to almost everything but pings...
	The resposiveness stops before 10 MB are written, maybe even as
	soon as the writev() starts.

	Running an "rtprio 0 cat" on console gives me a cat that
	responds to ^T (stty status ^T) with:

	load: 0.72  cmd: cat 673 [flswai] 0.00u 0.00s 0% 652k
	load: 8.87  cmd: cat 673 [flswai] 0.00u 0.29s 0% 652k

	and that still consumes input, but doesn't output anything
	anymore.

How-To-Repeat: 	writev 60 foo on an NFS mounted filesystem.

Source:
----------------------------------------------------------------------
/* Copyright (C) 2005 by ilse technology - Marc Olzheim */

#include	<err.h>
#include	<fcntl.h>
#include	<stdio.h>
#include	<stdlib.h>
#include	<strings.h>
#include	<sys/stat.h>
#include	<sys/types.h>
#include	<sys/uio.h>
#include	<unistd.h>

int
main(int argc, char *argv[])
{
	int		fd, i, iovcnt;
	struct iovec	*iov;
	char		buffer[1048576];
	ssize_t		written;

	if (3 != argc)
	{
		fprintf(stderr, "Usage: %s <megabytes> <file>\n", argv[0]);
		return(1);
	}

	if (-1 == (fd = open(argv[2], O_WRONLY | O_TRUNC | O_CREAT, S_IWUSR | S_IRUSR)))
		perror("open()");

	iovcnt = atoi(argv[1]);

	iov = malloc(iovcnt * sizeof(struct iovec));
	if (!iov)
		perror("malloc()");

	for (i = 0; i < iovcnt; i++)
	{
		iov[i].iov_base = buffer;
		iov[i].iov_len = sizeof(buffer);
	}

	bzero(buffer, sizeof(buffer));

	written = writev(fd, iov, iovcnt);
	if (written == (ssize_t)-1)
		err(1, "writev()");

	if (written != (ssize_t)(iovcnt * sizeof(buffer)))
		err(1, "Short write (%ld of %lu)",
			(long)written,
			(unsigned long)(iovcnt * sizeof(buffer)));

	if (close(fd))
		perror("close()");

	return(0);
}
----------------------------------------------------------------------
Comment 1 Marc Olzheim 2005-03-25 13:12:13 UTC
On a FreeBSD 6.0-CURRENT (Fri Mar 25 11:09:51 CET 2005) machine,
the same happens.

This machine has less memory 512 MB instead of 4GB, so the limit is
lower than 60 MB: 31 MB is enough. It produces a file of 31506432 bytes
on the NFS server, during which the machine becomes unresponsive and
then never recovers.

ping/ping6 still works, but that's all. Power cycling the machine is all
that's left...

Oh yeah, forgot to mention: this is as mortal user, not as root, so it's
a simple local Denial of Service attack. :-(

On FreeBSD-4.11 it works as expected, no hangs.

Marc
Comment 2 Marc Olzheim 2005-04-04 14:20:33 UTC
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

FreeBSD 5.3-RELEASE-p6 (SMP) #0: Wed Mar 30 14:28:31 CEST 2005
on i386 is vulnerable as well, so it has been b0rken for some time now,
I'm afraid.

Marc
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (FreeBSD)

iD8DBQFCUT8hezjnobFOgrERAg7xAKCVF1sLLaKzu7Mm5HaD96LXJFajfgCfYK+g
lga7RE/3bmqEM5tr6Ru4EGY=
=xpuk
-----END PGP SIGNATURE-----
Comment 3 Marc Olzheim 2005-04-05 14:21:07 UTC
Hmm, network services like an ircd, keep functioning until they need to
write a log entry, so it seems that the only thing that's "hanging", is
the VFS layer.

Marc
Comment 4 cel freebsd_committer freebsd_triage 2006-05-12 22:03:56 UTC
Responsible Changed
From-To: freebsd-bugs->cel
Comment 5 cel freebsd_committer freebsd_triage 2007-03-12 15:30:07 UTC
Responsible Changed
From-To: cel->freebsd-bugs

Back to the public pool.
Comment 6 K. Macy freebsd_committer freebsd_triage 2007-11-16 07:30:15 UTC
State Changed
From-To: open->feedback


Sounds like Giant contention. Is this still an issue on RELENG_7?
Comment 7 K. Macy freebsd_committer freebsd_triage 2007-11-17 03:18:55 UTC
State Changed
From-To: feedback->closed


Submitter says that this is fixed.