Bug 32338
| Summary: | [patch] sis(4):Network to disk write performance low under ATA with DMA | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Base System | Reporter: | David S Madole <david> | ||||
| Component: | kern | Assignee: | freebsd-bugs (Nobody) <bugs> | ||||
| Status: | Closed FIXED | ||||||
| Severity: | Affects Only Me | ||||||
| Priority: | Normal | ||||||
| Version: | 4.4-RELEASE | ||||||
| Hardware: | Any | ||||||
| OS: | Any | ||||||
| Attachments: |
|
||||||
|
Description
David S Madole
2001-11-27 19:30:01 UTC
It seems David S Madole wrote: > >Description: > Data transfers from network to ATA hard drive are very slow. Visible > with Samba, HTTP PUT, etc., but most easily demonstrated with FTP. > Only occurs when DMA is enabled on the ATA controller. > > Interestingly, only seems to happen when network data is being written > to the drive. Doing a large write to the drive while simultaneously > doing a 'ping -f -s 1400' to another node from another session does > not slow the disk writes significantly. > > The drive is a Maxtor 60GB (model number in included dmesg). NIC is > NetGear with sis driver, although same problem occurs with LinkSys > card on dc driver. Also occurs with older Maxtor 6GB drive. Hmm, this sounds like the SiS network card may be using DMA too and badly at that, making the net card and the ATA driver compete for the bus... > >Fix: > I wish I knew where to look! Try another netcard, if that helps you know where to look :) -Søren >===== Original Message From sos@freebsd.dk ===== >It seems David S Madole wrote: >> >Description: >> Data transfers from network to ATA hard drive are very slow. Visible >> with Samba, HTTP PUT, etc., but most easily demonstrated with FTP. >> Only occurs when DMA is enabled on the ATA controller. >> >> Interestingly, only seems to happen when network data is being written >> to the drive. Doing a large write to the drive while simultaneously >> doing a 'ping -f -s 1400' to another node from another session does >> not slow the disk writes significantly. >> >> The drive is a Maxtor 60GB (model number in included dmesg). NIC is >> NetGear with sis driver, although same problem occurs with LinkSys >> card on dc driver. Also occurs with older Maxtor 6GB drive. > >Hmm, this sounds like the SiS network card may be using DMA too >and badly at that, making the net card and the ATA driver >compete for the bus... > >> >Fix: >> I wish I knew where to look! > >Try another netcard, if that helps you know where to look :) I think it's some kind of funniness with the chipset. As I mentioned originally, I get the same problem with a ADMtek AN985 on the dc driver. I also do not get the problem on an Intel Triton-based board with the same NIC and hard drive (and a much slower CPU - P-133 instead of K6-III-550). I added a little bit of instrumentation to the sis driver to keep track of receive descriptor/buffer overflows and it appears to be the problem. If I FTP more than about 150K, I get an overflow and slow performance (from TCP throttling back in response to packet loss). I changed the sis driver code to up the buffers from 128 to 1536, it still overflows. It appears that a NIC interrupt is getting lost, or the interrupt latency is going way up. So what in running the drive under DMA could cause interrupt problems for the NIC? I also forced the drive to WDMA2 instead of UDMA2 to match what the Triton-based machine was doing, but it made no difference. Didn't really expect it would. Dave >===== Original Message From sos@freebsd.dk ===== >It seems David S Madole wrote: >> >Description: >> Data transfers from network to ATA hard drive are very slow. Visible >> with Samba, HTTP PUT, etc., but most easily demonstrated with FTP. >> Only occurs when DMA is enabled on the ATA controller. >> >> Interestingly, only seems to happen when network data is being written >> to the drive. Doing a large write to the drive while simultaneously >> doing a 'ping -f -s 1400' to another node from another session does >> not slow the disk writes significantly. >> >> The drive is a Maxtor 60GB (model number in included dmesg). NIC is >> NetGear with sis driver, although same problem occurs with LinkSys >> card on dc driver. Also occurs with older Maxtor 6GB drive. > >Hmm, this sounds like the SiS network card may be using DMA too >and badly at that, making the net card and the ATA driver >compete for the bus... ( my apologies if anyone has received this already, but I don't think the copy I sent previously went out correctly since it never showed up on gnats and I was having mail trouble for a few days ) Bill, I copied you since this is your driver. Luigi, I apologize if inappropriate to copy you on this, but I see you've been active in the sis driver lately and thought you might be interested. Søren, thanks for the response. As I mentioned before, I did have this problem with the dc driver as well, on an ADMtek AN985. It turns out that the NIC was unable to DMA it's FIFO into memory fast enough, causing it to overflow and packets to get lost, throttling back TCP. It seems that my BIOS initializes the PCI latency timer on the card to 32, which doesn't let it move enough data at a time to keep up with 100Mbs while competing with the ATA controller. I am heistant to second-quess every BIOS, so I decided to try an adaptive approach and bump the timer up a little bit (32) each time an overflow occurs. On my machine, once it reaches 96 the card is stable and never overflows again. This increased my FTP-to-disk throughput to a little over 9MBps, pretty good. I also changed the driver to not reinitialize the card when an overflow or receive error occurs -- it seems unnecessary and takes a relatively long time, causing many packets to get lost. I also adjusted some NIC settings particular to the National chip to make them conditional on the silicon version, as the latest datasheet specifies. I'm guessing it doesn't really matter much since these are probably just defaults on the later chips. I think some other DMA-based NIC drivers, like dc could probably benefit from this as well. I can fix up the dc driver as well, if this seems like the best way to go, and see if it fixes the issue there, too. By the way, the attached diff is against 4.4-RELEASE. Thanks, Dave Hi, sorry for the delay with which I reply but i only had a chance to test this now. It seems that the removal of sis_init from sis_rxeoc() is definitely an important part of the fix, as in presence of bidirectional traffic an overflow on the receive side will cause a reset of the chip and drop all packets queued on the transmit side. Bill, do you think the removal of sis_init() has ill side effects otherwise ? I am bombing the interface with 148kpps (on a 486-133 so there are plenty of overflows) and things seem to work fine, much better indeed than when sis_init() was included in sis_rxeoc(). Not sure about the PCI latency register, i would like to test this a bit more, as there are many reasons why rxeoc is called. cheers luigi On Mon, Jan 07, 2002 at 08:13:19PM -0500, David S. Madole wrote: > >===== Original Message From sos@freebsd.dk ===== > >It seems David S Madole wrote: > >> >Description: > >> Data transfers from network to ATA hard drive are very slow. Visible > >> with Samba, HTTP PUT, etc., but most easily demonstrated with FTP. > >> Only occurs when DMA is enabled on the ATA controller. > >> > >> Interestingly, only seems to happen when network data is being written > >> to the drive. Doing a large write to the drive while simultaneously > >> doing a 'ping -f -s 1400' to another node from another session does > >> not slow the disk writes significantly. > >> > >> The drive is a Maxtor 60GB (model number in included dmesg). NIC is > >> NetGear with sis driver, although same problem occurs with LinkSys > >> card on dc driver. Also occurs with older Maxtor 6GB drive. > > > >Hmm, this sounds like the SiS network card may be using DMA too > >and badly at that, making the net card and the ATA driver > >compete for the bus... > > ( my apologies if anyone has received this already, but I don't think the > copy I sent previously went out correctly since it never showed up on > gnats > and I was having mail trouble for a few days ) > > Bill, I copied you since this is your driver. > > Luigi, I apologize if inappropriate to copy you on this, but I see you've > been > active in the sis driver lately and thought you might be interested. > > Søren, thanks for the response. As I mentioned before, I did have this > problem with the dc driver as well, on an ADMtek AN985. > > It turns out that the NIC was unable to DMA it's FIFO into memory fast > enough, > causing it to overflow and packets to get lost, throttling back TCP. It > seems > that my BIOS initializes the PCI latency timer on the card to 32, which > doesn't let it move enough data at a time to keep up with 100Mbs while > competing with the ATA controller. > > I am heistant to second-quess every BIOS, so I decided to try an adaptive > approach and bump the timer up a little bit (32) each time an overflow > occurs. > On my machine, once it reaches 96 the card is stable and never overflows > again. This increased my FTP-to-disk throughput to a little over 9MBps, > pretty > good. I also changed the driver to not reinitialize the card when an > overflow > or receive error occurs -- it seems unnecessary and takes a relatively long > time, causing many packets to get lost. > > I also adjusted some NIC settings particular to the National chip to make > them > conditional on the silicon version, as the latest datasheet specifies. I'm > guessing it doesn't really matter much since these are probably just > defaults > on the later chips. > > I think some other DMA-based NIC drivers, like dc could probably benefit > from > this as well. I can fix up the dc driver as well, if this seems like the > best > way to go, and see if it fixes the issue there, too. > > By the way, the attached diff is against 4.4-RELEASE. > > Thanks, > Dave > > > *** if_sisreg.h.orig Wed Feb 21 22:17:51 2001 > --- if_sisreg.h Thu Nov 29 19:28:35 2001 > *************** > *** 76,81 **** > --- 76,82 ---- > > /* NS DP83815 registers */ > #define NS_CLKRUN 0x3C > + #define NS_SRR 0x58 > #define NS_BMCR 0x80 > #define NS_BMSR 0x84 > #define NS_PHYIDR1 0x88 > *************** > *** 401,406 **** > --- 402,408 ---- > struct sis_list_data *sis_ldata; > struct sis_ring_data sis_cdata; > struct callout_handle sis_stat_ch; > + device_t sis_dev; > }; > > /* > *** if_sis.c.orig Wed Feb 21 22:17:51 2001 > --- if_sis.c Thu Nov 29 19:41:31 2001 > *************** > *** 723,728 **** > --- 723,730 ---- > unit = device_get_unit(dev); > bzero(sc, sizeof(struct sis_softc)); > > + sc->sis_dev = dev; > + > if (pci_get_device(dev) == SIS_DEVICEID_900) > sc->sis_type = SIS_TYPE_900; > if (pci_get_device(dev) == SIS_DEVICEID_7016) > *************** > *** 1159,1166 **** > void sis_rxeoc(sc) > struct sis_softc *sc; > { > sis_rxeof(sc); > - sis_init(sc); > return; > } > > --- 1161,1183 ---- > void sis_rxeoc(sc) > struct sis_softc *sc; > { > + int latency; > + > + /* > + * The BIOS may have initialized the maximum latency timer > + * too low to be able to keep up with a 100Mbs stream when > + * heavy disk or other DMA is taking place. Try to correct > + * for this adaptively by bumping it up a little each time > + * the receive FIFO overflows. > + */ > + > + latency = pci_read_config(sc->sis_dev, PCIR_LATTIMER, 1); > + if (latency < 255) { > + if ((latency += 32) > 255) latency = 255; > + pci_write_config(sc->sis_dev, PCIR_LATTIMER, latency, 1); > + } > + > sis_rxeof(sc); > return; > } > > *************** > *** 1293,1305 **** > sis_txeof(sc); > > if ((status & SIS_ISR_RX_DESC_OK) || > ! (status & SIS_ISR_RX_OK)) > sis_rxeof(sc); > > ! if ((status & SIS_ISR_RX_ERR) || > ! (status & SIS_ISR_RX_OFLOW)) { > sis_rxeoc(sc); > - } > > if (status & SIS_ISR_SYSERR) { > sis_reset(sc); > --- 1310,1321 ---- > sis_txeof(sc); > > if ((status & SIS_ISR_RX_DESC_OK) || > ! (status & SIS_ISR_RX_OK) || > ! (status & SIS_ISR_RX_ERR)) > sis_rxeof(sc); > > ! if (status & SIS_ISR_RX_OFLOW) > sis_rxeoc(sc); > > if (status & SIS_ISR_SYSERR) { > sis_reset(sc); > *************** > *** 1562,1569 **** > * performance." Note however that at least three > * of the registers are listed as "reserved" in > * the register map, so who knows what they do. > */ > ! if (sc->sis_type == SIS_TYPE_83815) { > CSR_WRITE_4(sc, NS_PHY_PAGE, 0x0001); > CSR_WRITE_4(sc, NS_PHY_CR, 0x189C); > CSR_WRITE_4(sc, NS_PHY_TDATA, 0x0000); > --- 1578,1593 ---- > * performance." Note however that at least three > * of the registers are listed as "reserved" in > * the register map, so who knows what they do. > + * > + * An appararently later version (December 2000) > + * has this data on page 78 and qualifies it as > + * applying only to silicon version 203h, which > + * is aparently a typo'd reference to version 302h > + * as referred to on page 64. > */ > ! if (sc->sis_type == SIS_TYPE_83815 && > ! CSR_READ_4(sc, NS_SRR) == 0x0302) { > ! > CSR_WRITE_4(sc, NS_PHY_PAGE, 0x0001); > CSR_WRITE_4(sc, NS_PHY_CR, 0x189C); > CSR_WRITE_4(sc, NS_PHY_TDATA, 0x0000); Luigi, Actually, in my case, it's the PCI latency that made all the difference. With the latency timer set to 32 -- the way my BIOS sets it up -- overflows occur many times a second. Removing sis_init helped a little because it caused only one or two packets to get dropped, instead of 10's or 100's while the card was begin reset. That change took me from 50Kbyte/sec to 500KByte/sec on FTP on 100MB full-duplex. Putting in the latency time patch causes two overflows to occur after boot-up, until the timer reaches 96. Then I never get an overflow again. From disk-to-disk between two machines with sis0 cards, I get 9.8Mbytes/second on FTP - fast! If I go from disk to /dev/null (ie GET test /dev/null), I get 11.8MByte/second, pretty much full wire-rate. I'm surprised you get a lot of overflows even on the slow machine, since the card empties through DMA. I can see mbufs might overflow, but not the card fifo at 33Mhz PCI DMA. You may need the latency timer increased on that machine, too. You can always bump it up by hand just to try (ie pciconf -w pci0:4:1 0xc 96). Thanks for looking into this, David On Saturday 09 February 2002 01:57 am, you wrote: > Hi, > sorry for the delay with which I reply but i only had a chance > to test this now. > It seems that the removal of sis_init from sis_rxeoc() is > definitely an important part of the fix, as in presence of > bidirectional traffic an overflow on the receive side will > cause a reset of the chip and drop all packets queued on > the transmit side. > > Bill, do you think the removal of sis_init() has ill side > effects otherwise ? I am bombing the interface with 148kpps > (on a 486-133 so there are plenty of overflows) > and things seem to work fine, much better indeed than > when sis_init() was included in sis_rxeoc(). > > Not sure about the PCI latency register, i would like > to test this a bit more, as there are many reasons why > rxeoc is called. > > cheers > luigi > > On Mon, Jan 07, 2002 at 08:13:19PM -0500, David S. Madole wrote: > > >===== Original Message From sos@freebsd.dk ===== > > > > > >It seems David S Madole wrote: > > >> >Description: > > >> > > >> Data transfers from network to ATA hard drive are very slow. > > >> Visible with Samba, HTTP PUT, etc., but most easily demonstrated > > >> with FTP. Only occurs when DMA is enabled on the ATA controller. > > >> > > >> Interestingly, only seems to happen when network data is being > > >> written to the drive. Doing a large write to the drive while > > >> simultaneously doing a 'ping -f -s 1400' to another node from > > >> another session does not slow the disk writes significantly. > > >> > > >> The drive is a Maxtor 60GB (model number in included dmesg). NIC > > >> is NetGear with sis driver, although same problem occurs with > > >> LinkSys card on dc driver. Also occurs with older Maxtor 6GB > > >> drive. > > > > > >Hmm, this sounds like the SiS network card may be using DMA too > > >and badly at that, making the net card and the ATA driver > > >compete for the bus... > > > > ( my apologies if anyone has received this already, but I don't > > think the copy I sent previously went out correctly since it never > > showed up on gnats > > and I was having mail trouble for a few days ) > > > > Bill, I copied you since this is your driver. > > > > Luigi, I apologize if inappropriate to copy you on this, but I see > > you've been > > active in the sis driver lately and thought you might be > > interested. > > > > Søren, thanks for the response. As I mentioned before, I did have > > this problem with the dc driver as well, on an ADMtek AN985. > > > > It turns out that the NIC was unable to DMA it's FIFO into memory > > fast enough, > > causing it to overflow and packets to get lost, throttling back > > TCP. It seems > > that my BIOS initializes the PCI latency timer on the card to 32, > > which doesn't let it move enough data at a time to keep up with > > 100Mbs while competing with the ATA controller. > > > > I am heistant to second-quess every BIOS, so I decided to try an > > adaptive approach and bump the timer up a little bit (32) each time > > an overflow occurs. > > On my machine, once it reaches 96 the card is stable and never > > overflows again. This increased my FTP-to-disk throughput to a > > little over 9MBps, pretty > > good. I also changed the driver to not reinitialize the card when > > an overflow > > or receive error occurs -- it seems unnecessary and takes a > > relatively long time, causing many packets to get lost. > > > > I also adjusted some NIC settings particular to the National chip > > to make them > > conditional on the silicon version, as the latest datasheet > > specifies. I'm guessing it doesn't really matter much since these > > are probably just defaults > > on the later chips. > > > > I think some other DMA-based NIC drivers, like dc could probably > > benefit from > > this as well. I can fix up the dc driver as well, if this seems > > like the best > > way to go, and see if it fixes the issue there, too. > > > > By the way, the attached diff is against 4.4-RELEASE. > > > > Thanks, > > Dave > > > > > > > > *** if_sisreg.h.orig Wed Feb 21 22:17:51 2001 > > --- if_sisreg.h Thu Nov 29 19:28:35 2001 > > *************** > > *** 76,81 **** > > --- 76,82 ---- > > > > /* NS DP83815 registers */ > > #define NS_CLKRUN 0x3C > > + #define NS_SRR 0x58 > > #define NS_BMCR 0x80 > > #define NS_BMSR 0x84 > > #define NS_PHYIDR1 0x88 > > *************** > > *** 401,406 **** > > --- 402,408 ---- > > struct sis_list_data *sis_ldata; > > struct sis_ring_data sis_cdata; > > struct callout_handle sis_stat_ch; > > + device_t sis_dev; > > }; > > > > /* > > *** if_sis.c.orig Wed Feb 21 22:17:51 2001 > > --- if_sis.c Thu Nov 29 19:41:31 2001 > > *************** > > *** 723,728 **** > > --- 723,730 ---- > > unit = device_get_unit(dev); > > bzero(sc, sizeof(struct sis_softc)); > > > > + sc->sis_dev = dev; > > + > > if (pci_get_device(dev) == SIS_DEVICEID_900) > > sc->sis_type = SIS_TYPE_900; > > if (pci_get_device(dev) == SIS_DEVICEID_7016) > > *************** > > *** 1159,1166 **** > > void sis_rxeoc(sc) > > struct sis_softc *sc; > > { > > sis_rxeof(sc); > > - sis_init(sc); > > return; > > } > > > > --- 1161,1183 ---- > > void sis_rxeoc(sc) > > struct sis_softc *sc; > > { > > + int latency; > > + > > + /* > > + * The BIOS may have initialized the maximum latency timer > > + * too low to be able to keep up with a 100Mbs stream when > > + * heavy disk or other DMA is taking place. Try to correct > > + * for this adaptively by bumping it up a little each time > > + * the receive FIFO overflows. > > + */ > > + > > + latency = pci_read_config(sc->sis_dev, PCIR_LATTIMER, 1); > > + if (latency < 255) { > > + if ((latency += 32) > 255) latency = 255; > > + pci_write_config(sc->sis_dev, PCIR_LATTIMER, latency, 1); > > + } > > + > > sis_rxeof(sc); > > return; > > } > > > > *************** > > *** 1293,1305 **** > > sis_txeof(sc); > > > > if ((status & SIS_ISR_RX_DESC_OK) || > > ! (status & SIS_ISR_RX_OK)) > > sis_rxeof(sc); > > > > ! if ((status & SIS_ISR_RX_ERR) || > > ! (status & SIS_ISR_RX_OFLOW)) { > > sis_rxeoc(sc); > > - } > > > > if (status & SIS_ISR_SYSERR) { > > sis_reset(sc); > > --- 1310,1321 ---- > > sis_txeof(sc); > > > > if ((status & SIS_ISR_RX_DESC_OK) || > > ! (status & SIS_ISR_RX_OK) || > > ! (status & SIS_ISR_RX_ERR)) > > sis_rxeof(sc); > > > > ! if (status & SIS_ISR_RX_OFLOW) > > sis_rxeoc(sc); > > > > if (status & SIS_ISR_SYSERR) { > > sis_reset(sc); > > *************** > > *** 1562,1569 **** > > * performance." Note however that at least three > > * of the registers are listed as "reserved" in > > * the register map, so who knows what they do. > > */ > > ! if (sc->sis_type == SIS_TYPE_83815) { > > CSR_WRITE_4(sc, NS_PHY_PAGE, 0x0001); > > CSR_WRITE_4(sc, NS_PHY_CR, 0x189C); > > CSR_WRITE_4(sc, NS_PHY_TDATA, 0x0000); > > --- 1578,1593 ---- > > * performance." Note however that at least three > > * of the registers are listed as "reserved" in > > * the register map, so who knows what they do. > > + * > > + * An appararently later version (December 2000) > > + * has this data on page 78 and qualifies it as > > + * applying only to silicon version 203h, which > > + * is aparently a typo'd reference to version 302h > > + * as referred to on page 64. > > */ > > ! if (sc->sis_type == SIS_TYPE_83815 && > > ! CSR_READ_4(sc, NS_SRR) == 0x0302) { > > ! > > CSR_WRITE_4(sc, NS_PHY_PAGE, 0x0001); > > CSR_WRITE_4(sc, NS_PHY_CR, 0x189C); > > CSR_WRITE_4(sc, NS_PHY_TDATA, 0x0000); On Sat, Feb 09, 2002 at 05:55:07AM +0000, David S Madole wrote: > Luigi, > > Actually, in my case, it's the PCI latency that made all the yes, i am not denying that, it's just that i havent had a chance to exercise that part of the code. > I'm surprised you get a lot of overflows even on the slow > machine, since the card empties through DMA. I can see mbufs well, it's ring buffer overflow, not DMA overflow. But the code calls rxeoc in both cases... cheers luigi Responsible Changed From-To: freebsd-bugs->sos Assign to ATA maintainer Responsible Changed From-To: sos->freebsd-bugs This is not an ATA issue State Changed From-To: open->feedback In the last two years several changes were committed to the sis driver. Did you test with a recent version of FreeBSD if your problem is fixed? From: "Tilman Linneweh" <arved@FreeBSD.org> > > Synopsis: Network to disk write performance low under ATA with DMA > > State-Changed-From-To: open->feedback > State-Changed-By: arved > State-Changed-When: Wed Aug 25 20:45:41 GMT 2004 > State-Changed-Why: > In the last two years several changes were committed to the sis driver. > Did you test with a recent version of FreeBSD if your problem is fixed? I don't know, as I no longer have a machine running day-to-day that is a good test case. After submitting this, I resigned to the fact that no one was interested, probably rightly so, as the driver is probably not the best place to fix what is really a BIOS PCI bus initialization issue, although there were other drivers, at the time at least, that did this, too. I maintained a local kernel patch for a little while, then realized it's really just as easy to do something like pciconf -w -b pci0:9:0 0xd 0x60 if rc.local, and that's what I did for a long time. In the last few months I have upgraded the machine that was the test case for this and it now has an Intel NIC. David State Changed From-To: feedback->closed Okay, since this is not easy reproducable anymore, I will close the PR. The pciconf-workaround is now documented in the mail-archive, in case the driver modifications of the last year(s) didn't fix this problem and someone else is running into it. Thanks for your quick response and your patience. |
