Bug 70988

In short:
netisr_queue() has a bug of its return value.  Please apply the patch.

Background:
When I used ipfw fwd action to realize transparent HTTP proxy,
it worked fine for other machines, but trying "telnet www.foobar.com 80"
on the FreeBSD box resulted in "Operation not permitted."
(Configurations were proven to work fine on 4-stable.
I've been migrating to 5.3-BETA1.)

Analysis:
The reason of the error EPERM(=1) is that ip_output() return 1,
and it is because netisr_queue() return 1.
I observed netisr_queue() always returned 1, even when forwarding
works fine for client machines.

Look at netisr_queue() in sys/net/netisr.c:
| int
| netisr_queue(int num, struct mbuf *m)
| {
| 	struct netisr *ni;
| 	
| 	KASSERT(!(num < 0 || num >= (sizeof(netisrs)/sizeof(*netisrs))),
| 	    ("bad isr %d", num));
| 	ni = &netisrs[num];
| 	if (ni->ni_queue == NULL) {
| 		isrstat.isrs_drop++;
| 		m_freem(m);
| 		return (1);
| 	}
| 	isrstat.isrs_queued++;
| 	if (!IF_HANDOFF(ni->ni_queue, m, NULL))
| 		return (0);
| 	schednetisr(num);
| 	return (1);
| }

In the last line netisr_queue returns 1, but it should return 0.

Also, it seems that netisr_queue assumes IF_HANDOFF returns 0 when succeeded,
but it is wrong. IF_HANDOFF returns 0 when the queue is full, and returns
1 when succeeded:

| #define	IF_HANDOFF(ifq, m, ifp)			\
| 	if_handoff((struct ifqueue *)ifq, m, ifp, 0)

| static __inline int
| if_handoff(struct ifqueue *ifq, struct mbuf *m, struct ifnet *ifp, int adjust)
| {
| 	int active = 0;
| 
| 	IF_LOCK(ifq);
| 	if (_IF_QFULL(ifq)) {
| 		_IF_DROP(ifq);
| 		IF_UNLOCK(ifq);
| 		m_freem(m);
| 		return (0);
| 	}
| 	if (ifp != NULL) {
| 		ifp->if_obytes += m->m_pkthdr.len + adjust;
| 		if (m->m_flags & (M_BCAST|M_MCAST))
| 			ifp->if_omcasts++;
| 		active = ifp->if_flags & IFF_OACTIVE;
| 	}
| 	_IF_ENQUEUE(ifq, m);
| 	IF_UNLOCK(ifq);
| 	if (ifp != NULL && !active)
| 		if_start(ifp);
| 	return (1);
| }

So, this part in netisr_queue() should be changed to return 1:
| 	if (!IF_HANDOFF(ni->ni_queue, m, NULL))
| 		return (0);

How-To-Repeat: 
1. Set up squid as a transparent HTTP proxy in jail (192.168.79.21).
2. Add ipfw rules for forwarding:
 ipfw add allow tcp from 192.168.79.21 to any 80 out setup keep-state
 ipfw add fwd 192.168.79.21,8080 tcp from any to any 80 out setup keep-state
3. Browse some web sites with client machines. It should be OK.
4. Try "telnet www.example.com 80" on the FreeBSD box.
   You will see "Operation not permitted."