Created attachment 214188 [details]
pcap that generated the sample log entry in the bug description
I have a VPS webserver running FreeBSD 12.1-RELEASE-p4, and the caddy-1.0.4 package from the FreeBSD repository. I'm having an intermittent issue; every so often the webserver stops listening on port 80 which serves up HTTP->HTTPS redirects. Port 443 still works so Caddy is still running and the website is still available as long as you manually enter 'https://'
So far, all I've worked out is that whenever this happens, a log entry is generated reading:
May 6 03:46:26 server caddy: 2020/05/06 03:46:26 set tcp my.ip:80->remote.ip:52024: setsockopt: connection reset by peer
After this point, `lsof -nni` shows no process listening on port 80 anymore and trying to access this port from an HTTP client results in 'connection refused' as expected when no process is bound to a port. I've had a rolling tcpdump capture running to try and figure out if there's a particular traffic pattern that triggers this, and so far only one pattern seems to stand out. At the point this happens, my server sees roughly the following:
So the boxes triggering this appear to be roughly a 150ms round trip away in all cases and it seems to me like immediately after they send their regular ACK to me, they also send a RST/ACK without waiting for any response to the regular ACK. This is happening from multiple source IP addresses (some registered to AWS, some registered to a company called Dedipath, possibly others but I haven't checked every IP that has one of those log entries) and it's very intermittent. Sometimes it won't happen for months, sometimes it happens multiple times in a day. I've tried using Scapy to replicate this exact traffic but have been unable to so far (either I'm doing it wrong, or there's something different about Scapy-generated traffic that means it's not a valid reproduction)
I've brought this up with the Caddy project on GitHub but they haven't really been able to offer any solid suggestions so I'm opening a bug here to try and ascertain if it's a FreeBSD issue rather than a problem with Caddy. A couple of other GitHub users suggested they've seen the same behaviour so I intend to put the link for this bug on the Caddy issue tracker to see if they can provide any further information.
A brief update; since I put the effort in to build Caddy v2 and adapt my configuration to the new format, this problem has not returned (confirmed by one other user over on Github). So this seems isolated to the Caddy v1.x series which I believe is now unsupported upstream. So this bug is probably obsolete now.